Related papers: Asymptotically Optimal Sequential Testing with Mar…
We address the problem of policy evaluation in discounted Markov decision processes, and provide instance-dependent guarantees on the $\ell_\infty$-error under a generative model. We establish both asymptotic and non-asymptotic versions of…
We consider synthesis of control policies that maximize the probability of satisfying given temporal logic specifications in unknown, stochastic environments. We model the interaction between the system and its environment as a Markov…
We study the problem of identity testing of markov chains. In this setting, we are given access to a single trajectory from a markov chain with unknown transition matrix $Q$ and the goal is to determine whether $Q = P$ for some known matrix…
In this work, we show existence of invariant ergodic measure for switched linear dynamical systems (SLDSs) under a norm-stability assumption of system dynamics in some unbounded subset of $\mathbb{R}^{n}$. Consequently, given a stationary…
Recurrent temporal dynamics is a phenomenon observed frequently in high-dimensional complex systems and its detection is a challenging task. Recurrence quantification analysis utilizing recurrence plots may extract such dynamics, however it…
We present a learning model predictive control (MPC) scheme for chance-constrained Markov jump systems with unknown switching probabilities. Using samples of the underlying Markov chain, ambiguity sets of transition probabilities are…
Standard Markovian optimal stopping problems are consistent in the sense that the first entrance time into the stopping set is optimal for each initial state of the process. Clearly, the usual concept of optimality cannot in a…
Markov chains are fundamental models for stochastic dynamics, with applications in a wide range of areas such as population dynamics, queueing systems, reinforcement learning, and Monte Carlo methods. Estimating the transition matrix and…
We prove explicit error bounds for Markov chain Monte Carlo (MCMC) methods to compute expectations of functions with unbounded stationary variance. We assume that there is a $p\in(1,2)$ so that the functions have finite $L_p$-norm. For…
This paper presents a new condition for the existence of optimal stationary policies in average-cost continuous-time Markov decision processes with unbounded cost and transition rates, arising from controlled queueing systems. This…
Consider a Markov chain $\{X_n\}_{n\ge 0}$ with an ergodic probability measure $\pi$. Let $\Psi$ a function on the state space of the chain, with $\alpha$-tails with respect to $\pi$, $\alpha\in (0,2)$. We find sufficient conditions on the…
Let \xi_0,\xi_1,...,\xi_{\omega-1} be observations from the hidden Markov model with probability distribution P^{\theta_0}, and let \xi_{\omega},\xi_{\omega+1},... be observations from the hidden Markov model with probability distribution…
This work deals with a general problem of testing multiple hypotheses about the distribution of a discrete-time stochastic process. Both the Bayesian and the conditional settings are considered. The structure of optimal sequential tests is…
We solve a linear quadratic optimal control problem for sampled-data systems with stochastic delays. The delays are stochastically determined by the last few delays. The proposed optimal controller can be efficiently computed by iteratively…
We study the sequential testing problem of two alternative hypotheses regarding an unknown parameter in an exponential family when observations are costly. In a Bayesian setting, the problem can be embedded in a Markovian framework. Using…
The Bayesian formulation of sequentially testing $M \ge 3$ hypotheses is studied in the context of a decentralized sensor network system. In such a system, local sensors observe raw observations and send quantized sensor messages to a…
This work provides test error bounds for iterative fixed point methods on linear predictors -- specifically, stochastic and batch mirror descent (MD), and stochastic temporal difference learning (TD) -- with two core contributions: (a) a…
We show that the optimal decision policy for several types of Bayesian sequential detection problems has a threshold switching curve structure on the space of posterior distributions. This is established by using lattice programming and…
We present a test for the problem of decentralized sequential hypothesis testing, which is asymptotically optimum. By selecting a suitable sampling mechanism at each sensor, communication between sensors and fusion center is asynchronous…
It has been recently shown that e-processes are sufficient for sequential testing in the following sense: every level-$\alpha$ sequential test can be obtained by thresholding an e-process at $1/\alpha$. However, in the above result, neither…