Related papers: Offline Estimation of Controlled Markov Chains: Mi…
This tutorial describes recently developed general optimality conditions for Markov Decision Processes that have significant applications to inventory control. In particular, these conditions imply the validity of optimality equations and…
We prove that under mild positivity assumptions the entropy rate of a hidden Markov chain varies analytically as a function of the underlying Markov chain parameters. A general principle to determine the domain of analyticity is stated. An…
We prove finite sample complexities for sequential Monte Carlo (SMC) algorithms which require only local mixing times of the associated Markov kernels. Our bounds are particularly useful when the target distribution is multimodal and global…
This article provides the first procedure for computing a fully data-dependent interval that traps the mixing time $t_{\text{mix}}$ of a finite reversible ergodic Markov chain at a prescribed confidence level. The interval is computed from…
In this paper we present elementary computations for some Markov modulated counting processes, also called counting processes with regime switching. Regime switching has become an increasingly popular concept in many branches of science. In…
Markov chain Monte Carlo (MCMC) methods asymptotically sample from complex probability distributions. The pseudo-marginal MCMC framework only requires an unbiased estimator of the unnormalized probability distribution function to construct…
We consider whether ergodic Markov chains with bounded step size remain bounded in probability when their transitions are modified by an adversary on a bounded subset. We provide counterexamples to show that the answer is no in general, and…
We develop a martingale approximation approach to studying the limiting behavior of quadratic forms of Markov chains. We use the technique to examine the asymptotic behavior of lag-window estimators in time series and we apply the results…
Combining information both within and across trajectories, we propose a simple estimator for the local regularity of the trajectories of a stochastic process. Independent trajectories are measured with errors at randomly sampled time…
In this paper, we propose a policy gradient method for confounded partially observable Markov decision processes (POMDPs) with continuous state and observation spaces in the offline setting. We first establish a novel identification result…
Many natural Markov chains fail to mix to their stationary distribution in polynomially many steps. Often, this slow mixing is inevitable since it is computationally intractable to sample from their stationary measure. Nevertheless, Markov…
Many probabilistic inference problems such as stochastic filtering or the computation of rare event probabilities require model analysis under initial and terminal constraints. We propose a solution to this bridging problem for the widely…
In this paper, we consider a modified version of the control problem in a model free Markov decision process (MDP) setting with large state and action spaces. The control problem most commonly addressed in the contemporary literature is to…
In this paper, we give quantitative bounds on the $f$-total variation distance from convergence of an Harris recurrent Markov chain on an arbitrary under drift and minorisation conditions implying ergodicity at a sub-geometric rate. These…
For a Markov chain $Y$ with values in a Polish space, consider the entrance Markov chain obtained by sampling $Y$ at the moments when it enters a fixed set $A$ from its complement $A^c$. Similarly, consider the exit Markov chain, obtained…
The problem of sampling from the stationary distribution of a Markov chain finds widespread applications in a variety of fields. The time required for a Markov chain to converge to its stationary distribution is known as the classical…
Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data…
Nonlinear Markov chains with finite state space have been introduced in Kolokoltsov (2010). The characteristic property of these processes is that the transition probabilities do not only depend on the state, but also on the distribution of…
Learning methods are increasingly used to synthesize controllers from data, yet existing sample-complexity characterizations for continuous control are sharp only in the fully observed setting. This paper studies the partially observed case…
We consider a piecewise deterministic Markov decision process, where the expected exponential utility of total (nonnegative) cost is to be minimized. The cost rate, transition rate and post-jump distributions are under control. The state…