Related papers: Ergodic Theory for Controlled Markov Chains with S…

A piecewise-deterministic Markov process, specified by random jumps and switching semi-flows, as well as the associated Markov chain given by its post-jump locations, are investigated in this paper. The existence of an exponentially…

We develop a practical approach to establish the stability, that is, the recurrence in a given set, of a large class of controlled Markov chains. These processes arise in various areas of applied science and encompass important numerical…

Whereas classical invariance principles for ergodic Markov chains address the situation in which the time horizon of observations is much larger than the mixing time, the quality of approximation is questionable when this is not the case…

In this work, we study a natural nonparametric estimator of the transition probability matrices of a finite controlled Markov chain. We consider an offline setting with a fixed dataset, collected using a so-called logging policy. We develop…

We address the problem of estimating the mixing time $t_{\mathsf{mix}}$ of an arbitrary ergodic finite-state Markov chain from a single trajectory of length $m$. The reversible case was addressed by Hsu et al. [2019], who left the general…

This paper aims to establish a central limit theorem for Markov processes conditioned not to be absorbed under a very general assumption on quasi-stationarity for the underlying process. To do so, a central limit theorem has been…

Many years ago B.S. Pitskel observed that the metric entropy of the shift transformation in the sample space of a stationary random process $X=\{X_n,\,n\in \mathbb Z\}$ with a countable number of states is equal to the conditional entropy…

We study a fairly general class of time-homogeneous stochastic evolutions driven by noises that are not white in time. As a consequence, the resulting processes do not have the Markov property. In this setting, we obtain constructive…

We present some new results on sample path optimality for the ergodic control problem of a class of non-degenerate diffusions controlled through the drift. The hypothesis most often used in the literature to ensure the existence of an a.s.…

Theory and application of stochastic approximation (SA) have become increasingly relevant due in part to applications in optimization and reinforcement learning. This paper takes a new look at SA with constant step-size $\alpha>0$, defined…

In the paper, we study a new rate of convergence estimate for homogeneous discrete-time nonlinear Markov chains based on the Markov-Dobrushin condition. This result generalizes the convergence estimates for any positive number of transition…

We study the problem of sequentially testing whether a given stochastic process is generated by a known Markov chain. Formally, given access to a stream of random variables, we want to quickly determine whether this sequence is a trajectory…

We introduce a statistical mechanics formalism for the study of constrained graph evolution as a Markovian stochastic process, in analogy with that available for spin systems, deriving its basic properties and highlighting the role of the…

We find the class, ${\cal{C}}_k, k \ge 0$, of all zero mean stationary Gaussian processes, $Y(t), ~t \in \reals$ with $k$ derivatives, for which \begin{equation} Z(t) \equiv (Y^{(0)}(t), Y^{(1)}(t), \ldots, Y^{(k)}(t) ), ~ t \ge 0…

This paper provides a new path method that can be used to determine when an ergodic continuous-time Markov chain on $\mathbb Z^d$ converges exponentially fast to its stationary distribution in $L^2$. Specifically, we provide general…

Markov chains are fundamental models for stochastic dynamics, with applications in a wide range of areas such as population dynamics, queueing systems, reinforcement learning, and Monte Carlo methods. Estimating the transition matrix and…

The generic behavior of quantum systems has long been of theoretical and practical interest. Any quantum process is represented by a sequence of quantum channels. We consider general ergodic sequences of stochastic channels with arbitrary…

For a network of discrete states with a periodically driven Markovian dynamics, we develop an inference scheme for an external observer who has access to some transitions. Based on waiting-time distributions between these transitions, the…

We consider the class of stationary-increment harmonizable stable processes with infinite control measure, which most notably includes real harmonizable fractional stable motions. We give conditions for the integrability of the paths of…

In this paper we discuss how the notion of subgeometric ergodicity in Markov chain theory can be exploited to study stationarity and ergodicity of nonlinear time series models. Subgeometric ergodicity means that the transition probability…