Related papers: Convergence of a Second Order Markov Chain

We provide a sufficient criterion for the unique parameter identification of combinatorially symmetric Hidden Markov Models based on the structure of their transition matrix. If the observed states of the chain form a zero forcing set of…

We consider metrics which are preserved under a $p$-Wasserstein transport map, up to a possible contraction. In the case $p=1$ this corresponds to a metric which is uniformly curved in the sense of coarse Ricci curvature. We investigate the…

This paper is a survey of various proofs of the so called {\em fundamental theorem of Markov chains}: every ergodic Markov chain has a unique positive stationary distribution and the chain attains this distribution in the limit independent…

In this work, we characterise the statistics of Markov chains by constructing an associated sequence of periodic differential operators. Studying the density of states of these operators reveals the absolutely continuous invariant measure…

In many branches of engineering, Banach contraction mapping theorem is employed to establish the convergence of certain deterministic algorithms. Randomized versions of these algorithms have been developed that have proved useful in…

We consider continuous-time Markov chain on a finite state space X. We assume X can be clustered into several subsets such that the intra-transition rates within these subsets are of order $\mathcal{O}(\frac{1}{\epsilon})$ comparing to the…

Let $\{X_n\}_{n\in\N}$ be a Markov chain on a measurable space $\X$ with transition kernel $P$ and let $V:\X\r[1,+\infty)$. The Markov kernel $P$ is here considered as a linear bounded operator on the weighted-supremum space $\cB_V$…

The theory of ``Markov-up'' processes is being developed. This is a new class of stochastic processes with ``partial'' markovian features; it could also be called ``one-sided Markov''. Such a behavior may be found in the real world and in…

Consider a sequence $(\eta^N(t) :t\ge 0)$ of continuous-time, irreducible Markov chains evolving on a fixed finite set $E$, indexed by a parameter $N$. Denote by $R_N(\eta,\xi)$ the jump rates of the Markov chain $\eta^N_t$, and assume that…

We define the spectral gap of a Markov chain on a finite state space as the second-smallest singular value of the generator of the chain, generalizing the usual definition of spectral gap for reversible chains. We then define the relaxation…

We fix $d \geq 2$ and denote $\mathcal S$ the semi-group of $d \times d$ matrices with non negative entries. We consider a sequence $(A_n, B_n)_{n \geq 1} $ of i. i. d. random variables with values in $\mathcal S\times \mathbb R_+^d$ and…

Markov decision process (MDP) is a decision making framework where a decision maker is interested in maximizing the expected discounted value of a stream of rewards received at future stages at various states which are visited according to…

The paper deals with the convergence properties of the products of random (row-)stochastic matrices. The limiting behavior of such products is studied from a dynamical system point of view. In particular, by appropriately defining a dynamic…

Let $X_n, n \ge 0$ be a Markov chain with finite state space $M$. If $x,y \in M$ such that $x$ is transient we have $P^y(X_n = x) \to 0$ for $n \to \infty$, and under mild aperiodicity conditions this convergence is monotone in that for…

We propose Markov two-components processes (M2CP) as a probabilistic model of asynchronous systems based on the trace semantics for concurrency. Considering an asynchronous system distributed over two sites, we introduce concepts and tools…

A new approach is developed for evaluating the convergence rate for nonlinear Markov chains (MC) based on the recently developed spectral radius technique of markovian coupling for linear MC and the idea of small nonlinear perturbations of…

Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a…

The paper studies the higher-order absolute differences taken from progressive terms of time-homogenous binary Markov chains. Two theorems presented are the limiting theorems for these differences, when their order $k$ converges to…

We develop a practical approach to establish the stability, that is, the recurrence in a given set, of a large class of controlled Markov chains. These processes arise in various areas of applied science and encompass important numerical…

We propose a new approach for estimating the finite dimensional transition matrix of a Markov chain using a large number of independent sample paths observed at random times. The sample paths may be observed as few as two times, and the…