Related papers: A solution to the reversible embedding problem for…
We consider backward stochastic differential equations (BSDEs) related to finite state, continuous time Markov chains. We show that appropriate solutions exist for arbitrary terminal conditions, and are unique up to sets of measure zero. We…
We consider the problem of finding the Perron-Frobenius eigenvector of a primitive matrix. Dividing each of the rows of the matrix by the sum of the elements in the row, the resulting new matrix is stochastic. We give a formula for the…
We study the convergence of random function iterations for finding an invariant measure of the corresponding Markov operator. We call the problem of finding such an invariant measure the stochastic fixed point problem. This generalizes…
Markov chains are one of the well-known tools for modeling and analyzing stochastic systems. At the same time, they are used for constructing random walks that can achieve a given stationary distribution. This paper is concerned with…
Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a…
We consider ergodic backward stochastic differential equations, in a setting where noise is generated by a countable state uniformly ergodic Markov chain. We show that for Lipschitz drivers such that a comparison theorem holds, these…
We establish a new Bernstein-type deviation inequality for general (non-reversible) discrete-time Markov chains via an elementary approach. More robust than existing works in the literature, our result only requires the Markov chain to…
We study the problem of sequentially testing whether a given stochastic process is generated by a known Markov chain. Formally, given access to a stream of random variables, we want to quickly determine whether this sequence is a trajectory…
We address the problem of estimating the mixing time $t_{\mathsf{mix}}$ of an arbitrary ergodic finite-state Markov chain from a single trajectory of length $m$. The reversible case was addressed by Hsu et al. [2019], who left the general…
We formulate some simple conditions under which a Markov chain may be approximated by the solution to a differential equation, with quantifiable error probabilities. The role of a choice of coordinate functions for the Markov chain is…
We propose an exact technique to calculate lower bounds of spectral gaps of discrete time reversible Markov chains on finite state sets. Spectral gaps are a common tool for evaluating convergence rates of Markov chains. As an illustration,…
This paper is a survey of various proofs of the so called {\em fundamental theorem of Markov chains}: every ergodic Markov chain has a unique positive stationary distribution and the chain attains this distribution in the limit independent…
We introduce a novel time-homogeneous Markov embedding of a class of time inhomogeneous Markov chains widely used in the context of Monte Carlo sampling algorithms which allows us to answer one of the most basic, yet hard, question about…
We give computable bounds on the rate of convergence of the transition probabilities to the stationary distribution for a certain class of geometrically ergodic Markov chains. Our results are different from earlier estimates of Meyn and…
Hidden Markov chains are widely applied statistical models of stochastic processes, from fundamental physics and chemistry to finance, health, and artificial intelligence. The hidden Markov processes they generate are notoriously…
In order to approximate a continuous time stochastic process by discrete time Markov chains one has several options to embed the Markov chains into continuous time processes. On the one hand there is the Markov embedding, which uses…
Reversibility is a key concept in Markov models and Master-equation models of molecular kinetics. The analysis and interpretation of the transition matrix encoding the kinetic properties of the model relies heavily on the reversibility…
The embeddability of reversible Markov matrices into time-homogeneous Markov semigroups is revisited, with some focus on simplifications and extensions. In particular, we do not demand irreducibility and consider weakly reversible matrices…
We revisit the classical problem of approximating a stochastic differential equation by a discrete-time and discrete-space Markov chain. Our construction iterates Caratheodory's theorem over time to match the moments of the increments…
In this paper we study various properties of finite stochastic systems or hidden Markov chains as they are alternatively called. We discuss their construction following different approaches and we also derive recursive filtering formulas…