Related papers: Relation between Hitting Times and Probabilities f…
We consider discrete-time Markov bridges, chains whose initial and final states coincide. We derive exact finite-time formulae for the joint probability distributions of additive functionals of trajectories. We apply our theory to…
For an ergodic Markov chain $\{X(t)\}$ on $\Bbb N$, with a stationary distribution $\pi$, let $T_n>0$ denote a hitting time for $[n]^c$, and let $X_n=X(T_n)$. Around 2005 Guy Louchard popularized a conjecture that, for $n\to \infty$, $T_n$…
In the classical theory of Markov chains, one may study the mean time to reach some chosen state, and it is well-known that in the irreducible, finite case, such quantity can be calculated in terms of the fundamental matrix of the walk, as…
Random walks (or Markov chains) are models extensively used in theoretical computer science. Several tools, including analysis of quantities such as hitting and mixing times, are helpful for devising randomized algorithms. A notable example…
We consider the safety evaluation of discrete time, stochastic systems over a finite horizon. Therefore, we discuss and link probabilistic invariance with reachability as well as reach-avoid problems. We show how to efficiently compute…
The first-passage time (FPT) is the time it takes a system variable to cross a given boundary for the first time. In the context of Markov networks, the FPT is the time a random walker takes to reach a particular node (target) by hopping…
For Markov jump processes on irreducible networks with finite number of sites, we derive a general and explicit expression of the squared coefficient of variation for the net number of transitions from one site to a connected site in a…
We present a learning model predictive control (MPC) scheme for chance-constrained Markov jump systems with unknown switching probabilities. Using samples of the underlying Markov chain, ambiguity sets of transition probabilities are…
We present a short and self-contained proof of a classical result due to Bollob\'as (1990): in the random hypercube process, with high probability the hitting time of connectedness equals the hitting time of having minimum degree at least…
We investigate the mixing properties of a finite Markov chain in random environment defined as a mixture of a deterministic chain and a chain whose state space has been permuted uniformly at random. This work is the counterpart of a…
Extending the approach of the paper [Mathieu, P. (1997) Hitting times and spectral gap inequalities, Ann. Inst. Henri Poincare 33, 4, 437 -- 465], we prove that the Poincare inequality for a (possibly non-symmetric) Markov process yields…
We consider continuous-time Markov chains on integers which allow transitions to adjacent states only, with alternating rates. We give explicit formulas for probability generating functions, and also for means, variances and state…
The convergence rate of a Markov chain to its stationary distribution is typically assessed using the concept of total variation mixing time. However, this worst-case measure often yields pessimistic estimates and is challenging to infer…
Consider a Markov process \omega_t at equilibrium and some event C (a subset of the state-space of the process). A natural measure of correlations in the process is the pairwise correlation \Pr[\omega_0,\omega_t \in C] - \Pr[\omega_0 \in…
Computing optimal conditional reachability probabilities in Markov decision processes (MDPs) is tractable by a reduction to reachability probabilities. Yet, this reduction yields cyclic, challenging MDPs that are often notoriously hard to…
Markov chains are one of the well-known tools for modeling and analyzing stochastic systems. At the same time, they are used for constructing random walks that can achieve a given stationary distribution. This paper is concerned with…
We study a large class of reversible Markov chains with discrete state space and transition matrix $P_N$. We define the notion of a set of {\it metastable points} as a subset of the state space $\G_N$ such that (i) this set is reached from…
We propose a discrete time discrete space Markov chain approximation with a Brownian bridge correction for computing curvilinear boundary crossing probabilities of a general diffusion process on a finite time interval. For broad classes of…
In the context of Markov decision processes running in continuous time, one of the most intriguing challenges is the efficient approximation of finite horizon reachability objectives. A multitude of sophisticated model checking algorithms…
We consider a simple but important class of metastable discrete time Markov chains, which we call perturbed Markov chains. Basically, we assume that the transition matrices depend on a parameter $\varepsilon$, and converge as $\varepsilon$.…