Related papers: A note on the state occupancy distribution for Mar…
We describe an exact approach for calculating transition probabilities and waiting times in finite-state discrete-time Markov processes. All the states and the rules for transitions between them must be known in advance. We can then…
For a given absorbing Markov chain $X^*$ on a finite state space, a chain $X$ is a sharp antidual of $X^*$ if the fastest strong stationary time of $X$ is equal, in distribution, to the absorption time of $X^*$. In this paper we show a…
Discrete-time discrete-state finite Markov chains are versatile mathematical models for a wide range of real-life stochastic processes. One of most common tasks in studies of Markov chains is computation of the stationary distribution.…
Consider a system evolving according to an absorbing discrete-time Markov chain with known transition matrix. The state of the system is observed at two points in time, separated by an unknown number of generations. We are interested in…
In this paper, we study quasi-stationary distributions of nonlinearly perturbed semi-Markov processes in discrete time. This type of distributions is of interest for the analysis of stochastic systems which have finite lifetimes, but are…
This work focuses on time-inhomogeneous Markov chains with two time scales. Our motivations stem from applications in reliability and dependability, queueing networks, financial engineering and manufacturing systems, where two-time-scale…
We develop some sufficient conditions for the stochastic ordering between hitting times, in a fixed state, for two Markov chains. In particular, we focus attention on the so called \emph{skip-free} case. In the analysis of such a case, we…
In this note we derive the exact probability that a specific state in a two-state Markov chain is visited exactly $k$ times after $N$ transitions. We provide a closed-form solution for $\mathbb{P}(N_l = k \mid N)$, considering initial state…
For a skip-free Markov process on non-negative integers with generator matrix Q, we evaluate the joint Laplace transform of the occupation times before hitting the state n (starting at 0). This Laplace transform has a very straightforward…
This article studies the expected occupancy probabilities on an alphabet. Unlike the standard situation, where observations are assumed to be independent and identically distributed (iid), we assume that they follow a regime switching…
Occupancy processes are a broad class of discrete time Markov chains on $\{0,1\}^{n}$ encompassing models from diverse areas. This model is compared to a collection of $n$ independent Markov chains on $\{0,1\}$, which we call the…
We present a systematic study of the statistics of the occupation time and related random variables for stochastic processes with independent intervals of time. According to the nature of the distribution of time intervals, the probability…
We consider continuous-time Markov chains on integers which allow transitions to adjacent states only, with alternating rates. We give explicit formulas for probability generating functions, and also for means, variances and state…
We consider an additive functional driven by a time-inhomogeneous Markov chain with a finite state space. Our study focuses on the joint distribution of the two-sided exit time and the state of the driving Markov chain at the time of exit,…
This article describes an accurate procedure for computing the mean first passage times of a finite irreducible Markov chain and a Markov renewal process. The method is a refinement to the Kohlas, Zeit fur Oper Res, 30,197-207, (1986)…
A method of constructing Markov chains on finite state spaces is provided. The chain is specified by three constraints: stationarity, dependence and marginal distributions. The generalized Pythagorean theorem in information geometry plays a…
We describe a simple method that can be used to sample the rare fluctuations of discrete-time Markov chains. We focus on the case of Markov chains with well-defined steady-state measures, and derive expressions for the large-deviation rate…
Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such…
In this paper, we consider a general class of two-time-scale Markov chains whose transition rate matrix depends on a parameter $\lambda>0$. We assume that some transition rates of the Markov chain will tend to infinity as…
The cover time of a Markov chain on a finite state space is the expected time until all states are visited. We show that if the cover time of a discrete-time Markov chain with rational transitions probabilities is bounded, then it is a…