Related papers: Finiteness of hitting times under taboo

Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not --…

For each $n$ let $Y^n_t$ be a continuous time symmetric Markov chain with state space $n^{-1} \Z^d$. A condition in terms of the conductances is given for the convergence of the $Y^n_t$ to a symmetric Markov process $Y_t$ on $\R^d$. We have…

Arguing about the equilibrium distribution of continuous-time Markov chains can be vital for showing properties about the underlying systems. For example in biological systems, bistability of a chemical reaction network can hint at its…

Let X be a continuous-time Markov chain in a finite set I, let h be a mapping of I onto another set, and let Y be defined by Y_t=h(X_t), (for t nonnegative). We address the filtering problem for X in terms of the observation Y, which is not…

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterise the value function via HJB equation…

Scaled type Markov renewal processes generalize classical renewal processes: renewal times come from a one parameter family of probability laws and the sequence of the parameters is the trajectory of an ergodic Markov chain. Our primary…

Let $(Z_n)_{n\geqslant 0}$ be a branching process in a random environment defined by a Markov chain $(X_n)_{n\geqslant 0}$ with values in a finite state space $\mathbb X$ starting at $X_0=i \in\mathbb X$. We extend from the i.i.d.…

This paper concentrates on the minimal hitting probability of continuous-time controlled Markov systems (CTCMSs) with countable state and finite admissible action spaces. The existence of an optimal policy is first proved. In particular,…

For a class of irreducible Markov chains with an infinitely countable set of states, we establish a new verifiable necessary and sufficient condition for recurrence and transience. We show that if one of the basic assumptions is not…

Probabilistic guarantees of safety and performance are important in constrained dynamical systems with stochastic uncertainty. We consider the stochastic reachability problem, which maximizes the probability that the state remains within…

For one-dimensional symmetric L\'{e}vy processes, which hit every point with positive probability, we give sharp bounds for the tail function of the first hitting time of B which is either a single point or an interval. The estimates are…

This paper focuses on the design of time-invariant memoryless control policies for fully observed controlled Markov chains, with a finite state space. Safety constraints are imposed through a pre-selected set of forbidden states. A state is…

A well-known theorem for an irreducible skip-free chain with absorbing state $d$, under some conditions, is that the hitting (absorbing) time of state $d$ starting from state 0 is distributed as the sum of $d$ independent geometric (or…

Using terminologies of information geometry, we derive upper and lower bounds of the tail probability of the sample mean. Employing these bounds, we obtain upper and lower bounds of the minimum error probability of the 2nd kind of error…

In the context of Markov decision processes running in continuous time, one of the most intriguing challenges is the efficient approximation of finite horizon reachability objectives. A multitude of sophisticated model checking algorithms…

We study the limiting object of a sequence of Markov chains analogous to the limits of graphs, hypergraphs, and other objects which have been studied. Following a suggestion of Aldous, we assign to a sequence of finite Markov chains with…

We study the asymptotic hitting time $\tau^{(n)}$ of a family of Markov processes $X^{(n)}$ to a target set $G^{(n)}$ when the process starts from a trap defined by very general properties. We give an explicit description of the law of…

We consider the problem of bounding mean first passage times for a class of continuous-time Markov chains that captures stochastic interactions between groups of identical agents. The quantitative analysis of such probabilistic population…

A sequence of real numbers (x_n) is Benford if the significands, i.e. the fraction parts in the floating-point representation of (x_n) are distributed logarithmically. Similarly, a discrete-time irreducible and aperiodic finite-state Markov…

We consider Markov decision processes with synchronizing objectives, which require that a probability mass of $1-\epsilon$ accumulates in a designated set of target states, either once, always, infinitely often, or always from some point…