Related papers: Finiteness of hitting times under taboo
Markov decision processes model systems subject to nondeterministic and probabilistic uncertainty. A plethora of verification techniques addresses variations of reachability properties, such as: Is there a scheduler resolving the…
We consider discrete-time Markov decision processes in which the decision maker is interested in long but finite horizons. First we consider reachability objective: the decision maker's goal is to reach a specific target state with the…
Markov branching systems form a fundamental class of stochastic models that are extensively applied in biology, physics, finance, and other domains. These systems are distinguished by their continuous-time evolution and inherent branching…
For hidden Markov models one of the most popular estimates of the hidden chain is the Viterbi path -- the path maximising the posterior probability. We consider a more general setting, called the pairwise Markov model, where the joint…
The Doeblin Graph of a countable state space Markov chain describes the joint pathwise evolutions of the Markov dynamics starting from all possible initial conditions, with two paths coalescing when they reach the same point of the state…
The paper is largely of a review nature. It considers two main methods used to study stability and obtain appropriate quantitative estimates of perturbations of (inhomogeneous) Markov chains with continuous time and a finite or countable…
In this paper we propose an alternative construction of the self-similar entrance laws for positive self-similar Markov processes. The study of entrance laws has been carried out in previous papers using different techniques, depending on…
The hitting time is the required minimum time for a Markov chain-based walk (classical or quantum) to reach a target state in the state space. We investigate the effect of the perturbation on the hitting time of a quantum walk. We obtain an…
We study the synchronization behavior of discrete-time Markov chains on countable state spaces. Representing a Markov chain in terms of a random dynamical system, which describes the collective dynamics of trajectories driven by the same…
We study a large class of reversible Markov chains with discrete state space and transition matrix $P_N$. We define the notion of a set of {\it metastable points} as a subset of the state space $\G_N$ such that (i) this set is reached from…
Consider a parametric statistical model $P(\mathrm{d}x|\theta)$ and an improper prior distribution $\nu(\mathrm{d}\theta)$ that together yield a (proper) formal posterior distribution $Q(\mathrm{d}\theta|x)$. The prior is called strongly…
Time-limited states characterise many dynamical processes on networks: disease infected individuals recover after some time, people forget news spreading on social networks, or passengers may not wait forever for a connection. These…
In this paper, we consider two time-inhomogeneous Markov chains $X^{(l)}_t$, $l\in\{1,2\}$, with discrete time on a general state space. We assume the existence of some renewal set $C$ and investigate the time of simultaneous renewal, that…
We investigate stationary hidden Markov processes for which mutual information between the past and the future is infinite. It is assumed that the number of observable states is finite and the number of hidden states is countably infinite.…
For a countable-state Markov decision process we introduce an embedding which produces a finite-state Markov decision process. The finite-state embedded process has the same optimal cost, and moreover, it has the same dynamics as the…
We propose a solution to a time-varying variant of Markov Decision Processes which can be used to address decision-theoretic planning problems for autonomous systems operating in unstructured outdoor environments. We explore the time…
We calculate exact convergence times to reach random bipartite entanglement for various random protocols. The eigenproblem of a Markovian chain governing the process is mapped to a spin chain, thereby obtaining exact expression for the gap…
In the design of probabilistic timed systems, bounded requirements concerning behaviour that occurs within a given time, energy, or more generally cost budget are of central importance. Traditionally, such requirements have been…
This paper presents algorithms for identifying and reducing a dedicated set of controllable transition rates of a state-labelled continuous-time Markov chain model. The purpose of the reduction is to make states to satisfy a given…
Let $\{X_n\}$ be a stationary and ergodic time series taking values from a finite or countably infinite set ${\cal X}$. Assume that the distribution of the process is otherwise unknown. We propose a sequence of stopping times $\lambda_n$…