Related papers: Computing hitting times via fluid approximation: a…
We consider a Markov control model in discrete time with countable both state space and action space. Using the value function of a suitable long-run average reward problem, we study various reachability/controllability problems. First, we…
This paper concerns rollout and certainty-equivalent rollout policies for stochastic shortest path problems with absorbing terminal states. The main result provides a direct non-asymptotic performance certificate for a fixed rollout policy:…
Tipping points characterize situations where a regulated system may experience a sudden and irreversible change and are generally associated with a random state of the system below which the change materializes. In this paper, we study a…
The embedding problem of Markov transition matrices into continuous-time Markov semigroups is a classic problem that regained a lot of impetus and activities in recent years. We consider it here for the following generalisation of the…
We define the hitting time for a model of continuous-time open quantum walks in terms of quantum jumps. Our starting point is a master equation in Lindblad form, which can be taken as the quantum analogue of the rate equation for a…
We consider the safety evaluation of discrete time, stochastic systems over a finite horizon. Therefore, we discuss and link probabilistic invariance with reachability as well as reach-avoid problems. We show how to efficiently compute…
Hybrid systems whose mode dynamics are governed by non-linear ordinary differential equations (ODEs) are often a natural model for biological processes. However such models are difficult to analyze. To address this, we develop a…
We establish sufficient conditions for the existence of moments of the steady-state queue in polling systems operating under the binomial-exhaustive policy (BEP). We assume that the server switches between the different buffers according to…
A classic reachability problem for safety of dynamic systems is to compute the set of initial states from which the state trajectory is guaranteed to stay inside a given constraint set over a given time horizon. In this paper, we leverage…
Many classical randomized algorithms (e.g., approximation algorithms for #P-complete problems) utilize the following random walk algorithm for {\em almost uniform sampling} from a state space $S$ of cardinality $N$: run a symmetric ergodic…
We analyse the problem of meeting times for interdependent stochastic agents: random walkers whose behaviour is stochastic but controlled by their selections from some set of allowed actions, and the inference problem of when these agents…
Motivated by broad applications in reinforcement learning and federated learning, we study local stochastic approximation over a network of agents, where their goal is to find the root of an operator composed of the local operators at the…
We present a numerical method to compute expectations of functionals of a piecewise-deterministic Markov process. We discuss time dependent functionals as well as deterministic time horizon problems. Our approach is based on the…
This paper calculates transient distributions of a special class of Markov processes with continuous state space and in continuous time, up to an explicit error bound. We approximate specific queues on R with one-sided L\'evy input, such as…
We study the computation of lower and upper probabilities of hitting a target set of states for imprecise Markov chains, where transition uncertainty is modelled by a convex set of transition matrices. In the precise case, hitting…
Continuous-time stochastic processes play an important role in the description of random phenomena, it is therefore of prime interest to study particular variables depending on their paths, like stopping time for example. One approach…
We study a classical model for the accumulation of errors in multi-qubit quantum computations. By modeling the error process in a quantum computation using two coupled Markov chains, we are able to capture a weak form of time-dependency…
We consider random walks in which the walk originates in one set of nodes and then continues until it reaches one or more nodes in a target set. The time required for the walk to reach the target set is of interest in understanding the…
A fluid queue is a stochastic process which moves linearly with a rate that is determined by the state of a continuous-time Markov chain (CTMC). In this paper we construct an approximation to a fluid queue using a quasi birth-and-death…
This simple note lays out a few observations which are well known in many ways but may not have been said in quite this way before. The basic idea is that when comparing two different Markov chains it is useful to couple them is such a way…