Related papers: Persistent-Transient Policy Evaluation for Markov …
We introduce a unified operator-theoretic framework for analyzing mixing times of finite-state ergodic Markov chains that applies to both reversible and non-reversible dynamics. The central object in our analysis is the projected transition…
Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not --…
We consider nonparametric estimation of the transition operator $P$ of a Markov chain and its transition density $p$ where the singular values of $P$ are assumed to decay exponentially fast. This is for instance the case for periodised,…
Experience replay is a core ingredient of modern deep reinforcement learning, yet its benefits in policy optimization are poorly understood beyond empirical heuristics. This paper develops a novel theoretical framework for experience replay…
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of policy gradients from…
A wide variety of physical systems ranging from the firing of neurons to eutrophication of lakes to the presence of Arctic summer sea ice exhibit a phenomenon known as tipping. In mathematical models, tipping can be caused by bifurcations,…
This paper is concerned with the development of rigorous approximations to various expectations associated with Markov chains and processes having non-stationary transition probabilities. Such non-stationary models arise naturally in…
In this paper, we study a mean-variance optimization problem in an infinite horizon discrete time discounted Markov decision process (MDP). The objective is to minimize the variance of system rewards with the constraint of mean performance.…
We provide a novel method for sensitivity analysis of parametric robust Markov chains. These models incorporate parameters and sets of probability distributions to alleviate the often unrealistic assumption that precise probabilities are…
We study random walks on contingency tables with fixed marginals, corresponding to a (log-linear) hierarchical model. If the set of allowed moves is not a Markov basis, then there exist tables with the same marginals that are not connected.…
Accurate characterization of coherent and non-Markovian errors remains a central challenge in quantum information processing, as conventional benchmarking techniques typically rely on Markovian and time-independent noise assumptions. In…
We propose a new approach for estimating the finite dimensional transition matrix of a Markov chain using a large number of independent sample paths observed at random times. The sample paths may be observed as few as two times, and the…
Markov-modulated Brownian motion is a popular tool to model continuous-time phenomena in a stochastic context. The main quantity of interest is the invariant density, which satisfies a differential equation associated with the quadratic…
The spectral gap of a Markov chain can be bounded by the spectral gaps of constituent "restriction" chains and a "projection" chain, and the strength of such a bound is the content of various decomposition theorems. In this paper, we…
Motivated by applications arising in networked systems, this work examines controlled regime-switching systems that stem from a mean-variance formulation. A main point is that the switching process is a hidden Markov chain. An additional…
We present a coherent approach to recurrence and transience, starting from a version of the Riesz decomposition theorem for superharmonic elements. Our approach allows straightforward proofs of some known results, entails new theorems, and…
We consider Markov Decision Processes (MDPs) in which every stationary policy induces the same graph structure for the underlying Markov chain and further, the graph has the following property: if we replace each recurrent class by a node,…
We study finite horizon optimal switching problems for hidden Markov chain models under partially observable Poisson processes. The controller possesses a finite range of strategies and attempts to track the state of the unobserved state…
Consider a sequence $(\eta^N(t) :t\ge 0)$ of continuous-time, irreducible Markov chains evolving on a fixed finite set $E$, indexed by a parameter $N$. Denote by $R_N(\eta,\xi)$ the jump rates of the Markov chain $\eta^N_t$, and assume that…
This paper investigates recursive feasibility, recursive robust stability and near-optimality properties of policy iteration (PI). For this purpose, we consider deterministic nonlinear discrete-time systems whose inputs are generated by PI…