Related papers: Markov Chains with Rewinding
We consider an opportunistic spectrum access (OSA) problem where the time-varying condition of each channel (e.g., as a result of random fading or certain primary users' activities) is modeled as an arbitrary finite-state Markov chain. At…
Reversibility is a key property of Markov chains, central to algorithms such as Metropolis-Hastings and other MCMC methods. Yet many applications yield non-reversible chains, motivating the problem of approximating them by reversible ones…
Reinforcement learning algorithms assume that observations satisfy the Markov property, yet real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability. Standard performance metrics…
Analyzing, understanding, and describing human behavior is advantageous in different settings, such as web browsing or traffic navigation. Understanding human behavior naturally helps to improve and optimize the underlying infrastructure or…
Markov chains are an important tool for modelling and evaluating systems in computer science, economics, biology and numerous other fields. Thus, approximating Markov chains is a useful tool for decreasing the computational effort needed…
Many applications in networked control require intermittent access of a controller to a system, as in event-triggered systems or information constrained control applications. Motivated by such applications and extending previous work on…
Parametric Markov chains occur quite naturally in various applications: they can be used for a conservative analysis of probabilistic systems (no matter how the parameter is chosen, the system works to specification); they can be used to…
This article describes an accurate procedure for computing the mean first passage times of a finite irreducible Markov chain and a Markov renewal process. The method is a refinement to the Kohlas, Zeit fur Oper Res, 30,197-207, (1986)…
We investigate nonlinear regression for nonstationary sequential data. In most real-life applications such as business domains including finance, retail, energy and economy, timeseries data exhibits nonstationarity due to the temporally…
Markov state modeling has gained popularity in various scientific fields since it reduces complex time-series data sets into transitions between a few states. Yet common Markov state modeling frameworks assume a single Markov chain…
Sequential scaling is a prominent inference-time scaling paradigm, yet its performance improvements are typically modest and not well understood, largely due to the prevalence of heuristic, non-principled approaches that obscure clear…
A key question in brain sciences is how to identify time-evolving functional connectivity, such as that obtained from recordings of neuronal activity over time. We wish to explain the observed phenomena in terms of latent states which, in…
We consider a class of optimal control problems, with finite or infinite horizon, for a continuous-time Markov chain with finite state space. In this case, the control process affects the transition rates. We suppose that the controlled…
Continuous-time Markov chains are mathematical models that are used to describe the state-evolution of dynamical systems under stochastic uncertainty, and have found widespread applications in various fields. In order to make these models…
Motivated by applications of distributed linear estimation, distributed control and distributed optimization, we consider the question of designing linear iterative algorithms for computing the average of numbers in a network. Specifically,…
In many dynamical systems in nature, the law of the dynamics changes along with the temporal evolution of the system. These changes are often associated with the occurrence of certain events. The timing of occurrence of these events…
We consider a unified framework of sequential change-point detection and hypothesis testing modeled by means of hidden Markov chains. One observes a sequence of random variables whose distributions are functionals of a hidden Markov chain.…
Filtering is concerned with the sequential estimation of the state, and uncertainties, of a Markovian system, given noisy observations. It is particularly difficult to achieve accurate filtering in complex dynamical systems, such as those…
We are interested in understanding stability (almost sure boundedness) of stochastic approximation algorithms (SAs) driven by a `controlled Markov' process. Analyzing this class of algorithms is important, since many reinforcement learning…
This paper presents algorithms for identifying and reducing a dedicated set of controllable transition rates of a state-labelled continuous-time Markov chain model. The purpose of the reduction is to make states to satisfy a given…