Related papers: Offline Estimation of Controlled Markov Chains: Mi…

Finite state space hidden Markov models are flexible tools to model phenomena with complex time dependencies: any process distribution can be approximated by a hidden Markov model with enough hidden states.We consider the problem of…

We study the efficient off-policy evaluation of natural stochastic policies, which are defined in terms of deviations from the behavior policy. This is a departure from the literature on off-policy evaluation where most work consider the…

Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline datasets, the principle…

We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an…

The mixing time of an ergodic, reversible Markov chain can be bounded in terms of the eigenvalues of the chain: specifically, the second-largest eigenvalue and the smallest eigenvalue. It has become standard to focus only on the…

Real-world autonomous systems operate under uncertainty about both their pose and dynamics. Autonomous control systems must simultaneously perform estimation and control tasks to maintain robustness to changing dynamics or modeling errors.…

The concepts of probability, statistics and stochastic theory are being successfully used in structural engineering. Markov Chain modelling is a simple stochastic process model that has found its application in both describing stochastic…

Mixing of finite time-homogeneous Markov chains is well understood nowadays, with a rich set of techniques to estimate their mixing time. In this paper, we study the mixing time of random walks in dynamic random environments. To that end,…

In this paper, we propose a variance reduction approach for Markov chains based on additive control variates and the minimization of an appropriate estimate for the asymptotic variance. We focus on the particular case when control variates…

Majority-vote ensembles achieve variance reduction by averaging over diverse, approximately independent base learners. When training data exhibits Markov dependence, as in time-series forecasting, reinforcement learning (RL) replay buffers,…

Through a Metropolis-like algorithm with single step computational cost of order one, we build a Markov chain that relaxes to the canonical Fermi statistics for k non-interacting particles among m energy levels. Uniformly over the…

To quantify the randomness of Markov trajectories with fixed initial and final states, Ekroot and Cover proposed a closed-form expression for the entropy of trajectories of an irreducible finite state Markov chain. Numerous applications,…

Performing numerical integration when the integrand itself cannot be evaluated point-wise is a challenging task that arises in statistical analysis, notably in Bayesian inference for models with intractable likelihood functions. Markov…

Dynamic and evolving operational and economic environments present significant challenges for decision-making. We explore a simulation optimization problem characterized by non-stationary input distributions with regime-switching dynamics…

We consider synthesis of control policies that maximize the probability of satisfying given temporal logic specifications in unknown, stochastic environments. We model the interaction between the system and its environment as a Markov…

A divide-and-conquer approach to analyzing Markov chains (MCs) is not utilized as widely as it could be, despite its potential benefits. One primary reason for this is the fact that most MC decomposition approaches involve a complex and…

We study the problem of clustering $T$ trajectories of length $H$, each generated by one of K unknown ergodic Markov chains over a finite state space of size $S$. We derive an instance-dependent, high-probability lower bound on the…

We give computable bounds on the rate of convergence of the transition probabilities to the stationary distribution for a certain class of geometrically ergodic Markov chains. Our results are different from earlier estimates of Meyn and…

Motivated by applications arising in networked systems, this work examines controlled regime-switching systems that stem from a mean-variance formulation. A main point is that the switching process is a hidden Markov chain. An additional…

This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data coverage. However,…