Related papers: Markov Chains with Rewinding

We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1} = F(\Phi_k, \alpha(\Phi_k, U_{k+1}))$ where the function $F$ is $C^1$ while $\alpha$ is typically discontinuous and $\{U_k: k \in…

We present a new statistical test to detect that a presented state of a reversible Markov chain was not chosen from a stationary distribution. In particular, given a value function for the states of the Markov chain, we would like to…

Motivated by the study of the time evolution of random dynamical systems arising in a vast variety of domains --- ranging from physics to ecology ---, we establish conditions for the occurrence of a non-trivial asymptotic behaviour for…

We derive some key extremal features for $k$th order Markov chains that can be used to understand how the process moves between an extreme state and the body of the process. The chains are studied given that there is an exceedance of a…

We consider a finite state discrete time process X. Without loss of generality the finite state space can be identified with the set of unit vectors {e1, e2, . . . , eN} with ei = (0, . . . , 0, 1, 0, . . . , 0)0 2 RN. For a Markov chain…

Adaptive importance sampling is a powerful tool to sample from complicated target densities, but its success depends sensitively on the initial proposal density. An algorithm is presented to automatically perform the initialization using…

We develop a practical approach to establish the stability, that is, the recurrence in a given set, of a large class of controlled Markov chains. These processes arise in various areas of applied science and encompass important numerical…

Consider a sequence (indexed by n) of Markov chains Z^n in R^d characterized by transition kernels that approximately (in n) depend only on the rescaled state n^{-1} Z^n. Subject to a smoothness condition, such a family can be closely…

The assumption that data samples are independently identically distributed is the backbone of many learning algorithms. Nevertheless, datasets often exhibit rich structure in practice, and we argue that there exist some unknown order within…

We study the performance of a stochastic algorithm based on the power method that adaptively learns the large deviation functions characterizing the fluctuations of additive functionals of Markov processes, used in physics to model…

In this paper, we propose a class of efficient, accurate, and general methods for solving state-estimation problems with equality and inequality constraints. The methods are based on recent developments in variable splitting and partially…

Predictability of behavior has emerged an an important characteristic in many fields including biology, medicine, and marketing. Behavior can be recorded as a sequence of actions performed by an individual over a given time period. This…

A fundamental problem when aggregating Markov chains is the specification of the number of state groups. Too few state groups may fail to sufficiently capture the pertinent dynamics of the original, high-order Markov chain. Too many state…

This paper studies the estimation of low-rank Markov chains from empirical trajectories. We propose a non-convex estimator based on rank-constrained likelihood maximization. Statistical upper bounds are provided for the Kullback-Leiber…

Many systems are partially stochastic in nature. We have derived data driven approaches for extracting stochastic state machines (Markov models) directly from observed data. This chapter provides an overview of our approach with numerous…

Markov decision processes are typically used for sequential decision making under uncertainty. For many aspects however, ranging from constrained or safe specifications to various kinds of temporal (non-Markovian) dependencies in task and…

Markov chains for probability distributions related to matrix product states and 1D Hamiltonians are introduced. With appropriate 'inverse temperature' schedules, these chains can be combined into a random approximation scheme for ground…

I describe a new Markov chain method for sampling from the distribution of the state sequences in a non-linear state space model, given the observation sequence. This method updates all states in the sequence simultaneously using an…

Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are fixed, that is, the underlying Markov decision process is stationary. However, in many real-world applications, this…

The deterministic dynamics of randomly connected neural networks are studied, where a state of binary neurons evolves according to a discreet-time synchronous update rule. We give a theoretical support that the overlap of systems' states…