Related papers: Asymptotically linear iterated function systems on…

We review a recent development at the interface between discrete mathematics on one hand and probability theory and statistics on the other, specifically the use of Markov chains and their boundary theory in connection with the asymptotics…

We consider infinite-horizon $\gamma$-discounted Markov Decision Processes, for which it is known that there exists a stationary optimal policy. We consider the algorithm Value Iteration and the sequence of policies $\pi_1,...,\pi_k$ it…

We consider additive functionals of stationary Markov processes and show that under Kipnis-Varadhan type conditions they converge in rough path topology to a Stratonovich Brownian motion, with a correction to the Levy area that can be…

The distribution of the "mixing time" or the "time to stationarity" in a discrete time irreducible Markov chain, starting in state i, can be defined as the number of trials to reach a state sampled from the stationary distribution of the…

In the analysis of Markov chains and processes, it is sometimes convenient to replace an unbounded state space with a "truncated" bounded state space. When such a replacement is made, one often wants to know whether the equilibrium behavior…

For the iterations of $x\mapsto |x-\theta|$ random functions with Lipschitz number one, we represent the dynamics as a Markov chain and prove its convergence under mild conditions. We also demonstrate that the Wasserstein metric of any two…

We consider the Markov renewal equation $F(t) = f(t) + \boldsymbol{\mu}*F(t)$ for vector-valued functions $f,F: \mathbb{R} \to \mathbb{R}^{p}$ and a $p \times p$ matrix $\boldsymbol{\mu}$ of locally finite measures $\mu^{i,j}$ on…

We derive a finite-sample probabilistic bound on the parameter estimation error of a system identification algorithm for Linear Switched Systems. The algorithm estimates Markov parameters from a single trajectory and applies a variant of…

Suppose that $X=(X_{t})_{t\ge 0}$ is either a general supercritical non-local branching Markov process, or a general supercritical non-local superprocess, on a Luzin space. Here, by ``supercritical" we mean that the mean semigroup of $X$…

This paper is a survey of various proofs of the so called {\em fundamental theorem of Markov chains}: every ergodic Markov chain has a unique positive stationary distribution and the chain attains this distribution in the limit independent…

The linear response of a dynamical system refers to changes to properties of the system when small external perturbations are applied. We consider the little-studied question of selecting an optimal perturbation so as to (i) maximise the…

We consider a class of stochastic dynamical systems, called piecewise deterministic Markov processes, with states $(x, \s)\in \O\times \G$, $\O$ being a region in $\bbR^d$ or the $d$--dimensional torus, $\G$ being a finite set. The…

This work studies discrete-time discounted Markov decision processes with continuous state and action spaces and addresses the inverse problem of inferring a cost function from observed optimal behavior. We first consider the case in which…

Reinforcement Learning Algorithms are predominantly developed for stationary environments, and the limited literature that considers nonstationary environments often involves specific assumptions about changes that can occur in transition…

We introduce a Markov product structure for multivariate tail dependence functions, building upon the well-known Markov product for copulas. We investigate algebraic and monotonicity properties of this new product as well as its role in…

Order-preserving couplings are elegant tools for obtaining robust estimates of the time-dependent and stationary distributions of Markov processes that are too complex to be analyzed exactly. The starting point of this paper is to study…

We study dynamical reversibility in stationary stochastic processes from an information theoretic perspective. Extending earlier work on the reversibility of Markov chains, we focus on finitary processes with arbitrarily long conditional…

Many complex systems exhibit interactions that depend not only on pairwise connections, but also group structures and memory effects. To capture such effects, we develop a unified tensor framework for modeling higher-order Markov chains…

An autoregressive process with Markov regime is an autoregressive process for which the regression function at each time point is given by a nonobservable Markov chain. In this paper we consider the asymptotic properties of the maximum…

In this paper we study various properties of finite stochastic systems or hidden Markov chains as they are alternatively called. We discuss their construction following different approaches and we also derive recursive filtering formulas…