Related papers: A Compositional Framework for Markov Processes

Markov Decision Processes (MDPs) have been used to formulate many decision-making problems in science and engineering. The objective is to synthesize the best decision (action selection) policies to maximize expected rewards (or minimize…

Given a strongly stationary Markov chain and a finite set of stopping rules, we prove the existence of a polynomial algorithm which projects the Markov chain onto a minimal Markov chain without redundant information. Markov complexity is…

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as…

We introduce discrete time Markov chains that preserve uniform measures on boxed plane partitions. Elementary Markov steps change the size of the box from (a x b x c) to ((a-1) x (b+1) x c) or ((a+1) x (b-1) x c). Algorithmic realization of…

We develop a framework for the compression of reversible Markov chains with rigorous error control. Given a subset of selected states, we construct reduced dynamics that can be lifted to an approximation of the full dynamics, and we prove…

The fluctuation-dissipation theorem is a central result in statistical mechanics and is usually formulated for systems described by diffusion processes. In this paper, we propose a generalization for a wider class of stochastic processes,…

We study the large deviations principle for one dimensional, continuous, homogeneous, strong Markov processes that do not necessarily behave locally as a Wiener process. Any strong Markov process $X_{t}$ in $\mathbb{R}$ that is continuous…

Probabilistic generative models based on measure transport, such as diffusion and flow-based models, are often formulated in the language of Markovian stochastic dynamics, where the choice of the underlying process impacts both algorithmic…

A general formalism is developed to construct a Markov chain model that converges to a one-dimensional map in the infinite population limit. Stochastic fluctuations are therefore internal to the system and not externally specified. For…

Starting at the mesoscopic level with a general formulation of stochastic thermodynamics in terms of Markov jump processes, we identify the scaling conditions that ensure the emergence of a (typically nonlinear) deterministic dynamics and…

We study the multiple definitions of the entropy production for discrete-time Markov processes in single systems and composite systems. These definitions have been studied in single systems, but less so in composite systems. With a clear…

We consider a robust approach to address uncertainty in model parameters in Markov Decision Processes (MDPs), which are widely used to model dynamic optimization in many applications. Most prior works consider the case where the uncertainty…

Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not --…

We classify the rare events of structured, memoryful stochastic processes and use this to analyze sequential and parallel generators for these events. Given a stochastic process, we introduce a method to construct a new process whose…

The fluctuation theorem for entropy production is a remarkable symmetry of the distribution of produced entropy that holds universally in non-equilibrium steady states with Markovian dynamics. However, in systems with slow degrees of…

We investigate the classical active pure exploration problem in Markov Decision Processes, where the agent sequentially selects actions and, from the resulting system trajectory, aims at identifying the best policy as fast as possible. We…

We consider the problem of conditioning a Markov process on a rare event and of representing this conditioned process by a conditioning-free process, called the effective or driven process. The basic assumption is that the rare event used…

Suppose that $(X_t)_{t \ge 0}$ is a one-dimensional Brownian motion with negative drift $-\mu$. It is possible to make sense of conditioning this process to be in the state $0$ at an independent exponential random time and if we kill the…

In this paper, we propose a compositional approach for the construction of finite abstractions (a.k.a. finite Markov decision processes (MDPs)) for networks of discrete-time stochastic control subsystems that are not necessarily…

We study time-changed Markov processes to speed up the convergence of Markov chain Monte Carlo (MCMC) algorithms. The time-changed process is defined by adjusting the speed of time of a base process via a user-chosen, state-dependent…