Related papers: Some simple but challenging Markov processes

Markov Decision Processes (Mdps) form a versatile framework used to model a wide range of optimization problems. The Mdp model consists of sets of states, actions, time steps, rewards, and probability transitions. When in a given state and…

We consider additive functionals of Markov processes in continuous time with general (metric) state spaces. We derive concentration bounds for their exponential moments and moments of finite order. Applications include diffusions,…

We investigate a piecewise-deterministic Markov process, evolving on a Polish metric space, whose deterministic behaviour between random jumps is governed by some semi-flow, and any state right after the jump is attained by a randomly…

We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous action and state spaces. We extend policy graphs to include…

An extension of non-deterministic processes driven by the random telegraph signal is introduced in the framework of "piecewise deterministic Markov processes" [Davis], including a broader category of random systems. The corresponding…

This paper considers large families of Markov chains (MCs) that are defined over a set of parameters with finite discrete domains. Such families occur in software product lines, planning under partial observability, and sketching of…

Probabilistic models help us encode latent structures that both model the data and are ideally also useful for specific downstream tasks. Among these, mixture models and their time-series counterparts, hidden Markov models, identify…

The parameters of a discrete stationary Markov model are transition probabilities between states. Traditionally, data consist in sequences of observed states for a given number of individuals over the whole observation period. In such a…

Semi-Markov processes are Markovian processes in which the firing time of the transitions is modelled by probabilistic distributions over positive reals interpreted as the probability of firing a transition at a certain moment in time. In…

We study properties of a piecewise deterministic Markov process modeling the changes in concentration of specific antibodies. The evolution of densities of the process is described by a stochastic semigroup. The long-time behaviour of this…

In this paper, we first use PDE techniques and probabilistic methods to identify a kind of quasi-continuous random variables. Then we give a characterization of the $G$-integrable processes and get a kind of quasi-continuous processes by…

The work treats systems combining slow and fast motions depending on each other where fast motions are perturbations of families of either dynamical systems or Markov processes with freezed slow variable. In the first case we consider…

We introduce a novel class of generative models based on piecewise deterministic Markov processes (PDMPs), a family of non-diffusive stochastic processes consisting of deterministic motion and random jumps at random times. Similarly to…

This chapter reviews how molecular dynamics simulations, experimental data, and Markov state models can synergize to map-out the mechanism of protein-protein association and dissociation.

The reduction of a continuous Markov process with multiple metastable states to a discrete rate process is investigated in the presence of slow time dependent parameters such as periodic external forces or slowly fluctuating barrier…

These lecture notes present some new concentration inequalities for Feynman-Kac particle processes. We analyze different types of stochastic particle models, including particle profile occupation measures, genealogical tree based evolution…

We consider a class of non-homogeneous Markov chains, that contains many natural examples. Next, using martingale methods, we establish some deviation and moment inequalities for separately Lipschitz functions of such a chain, under moment…

We show that stochastic processes with linear conditional expectations and quadratic conditional variances are Markov, and their transition probabilities are related to a three-parameter family of orthogonal polynomials which generalize the…

Studying systems where many individual bodies in motion interact with one another is a complex and interesting area. Simple mechanisms that may be determined for biological, chemical, or physical reasons can lead to astonishingly complex…

In this paper we consider a control problem for a Partially Observable Piecewise Deterministic Markov Process of the following type: After the jump of the process the controller receives a noisy signal about the state and the aim is to…