Related papers: Piecewise Deterministic Markov Processes and their…

Learning-based approaches to verifying unknown Markov decision processes (MDPs) often employ uncertain MDPs. These models use, for example, confidence intervals to capture transition uncertainty and allow synthesis of policies that are…

We develop a convergent variational perturbation theory for conditional probability densities of Markov processes. The power of the theory is illustrated by applying it to the diffusion of a particle in an anharmonic potential.

We propose a numerical method to approximate the value function for the optimal stopping problem of a piecewise deterministic Markov process (PDMP). Our approach is based on quantization of the post jump location---inter-arrival time Markov…

We study invariant boundary conditions for one dimensional discrete Gaussian Markov processes, basic toy models of spatial Markov processes in statistical mechanics. More precisely, we give a decomposition of boundary objects in a non…

We consider a class of piecewise-deterministic Markov processes where the state evolves according to a linear dynamical system. This continuous time evolution is interspersed by discrete events that occur at random times and change (reset)…

We identify the linear space spanned by the real-valued excessive functions of a Markov process with the set of those functions which are quasimartingales when we compose them with the process. Applications to semi-Dirichlet forms are…

Empirical processes for stationary, causal sequences are considered. We establish empirical central limit theorems for classes of indicators of left half lines, absolutely continuous functions and piecewise differentiable functions. Sample…

This paper studies parametric Markov decision processes (pMDPs), an extension to Markov decision processes (MDPs) where transitions probabilities are described by polynomials over a finite set of parameters. Fixing values for all parameters…

In this paper, we analyse piecewise deterministic Markov processes, as introduced in Davis (1984). Many models in insurance mathematics can be formulated in terms of the general concept of piecewise deterministic Markov processes. In this…

Consider a one-sided Markov additive process with an upper and a lower barrier, where each can be either reflecting or terminating. For both defective and non-defective processes and all possible scenarios we identify the corresponding…

We provide a systematic study of the notion of duality of Markov processes with respect to a function. We discuss the relation of this notion with duality with respect to a measure as studied in Markov process theory and potential theory…

This paper is concerned with the general theme of relating the Large Deviation Principle (LDP) for the invariant measures of stochastic processes to the associated sample path LDP. It is shown that if the sample path deviation function…

In this paper, the development of a mathematical method is presented to explore spatially non-uniform phases with no long-range order in mathematical models of first order phase transitions. We use essential results regarding the…

Partially observable Markov decision processes (POMDPs) provide an elegant mathematical framework for modeling complex decision and planning problems in stochastic domains in which states of the system are observable only indirectly, via a…

We consider a class of semi-Markov processes (SMP) such that the embedded discrete time Markov chain may be non-homogeneous. The corresponding augmented processes are represented as semi-martingales using stochastic integral equation…

In this paper, we study a class of quasi-invariant measures on paths generated by discrete dynamical systems. Our main result characterizes the subfamily of these measures which admit a certain desintegration. This is a desintegration with…

Markov chains and Markov decision processes (MDPs) are well-established probabilistic models. While finite Markov models are well-understood, analysing their infinite counterparts remains a significant challenge. Decisiveness has proven to…

We establish convergence to an invariant measure as time tends to infinity, for a large class of (possibly non-Markovian) stochastic volatility models. Our arguments are based on a novel coupling idea for Markov chains which also extends to…

The article studies segmentation problem (also known as classification problem) with pairwise Markov models (PMMs). A PMM is a process where the observation process and underlying state sequence form a two-dimensional Markov chain, it is a…

There is much interest in using partially observable Markov decision processes (POMDPs) as a formal model for planning in stochastic domains. This paper is concerned with finding optimal policies for POMDPs. We propose several improvements…