Related papers: Some simple but challenging Markov processes
This paper presents two new approaches to decomposing and solving large Markov decision problems (MDPs), a partial decoupling method and a complete decoupling method. In these approaches, a large, stochastic decision problem is divided into…
In the paper we study continuous time controlled Markov processes using discrete time controlled Markov processes. We consider long run functionals: average reward per unit time or long run risk sensitive functional. We also investigate…
Planning under uncertainty is critical to robotics. The Partially Observable Markov Decision Process (POMDP) is a mathematical framework for such planning problems. It is powerful due to its careful quantification of the non-deterministic…
The article studies segmentation problem (also known as classification problem) with pairwise Markov models (PMMs). A PMM is a process where the observation process and underlying state sequence form a two-dimensional Markov chain, it is a…
We treat the class of universal Markov processes on the d-dimensional Euklidean space which do not depend on random. For these, as well as for several subclasses, we prove criteria whether a function f, defined on the positive half-line,…
Multistate Markov models are a canonical parametric approach for data modeling of observed or latent stochastic processes supported on a finite state space. Continuous-time Markov processes describe data that are observed irregularly over…
We are studying stationary random processes with conditional polynomial moments that allow a continuous path modification. Processes with continuous path modification, are important because they are relatively easy to simulate. One does not…
We provide a systematic study of the notion of duality of Markov processes with respect to a function. We discuss the relation of this notion with duality with respect to a measure as studied in Markov process theory and potential theory…
It is shown how simple assumptions lead to piecewise linear behavior, which is observed in certain phase transitions.
We present limit theorems for a sequence of Piecewise Deterministic Markov Processes (PDMPs) taking values in a separable Hilbert space. This class of processes provides a rigorous framework for stochastic spatial models in which discrete…
Many biological and medical questions can be modeled using time-to-event data in finite-state Markov chains, with the phase-type distribution describing intervals between events. We solve the inverse problem: given a phase-type…
Necessary and sufficient conditions are given for a substochastic semigroup on $L^1$ obtained through the Kato--Voigt perturbation theorem to be either stochastic or strongly stable. We show how such semigroups are related to piecewise…
In this paper, we study consistent and partially exchangeable sequences of Markov chains on a finite state space. We provide a characterisation of the admissible transition rates via a decomposition into individual and coordinated motion of…
Order-preserving couplings are elegant tools for obtaining robust estimates of the time-dependent and stationary distributions of Markov processes that are too complex to be analyzed exactly. The starting point of this paper is to study…
Markov chains and Markov decision processes (MDPs) are well-established probabilistic models. While finite Markov models are well-understood, analysing their infinite counterparts remains a significant challenge. Decisiveness has proven to…
A simple construction is presented, which generalises piecewise linear one-dimensional Markov maps to an arbitrary number of dimensions. The corresponding coupled map lattice, known as a simplicial mapping in the mathematical literature,…
Markov processes are widely used in modeling random phenomena/problems. However, they may not be adequate in some cases where more general processes are needed. The conditionally Markov (CM) process is a generalization of the Markov process…
A wide class of ``counting'' problems have been studied in Computer Science. Three typical examples are the estimation of - (i) the permanent of an $n\times n$ 0-1 matrix, (ii) the partition function of certain $n-$ particle Statistical…
The concepts of probability, statistics and stochastic theory are being successfully used in structural engineering. Markov Chain modelling is a simple stochastic process model that has found its application in both describing stochastic…
A brief presentation of basics of the theory of Tchebycheff and Markov systems of functions and its applications to extremal problems for integrals of such functions is given. The results, as well as all the necessary definitions, are…