Related papers: Realizable monotonicity for continuous-time Markov…
We study the synchronization behavior of discrete-time Markov chains on countable state spaces. Representing a Markov chain in terms of a random dynamical system, which describes the collective dynamics of trajectories driven by the same…
In the paper we study continuous time controlled Markov processes using discrete time controlled Markov processes. We consider long run functionals: average reward per unit time or long run risk sensitive functional. We also investigate…
In this brief note, we investigate some constructions of Lyapunov functions for stochastic discrete-time stabilizable dynamical systems, in other words, controlled Markov chains. The main question here is whether a Lyapunov function in some…
A discrete-time Markov chain can be transformed into a new Markov chain by looking at its states along iterations of an almost surely finite stopping time. By the optional stopping theorem, any bounded harmonic function with respect to the…
We study stochastic particle systems that conserve the particle density and exhibit a condensation transition due to particle interactions. We restrict our analysis to spatially homogeneous systems on finite lattices with stationary product…
This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies Markov Decision Processes with Incomplete Information and with semi-uniform Feller transition probabilities. The important…
We consider a class of continuous time Markov chains on $\Z^d$. These chains are the discrete space analogue of Markov processes with jumps. Under some conditions, we show that harmonic functions associated with these Markov chains are…
For time-inconsistent stochastic controls in discrete time and finite horizon, an open problem in Bj\"ork and Murgoci (Finance Stoch, 2014) is the existence of an equilibrium control. A nonrandomized Borel measurable Markov equilibrium…
This work focuses on time-inhomogeneous Markov chains with two time scales. Our motivations stem from applications in reliability and dependability, queueing networks, financial engineering and manufacturing systems, where two-time-scale…
This paper focuses on optimizing probabilities of events of interest defined over general controlled discrete-time Markov processes. It is shown that the optimization over a wide class of $\omega$-regular properties can be reduced to the…
In the last years, many authors studied a class of continuous time semi-Markov processes obtained by time-changing Markov processes by hitting times of independent subordinators. Such processes are governed by integro-differential…
In this paper, a link between monotonicity of deterministic dynamical systems and propagation of order by Markov processes is established. The order propagation has received considerable attention in the literature, however, this notion is…
We investigate multivariate regular variation in the context of time-homogeneous Markov chains on general vector spaces and in random coefficient linear models. In the first part, we show that the regular variation of the stationary…
Markov automata combine non-determinism, probabilistic branching, and exponentially distributed delays. This compositional variant of continuous-time Markov decision processes is used in reliability engineering, performance evaluation and…
We revisit closed-loop performance guarantees for Model Predictive Control in the deterministic and stochastic cases, which extend to novel performance results applicable to receding horizon control of Partially Observable Markov Decision…
We discuss problems posed by the quantitative study of time inhomogeneous Markov chains. The two main notions for our purpose are merging and stability. Merging (also called weak ergodicity) occurs when the chain asymptotically forgets…
This paper integrates two strands of the literature on stability of general state Markov chains: conventional, total variation based results and more recent order-theoretic results. First we introduce a complete metric over Borel…
Stochastically monotone Markov chains arise in many applied domains, especially in the setting of queues and storage systems. Poisson's equation is a key tool for analyzing additive functionals of such models, such as cumulative sums of…
Multi-class queueing networks (McQNs) extend the classical concept of Jackson network by allowing jobs of different classes to visit the same server. While such a generalization seems rather natural, from a structural perspective there is a…
As a main step in the numerical solution of control problems in continuous time, the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space. A new feature in this context is to allow…