Related papers: Realizable monotonicity for continuous-time Markov…
For a discrete time Markov chain and in line with Strotz' consistent planning we develop a framework for problems of optimal stopping that are time-inconsistent due to the consideration of a non-linear function of an expected reward. We…
We consider a Markov control model in discrete time with countable both state space and action space. Using the value function of a suitable long-run average reward problem, we study various reachability/controllability problems. First, we…
The notion of a successful coupling of Markov processes, based on the idea that both components of the coupled system ``intersect'' in finite time with probability one, is extended to cover situations when the coupling is unnecessarily…
Reversal of the time direction in stochastic systems driven by white noise has been central throughout the development of stochastic realization theory, filtering and smoothing. Similar ideas were developed in connection with certain…
Extensions of Kemeny's constant, as derived for irreducible finite Markov chains in discrete time, to Markov renewal processes and Markov chains in continuous time are discussed. Three alternative Kemeny's functions and their variants are…
We address stability of a class of Markovian discrete-time stochastic hybrid systems. This class of systems is characterized by the state-space of the system being partitioned into a safe or target set and its exterior, and the dynamics of…
In order to approximate a continuous time stochastic process by discrete time Markov chains one has several options to embed the Markov chains into continuous time processes. On the one hand there is the Markov embedding, which uses…
The problem of appropriately matching items subject to compatibility constraints arises in a number of important applications. While most of the literature on matching theory focuses on a static setting with a fixed number of items, several…
We propose a solution to a time-varying variant of Markov Decision Processes which can be used to address decision-theoretic planning problems for autonomous systems operating in unstructured outdoor environments. We explore the time…
Stochastic models such as Continuous-Time Markov Chains (CTMC) and Stochastic Hybrid Automata (SHA) are powerful formalisms to model and to reason about the dynamics of biological systems, due to their ability to capture the stochasticity…
Under continuity and recurrence assumptions, we prove that the iteration of successive partial symmetrizations that form a time-homogeneous Markov process, converges to a symmetrization. We cover several settings, including the…
We suggest an approach to obtaining general two-sided bounds on the rate of convergence in terms of special "weighted" norms related to total variation. Some important classes of continuous-time Markov chains are considered:…
We study the convergence of random function iterations for finding an invariant measure of the corresponding Markov operator. We call the problem of finding such an invariant measure the stochastic fixed point problem. This generalizes…
We show that for several variations of partially observable Markov decision processes, polynomial-time algorithms for finding control policies are unlikely to or simply don't have guarantees of finding policies within a constant factor or a…
Stochastic approximation is a framework unifying many random iterative algorithms occurring in a diverse range of applications. The stability of the process is often difficult to verify in practical applications and the process may even be…
This paper generalizes the notion of stochastic order to a relation between probability measures over arbitrary measurable spaces. This generalization is motivated by the observation that for the stochastic ordering of two stationary Markov…
We consider sequences-indexed by time (discrete stages)-of families of multistage stochastic optimization problems. At each time, the optimization problems in a family are parameterized by some quantities (initial states, constraint…
We consider a class of stochastic dynamical systems, called piecewise deterministic Markov processes, with states $(x, \s)\in \O\times \G$, $\O$ being a region in $\bbR^d$ or the $d$--dimensional torus, $\G$ being a finite set. The…
We find that Markov chains with finite state space are Poincare chaotic. Moreover, finite realizations of the chains are arcs of each unpredictable orbit for sure. An illustrating example with a proper numerical simulation is provided.
Let X be a continuous-time Markov chain in a finite set I, let h be a mapping of I onto another set, and let Y be defined by Y_t=h(X_t), (for t nonnegative). We address the filtering problem for X in terms of the observation Y, which is not…