Related papers: Model Checking Markov Chains as Distribution Trans…
We consider a simple discrete-time Markov chain with values in $[0,\infty)^{Z^d}$. The Markov chain describes various interesting examples such as oriented percolation, directed polymers in random environment, time discretizations of binary…
We investigate recurrence and transience of Branching Markov Chains (BMC) in discrete time. Branching Markov Chains are clouds of particles which move (according to an irreducible underlying Markov Chain) and produce offspring…
In this paper, we abstract a kind of stochastic processes from evolving processes of growing networks, this process is called growing network Markov chains. Thus the existence and the formulas of degree distribution are transformed to the…
This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies Markov Decision Processes with Incomplete Information and with semi-uniform Feller transition probabilities. The important…
Markov decision processes (MDPs) are a fundamental model for decision making under uncertainty. They exhibit non-deterministic choice as well as probabilistic uncertainty. Traditionally, verification algorithms assume exact knowledge of the…
The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law,…
This article surveys recent advancements of strategy designs for persistent robotic surveillance tasks with the focus on stochastic approaches. The problem describes how mobile robots stochastically patrol a graph in an efficient way where…
Parametric Markov chains occur quite naturally in various applications: they can be used for a conservative analysis of probabilistic systems (no matter how the parameter is chosen, the system works to specification); they can be used to…
We present a system for online probabilistic event forecasting. We assume that a user is interested in detecting and forecasting event patterns, given in the form of regular expressions. Our system can consume streams of events and forecast…
Estimating the transition dynamics of controlled Markov chains is crucial in fields such as time series analysis, reinforcement learning, and system exploration. Traditional non-parametric density estimation methods often assume independent…
Markov decision processes (MDPs) are a popular model for decision-making in the presence of uncertainty. The conventional view of MDPs in verification treats them as state transformers with probabilities defined over sequences of states and…
We consider the discrete-time migration-recombination equation, a deterministic, nonlinear dynamical system that describes the evolution of the genetic type distribution of a population evolving under migration and recombination in a law of…
High-dimensional dynamical systems projected onto a reduced-order model cease to be deterministic and are best described by probability distributions in state space. Their equations of motion map onto an evolution operator with a…
Focusing on stochastic systems arising in mean-field models, the systems under consideration belong to the class of switching diffusions, in which continuous dynamics and discrete events coexist and interact. The discrete events are modeled…
We consider stochastic discrete event dynamic systems that have time evolution represented with two-dimensional state vectors through a vector equation that is linear in terms of an idempotent semiring. The state transitions are governed by…
We study irreducible time-homogenous Markov chains with finite state space in discrete time. We obtain results on the sensitivity of the stationary distribution and other statistical quantities with respect to perturbations of the…
We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains…
From the perspective of probability, the stability of growing network is studied in the present paper. Using the DMS model as an example, we establish a relation between the growing network and Markov process. Based on the concept and…
Verification of infinite-state Markov chains is still a challenge despite several fruitful numerical or statistical approaches. For decisive Markov chains, there is a simple numerical algorithm that frames the reachability probability as…
This paper describes a data reduction technique in case of a markov chain of specified order. Instead of observing all the transitions in a markov chain we record only a few of them and treat the remaining part as missing. The decision…