Related papers: Rate Reduction for State-labelled Markov Chains wi…
Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters…
We revisit the source coding problem for a Markov chain under the assumption that the transmission times and how fast the Markov chain transitions its state happen at the same time-scale. Specifically, we assume that the transmission of…
This paper investigates sensor scheduling for state estimation of complex networks over shared transmission channels. For a complex network of dynamical systems, referred to as nodes, a sensor network is adopted to measure and estimate the…
In this paper, we present a novel iterative Monte Carlo method for approximating the stationary probability of a single state of a positive recurrent Markov chain. We utilize the characterization that the stationary probability of a state…
Rate processes are simple and analytically tractable models for many dynamical systems which switch stochastically between a discrete set of quasi stationary states but they may also approximate continuous processes by coarse grained,…
Switched systems are capable of modeling processes with underlying dynamics that may change abruptly over time. To achieve accurate modeling in practice, one may need a large number of modes, but this may in turn increase the model…
We consider the constrained optimal control problem for the gradual-impulsive CTMDP model with the performance criteria being the expected total undiscounted costs (from the running cost and the cost from each time an impulse being…
Time series of conformational dynamics in proteins are usually evaluated with hidden Markov models (HMMs). This approach works well if the number of states and their connectivity is known. However, for the multi-domain protein Hsp90, a…
We consider stochastic descriptions of chemical reaction networks in which there are both fast and slow reactions, and for which the time scales are widely separated. We develop a computational algorithm that produces the generator of the…
In this work, we study a natural nonparametric estimator of the transition probability matrices of a finite controlled Markov chain. We consider an offline setting with a fixed dataset, collected using a so-called logging policy. We develop…
Probabilistic Computation Tree Logic (PCTL) is frequently used to formally specify control objectives such as probabilistic reachability and safety. In this work, we focus on model checking PCTL specifications statistically on Markov…
We study a variation of vanilla stochastic gradient descent where the optimizer only has access to a Markovian sampling scheme. These schemes encompass applications that range from decentralized optimization with a random walker (token…
We study stochastic optimization algorithms for constrained nonconvex stochastic optimization problems with Markovian data. In particular, we focus on the case when the transition kernel of the Markov chain is state-dependent. Such…
Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical…
We provide a framework for speeding up algorithms for time-bounded reachability analysis of continuous-time Markov decision processes. The principle is to find a small, but almost equivalent subsystem of the original system and only analyse…
This paper proposes a state reduction method for learning-based model predictive control (MPC) for train rescheduling in urban rail transit systems. The state reduction integrates into a control framework where the discrete decision…
We give a Markov chain based perfect sampler for uniform sampling solutions of constraint satisfaction problems (CSP). Under some mild Lov\'asz local lemma conditions where each constraint of the CSP has a small number of forbidden local…
Motivated by applications in telecommunications, computer scienceand physics, we consider a discrete-time Markov process withrestart. At each step the process eitherwith a positive probability restarts from a given distribution, orwith the…
Partially observable Markov decision processes (POMDPs) provide a modeling framework for autonomous decision making under uncertainty and imperfect sensing, e.g. robot manipulation and self-driving cars. However, optimal control of POMDPs…
We describe an exact approach for calculating transition probabilities and waiting times in finite-state discrete-time Markov processes. All the states and the rules for transitions between them must be known in advance. We can then…