Related papers: Computing hitting times via fluid approximation: a…
The model consists of a signal process $X$ which is a general Brownian diffusion process and an observation process $Y$, also a diffusion process, which is supposed to be correlated to the signal process. We suppose that the process $Y$ is…
We develop a new bidirectional algorithm for estimating Markov chain multi-step transition probabilities: given a Markov chain, we want to estimate the probability of hitting a given target state in $\ell$ steps after starting from a given…
In this paper we propose a compositional framework for the construction of approximations of the interconnection of a class of stochastic hybrid systems. As special cases, this class of systems includes both jump linear stochastic systems…
In the present paper, we investigate the relationship between hitting times and hitting probabilities in discrete-time imprecise Markov chains (IMCs). We define lower and upper hitting times and probabilities for IMCs whose set of…
Given a large connected graph $G=(V,E)$, and two vertices $w,\neq v$, let $T_{w,v}$ be the first hitting time to $v$ starting from $w$ for the simple random walk on $G$. We prove a general theorem that guarantees, under some assumptions on…
In this paper we propose two behavioral distances that support approximate reasoning on Stochastic Markov Models (SMMs), that are continuous-time stochastic transition systems where the residence time on each state is described by a generic…
We set out a general procedure which allows the approximation of certain Markov chains by the solutions of differential equations. The chains considered have some components which oscillate rapidly and randomly, while others are close to…
This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which…
We study and develop the stochastic Markov reward model (sMRM), which extends the Markov chain where transition time/reward as modelled as random variables. Techniques are presented to enable computing first-passage time distributions (or…
The problem of sampling from the stationary distribution of a Markov chain finds widespread applications in a variety of fields. The time required for a Markov chain to converge to its stationary distribution is known as the classical…
Generating accurate runtime safety estimates for autonomous systems is vital to ensuring their continued proliferation. However, exhaustive reasoning about future behaviors is generally too complex to do at runtime. To provide scalable and…
We propose an improved algorithm for counting the number of Hamiltonian cycles in a directed graph. The basic idea of the method is sequential acceptance/rejection, which is successfully used in approximating the number of perfect matchings…
In the presence of quantum measurements with direct photon detection the evolution of open quantum systems is usually described by stochastic master equations with jumps. Heuristically, from these equations one can obtain diffusion models…
In this paper, we provide a methodology for computing the probability distribution of sojourn times for a wide class of Markov chains. Our methodology consists in writing out linear systems and matrix equations for generating functions…
We study a Q learning algorithm for continuous time stochastic control problems. The proposed algorithm uses the sampled state process by discretizing the state and control action spaces under piece-wise constant control processes. We show…
We establish the consistency of a local time approximation of a diffusion at a sticky threshold based on high-frequency observations. First, we prove the result for sticky Brownian motion, and then extend it to It\^o diffusions with a…
We study the computational complexity of approximating general constrained Markov decision processes. Our primary contribution is the design of a polynomial time $(0,\epsilon)$-additive bicriteria approximation algorithm for finding optimal…
This paper establishes limit theorems for a class of stochastic hybrid systems (continuous deterministic dynamic coupled with jump Markov processes) in the fluid limit (small jumps at high frequency), thus extending known results for jump…
In this paper, we develop approximate dynamic programming methods for stochastic systems modeled as Markov Decision Processes, given both soft performance criteria and hard constraints in a class of probabilistic temporal logic called…
Given n elements with nonnegative integer weights w1,..., wn and an integer capacity C, we consider the counting version of the classic knapsack problem: find the number of distinct subsets whose weights add up to at most the given…