Related papers: Discrete-time, discrete-state multistate Markov mo…

In this work we introduce new approximate similarity relations that are shown to be key for policy (or control) synthesis over general Markov decision processes. The models of interest are discrete-time Markov decision processes, endowed…

Finite state space hidden Markov models are flexible tools to model phenomena with complex time dependencies: any process distribution can be approximated by a hidden Markov model with enough hidden states.We consider the problem of…

Markov models are often used to capture the temporal patterns of sequential data for statistical learning applications. While the Hidden Markov modeling-based learning mechanisms are well studied in literature, we analyze a…

Asymptotic properties of Markov Processes, such as steady state probabilities or hazard rate for absorbing states can be efficiently calculated by means of linear algebra even for large-scale problems. This paper discusses the methods for…

Continuous-time Markov process models of contagions are widely studied, not least because of their utility in predicting the evolution of real-world contagions and in formulating control measures. It is often the case, however, that…

Multistate models (MSM) are well developed for continuous and discrete times under a first order Markov assumption. Motivated by a cohort of COVID-19 patients, an MSM was designed based on 14 transitions among 7 states of a patient. Since a…

Continuous-time multistate models are widely used for analyzing interval-censored data on disease progression over time. Sometimes, diseases manifest differently and what appears to be a coherent collection of symptoms is the expression of…

We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous action and state spaces. We extend policy graphs to include…

State space models have long played an important role in signal processing. The Gaussian case can be treated algorithmically using the famous Kalman filter. Similarly since the 1970s there has been extensive application of Hidden Markov…

Transition Path Theory (TPT) provides a rigorous statistical characterization of the ensemble of trajectories connecting directly, i.e., without detours, two disconnected (sets of) states in a Markov chain, a stochastic process that…

The study of transformations among pure states via Local Operations assisted by Classical Communication (LOCC) plays a central role in entanglement theory. The main emphasis of these investigations is on the deterministic, or probabilistic…

By introducing extrinsic noise as well as intrinsic uncertainty into a network with stochastic events, this paper studies the dynamics of the resulting Markov random network and characterizes a novel phenomenon of intermittent…

The article studies segmentation problem (also known as classification problem) with pairwise Markov models (PMMs). A PMM is a process where the observation process and underlying state sequence form a two-dimensional Markov chain, it is a…

Markov branching systems form a fundamental class of stochastic models that are extensively applied in biology, physics, finance, and other domains. These systems are distinguished by their continuous-time evolution and inherent branching…

Markov chains provide a foundational framework for modeling sequential stochastic processes, with the transition probability matrix characterizing the dynamics of state evolution. While classical estimation methods such as maximum…

We consider Markov decision processes (MDPs) in which the transition probabilities and rewards belong to an uncertainty set parametrized by a collection of random variables. The probability distributions for these random parameters are…

In this paper, the recurrent events that can occur more than one over the follow-up time have been modeled by phase-type distributions. We use the finite-state continuous-time Markov process with multi states for patients with recurrent…

We address the problem of detecting an anomalous process among a large number of processes. At each time t, normal processes are in state zero (normal state), while the abnormal process may be in either state zero (normal state) or state…

The formal verification of large probabilistic models is important and challenging. Exploiting the concurrency that is often present is one way to address this problem. Here we study a restricted class of asynchronous distributed…

The paper deals with finite-state Markov decision processes (MDPs) with integer weights assigned to each state-action pair. New algorithms are presented to classify end components according to their limiting behavior with respect to the…