Related papers: Markovian Transition Counting Processes: An Altern…
Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters…
Markov decision processes (MDPs) in queues and networks have been an interesting topic in many practical areas since the 1960s. This paper provides a detailed overview on this topic and tracks the evolution of many basic results. Also, this…
Control theory plays a pivotal role in understanding and optimizing the behavior of complex dynamical systems across various scientific and engineering disciplines. Two key frameworks that have emerged for modeling and solving control…
Optimal control in non-stationary Markov decision processes (MDP) is a challenging problem. The aim in such a control problem is to maximize the long-term discounted reward when the transition dynamics or the reward function can change over…
When clustering molecular dynamics (MD) trajectories into a few metastable conformational states, the Markov state models (MSMs) assumption of timescale separation between fast intrastate fluctuations and rarely occurring interstate…
In this paper we propose a new method for approximating the nonstationary moment dynamics of one dimensional Markovian birth-death processes. By expanding the transition probabilities of the Markov process in terms of Poisson-Charlier…
We propose Markov two-components processes (M2CP) as a probabilistic model of asynchronous systems based on the trace semantics for concurrency. Considering an asynchronous system distributed over two sites, we introduce concepts and tools…
This paper studies the workload distribution of a finite-capacity queue driven by a spectrally one-sided Markov additive process (MAP). Our main result provides the Laplace-Stieltjes transform of the workload at an exponentially distributed…
Markov decision processes (MDPs) are a popular model for performance analysis and optimization of stochastic systems. The parameters of stochastic behavior of MDPs are estimates from empirical observations of a system; their values are not…
Markov decision processes (MDPs) are a standard model for sequential decision-making problems and are widely used across many scientific areas, including formal methods and artificial intelligence (AI). MDPs do, however, come with the…
We investigate Markovian queues that are examined by a controller at random times determined by a Poisson process. Upon examination, the controller sets the service speed to be equal to the minimum of the current number of customers in the…
Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical…
Folklore often treats the Markov Modulated Poisson Process as bursty because the variance divided by the expectation of counts is greater than unity. When viewed through the lens of the inter-event process, this ideally corresponds to a…
We consider a single-server queue where interarrival and service times depend linearly and randomly on customer waiting times, and establish a sample-path moderate deviation principle (MDP) for the waiting time process. The waiting times…
We construct a stochastic fluid process with an underlying piecewise deterministic Markov process (PDMP) akin to the one used in the construction of the rational arrival process (RAP), which we call the RAP-modulated fluid process. As…
In parallel-server systems with a single stream of arrivals (a.k.a. load balancing), Join-the-Shortest-Queue (JSQ) is a popular routing algorithm. There is extensive literature studying this system in various asymptotic regimes, but all…
We consider a change-point detection problem for a simple class of Piecewise Deterministic Markov Processes (PDMPs). A continuous-time PDMP is observed in discrete time and through noise, and the aim is to propose a numerical method to…
We introduce Markov Neural Processes (MNPs), a new class of Stochastic Processes (SPs) which are constructed by stacking sequences of neural parameterised Markov transition operators in function space. We prove that these Markov transition…
The parameters of a discrete stationary Markov model are transition probabilities between states. Traditionally, data consist in sequences of observed states for a given number of individuals over the whole observation period. In such a…
We introduce and study a multiparameter version of the generalized counting process (GCP), where there is a possibility of finitely many arrivals simultaneously. We call it the multiparameter GCP. In a particular case, it is uniquely…