Related papers: Sample Path Moderate Deviation Principle for Queue…
Non-stationary environments are challenging for reinforcement learning algorithms. If the state transition and/or reward functions change based on latent factors, the agent is effectively tasked with optimizing a behavior that maximizes…
We consider Markov decision processes (MDPs) with unknown disturbance distribution and address this problem using the robust Markov decision process (RMDP) approach. We construct the empirical distribution of the unknown disturbance…
In Markov Decision Processes (MDPs) with intermittent state information, decision-making becomes challenging due to periods of missing observations. Linear programming (LP) methods can play a crucial role in solving MDPs, in particular,…
Here, we explore the problem of error propagation mitigation in modular digital twins as a sequential decision process. Building on a companion study that used a Hidden Markov Model (HMM) to infer latent error regimes from surrogate-physics…
We establish a comprehensive sample path large deviation principle (LDP) for log-processes associated with multivariate time-inhomogeneous stochastic volatility models. Examples of models for which the new LDP holds include Gaussian models,…
Motivated by queueing applications, we study various reflected autoregressive processes with dependencies. Amongst others, we study cases where the interarrival and service times are proportionally dependent with additive and/or subtracting…
This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. The control…
We study a Q learning algorithm for continuous time stochastic control problems. The proposed algorithm uses the sampled state process by discretizing the state and control action spaces under piece-wise constant control processes. We show…
This note introduces a piecewise-deterministic queueing (PDQ) model to study the stability of traffic queues in parallel-link transportation systems facing stochastic capacity fluctuations. The saturation rate (capacity) of the PDQ model…
We consider a stable but nearly unstable autoregressive process of any order. The bridge between stability and instability is expressed by a time-varying companion matrix $A_{n}$ with spectral radius $\rho(A_{n}) < 1$ satisfying…
The purpose of this paper is to establish asymptotic behaviors of time-inhomogeneous multi-scale stochastic differential equations (SDEs). To achieve them, we analyze the evolution system of measures for time-inhomogeneous Markov…
In this paper, we propose a compositional approach for the construction of finite abstractions (a.k.a. finite Markov decision processes (MDPs)) for networks of discrete-time stochastic control subsystems that are not necessarily…
First, under a geometric ergodicity assumption, we provide some limit theorems and some probability inequalities for the bifurcating Markov chains (BMC). The BMC model was introduced by Guyon to detect cellular aging from cell lineage, and…
One-dimensional run-and-tumble processes may converge towards some localized non-equilibrium steady state when the two velocities and/or the two switching rates are space-dependent. A long dynamical trajectory can be then analyzed via the…
In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The…
We present a general framework for applying learning algorithms and heuristical guidance to the verification of Markov decision processes (MDPs). The primary goal of our techniques is to improve performance by avoiding an exhaustive…
Markov decision processes (MDP) are useful to model optimisation problems in concurrent systems. To verify MDPs with efficient Monte Carlo techniques requires that their nondeterminism be resolved by a scheduler. Recent work has introduced…
We discuss a single-server multi-station alternating queue where the preparation times and the service times are auto- and cross-correlated. We examine two cases. In the first case, preparation and service times depend on a common discrete…
We study the relation between different Markov Decision Process (MDP) frameworks in the machine learning and econometrics literatures, including the standard MDP, the entropy and general regularized MDP, and stochastic MDP, where the latter…
We propose a new approach for estimating the finite dimensional transition matrix of a Markov chain using a large number of independent sample paths observed at random times. The sample paths may be observed as few as two times, and the…