Related papers: A Notation for Markov Decision Processes
Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with…
Partially observable Markov decision processes (POMDPs) are standard models for dynamic systems with probabilistic and nondeterministic behaviour in uncertain environments. We prove that in POMDPs with long-run average objective, the…
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume…
We present some hardness results on finding the optimal policy for the static formulation of distributionally robust Markov decision processes. We construct problem instances such that when the considered policy class is Markovian and…
Generators of Markov processes on a countable state space can be represented as finite or infinite matrices. One key property is that the off-diagonal entries corresponding to jump rates of the Markov process are non-negative. Here we…
We develop a model for credit rating migration that accounts for the impact of economic state fluctuations on default probabilities. The joint process for the economic state and the rating is modelled as a time-homogeneous Markov chain.…
An analogue of the classical Mecke formula for Poisson point processes is proved for the class of space-time STIT tessellation processes. From this key identity the Markov property of a class of associated random processes is derived. This…
Markov decision processes capture sequential decision making under uncertainty, where an agent must choose actions so as to optimize long term reward. The paper studies efficient reasoning mechanisms for Relational Markov Decision Processes…
Matryoshka dolls, the traditional Russian nesting figurines, are known world-wide for each doll's encapsulation of a sequence of smaller dolls. In this paper, we identify a large class of Markov process whose moments are easy to compute by…
We consider almost upper semi-continuous processes defined on a finite Markov chain. The distributions of the functionals associated with the exit from a finite interval are studied. We also consider some modification of these processes.
This is a short tutorial of the Case Management Model and Notation (CMMN) version 1.0. It is targeted to readers with knowledge of basic process or workflow modeling, and it covers the complete CMMN notation. A simple complaints process is…
Under continuity and recurrence assumptions, we prove that the iteration of successive partial symmetrizations that form a time-homogeneous Markov process, converges to a symmetrization. We cover several settings, including the…
This paper is concentrated on the classification of permutation matrix with the permutation similarity relation, mainly about the canonical form of a permutational similar equivalence class, the cycle matrix decomposition of a permutation…
This paper discusses algorithms for solving Markov decision processes (MDPs) that have monotone optimal policies. We propose a two-stage alternating convex optimization scheme that can accelerate the search for an optimal policy by…
This is lecture notes on the course "Stochastic Processes". In this format, the course was taught in the spring semesters 2017 and 2018 for third-year bachelor students of the Department of Control and Applied Mathematics, School of Applied…
We consider statistical Markov Decision Processes where the decision maker is risk averse against model ambiguity. The latter is given by an unknown parameter which influences the transition law and the cost functions. Risk aversion is…
Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters…
This paper presents a semi-Markov decision process (SMDP) formulation of the satellite task scheduling problem. This formulation can consider multiple operational objectives simultaneously and plan transitions between distinct functional…
We extend the notion of Cantor-Kantorovich distance between Markov chains introduced by (Banse et al., 2023) in the context of Markov Decision Processes (MDPs). The proposed metric is well-defined and can be efficiently approximated given a…
In this paper, we investigate the concentration properties of cumulative reward in Markov Decision Processes (MDPs), focusing on both asymptotic and non-asymptotic settings. We introduce a unified approach to characterize reward…