Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel
Systems and Control
2022-10-11 v1 Systems and Control
Optimization and Control
Abstract
We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control-switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.
Cite
@article{arxiv.2210.04098,
title = {Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel},
author = {Nathan Dahlin and Subhonmesh Bose and Venugopal V. Veeravalli},
journal= {arXiv preprint arXiv:2210.04098},
year = {2022}
}