Efficient Algorithms for Robust Markov Decision Processes with $s$-Rectangular Ambiguity Sets
Abstract
Robust Markov decision processes (MDPs) have attracted significant interest due to their ability to protect MDPs from poor out-of-sample performance in the presence of ambiguity. In contrast to classical MDPs, which account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, a robust MDP additionally accounts for ambiguity by optimizing against the most adverse transition kernel from an ambiguity set constructed via historical data. In this paper, we develop a unified solution framework for a broad class of robust MDPs with -rectangular ambiguity sets, where the most adverse transition probabilities are considered independently for each state. Using our algorithms, we show that -rectangular robust MDPs with - and -norm as well as -divergence ambiguity sets can be solved several orders of magnitude faster than with state-of-the-art commercial solvers, and often only a logarithmic factor slower than classical MDPs. We demonstrate the favorable scaling properties of our algorithms on a range of synthetically generated as well as standard benchmark instances.
Cite
@article{arxiv.2602.05591,
title = {Efficient Algorithms for Robust Markov Decision Processes with $s$-Rectangular Ambiguity Sets},
author = {Chin Pang Ho and Marek Petrik and Wolfram Wiesemann},
journal= {arXiv preprint arXiv:2602.05591},
year = {2026}
}