MDP modeling for multi-stage stochastic programs
Machine Learning
2026-04-09 v2 Optimization and Control
Abstract
We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous action and state spaces. We extend policy graphs to include decision-dependent uncertainty for one-step transition probabilities as well as a limited form of statistical learning. We focus on the expressiveness of our modeling approach, illustrating ideas with a series of examples of increasing complexity. As a solution method, we develop new variants of stochastic dual dynamic programming, including approximations to handle non-convexities.
Cite
@article{arxiv.2509.22981,
title = {MDP modeling for multi-stage stochastic programs},
author = {David P. Morton and Oscar Dowson and Bernardo K. Pagnoncelli},
journal= {arXiv preprint arXiv:2509.22981},
year = {2026}
}