Dynamic Programming: From Local Optimality to Global Optimality
Optimization and Control
2025-05-13 v3
Abstract
In the theory of dynamic programming, an optimal policy is a policy whose lifetime value dominates that of all other policies from every possible initial condition in the state space. This raises a natural question: when does optimality from a single state imply optimality from every state? Working in a general setting, we provide sufficient conditions for this property that relate to reachability and irreducibility. Our results have significant implications for modern policy-based algorithms used to solve large-scale dynamic programs. We illustrate our findings by applying them to an optimal savings problem via an algorithm that implements gradient ascent in a policy space constructed from neural networks.
Cite
@article{arxiv.2411.11062,
title = {Dynamic Programming: From Local Optimality to Global Optimality},
author = {John Stachurski and Jingni Yang and Ziyue Yang},
journal= {arXiv preprint arXiv:2411.11062},
year = {2025}
}