Reinforcement Learning in POMDP's via Direct Gradient Ascent
Abstract
This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter , which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.
Cite
@article{arxiv.2512.02383,
title = {Reinforcement Learning in POMDP's via Direct Gradient Ascent},
author = {Jonathan Baxter and Peter L. Bartlett},
journal= {arXiv preprint arXiv:2512.02383},
year = {2025}
}