Discounted Reinforcement Learning Is Not an Optimization Problem
Artificial Intelligence
2019-11-28 v3
Abstract
Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks. It is not an optimization problem in its usual formulation, so when using function approximation there is no optimal policy. We substantiate these claims, then go on to address some misconceptions about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks.
Cite
@article{arxiv.1910.02140,
title = {Discounted Reinforcement Learning Is Not an Optimization Problem},
author = {Abhishek Naik and Roshan Shariff and Niko Yasui and Hengshuai Yao and Richard S. Sutton},
journal= {arXiv preprint arXiv:1910.02140},
year = {2019}
}
Comments
Accepted for presentation at the Optimization Foundations of Reinforcement Learning Workshop at NeurIPS 2019