Mitigating Planner Overfitting in Model-Based Reinforcement Learning
Abstract
An agent with an inaccurate model of its environment faces a difficult choice: it can ignore the errors in its model and act in the real world in whatever way it determines is optimal with respect to its model. Alternatively, it can take a more conservative stance and eschew its model in favor of optimizing its behavior solely via real-world interaction. This latter approach can be exceedingly slow to learn from experience, while the former can lead to "planner overfitting" - aspects of the agent's behavior are optimized to exploit errors in its model. This paper explores an intermediate position in which the planner seeks to avoid overfitting through a kind of regularization of the plans it considers. We present three different approaches that demonstrably mitigate planner overfitting in reinforcement-learning environments.
Cite
@article{arxiv.1812.01129,
title = {Mitigating Planner Overfitting in Model-Based Reinforcement Learning},
author = {Dilip Arumugam and David Abel and Kavosh Asadi and Nakul Gopalan and Christopher Grimm and Jun Ki Lee and Lucas Lehnert and Michael L. Littman},
journal= {arXiv preprint arXiv:1812.01129},
year = {2020}
}