This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.
@article{arxiv.2205.07802,
title = {The Primacy Bias in Deep Reinforcement Learning},
author = {Evgenii Nikishin and Max Schwarzer and Pierluca D'Oro and Pierre-Luc Bacon and Aaron Courville},
journal= {arXiv preprint arXiv:2205.07802},
year = {2022}
}
Comments
ICML 2022; code at https://github.com/evgenii-nikishin/rl_with_resets