Experience replay enables data-efficient learning from past experiences in online reinforcement learning agents. Traditionally, experiences were sampled uniformly from a replay buffer, regardless of differences in experience-specific learning potential. In an effort to sample more efficiently, researchers introduced Prioritized Experience Replay (PER). In this paper, we propose an extension to PER by introducing a novel measure of temporal difference error reliability. We theoretically show that the resulting transition selection algorithm, Reliability-adjusted Prioritized Experience Replay (ReaPER), enables more efficient learning than PER. We further present empirical results showing that ReaPER outperforms PER across various environment types, including the Atari-10 benchmark.
@article{arxiv.2506.18482,
title = {Reliability-Adjusted Prioritized Experience Replay},
author = {Leonard S. Pleiss and Tobias Sutter and Maximilian Schiffer},
journal= {arXiv preprint arXiv:2506.18482},
year = {2025}
}