ADDQ: Adaptive Distributional Double Q-Learning
Abstract
Bias problems in the estimation of -values are a well-known obstacle that slows down convergence of -learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We propose an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework is simple to implement, existing distributional algorithms can be improved with a few lines of code. We provide theoretical evidence and use double -learning to show how to include locally adaptive overestimation control in existing algorithms. Experiments are provided for tabular, Atari, and MuJoCo environments.
Cite
@article{arxiv.2506.19478,
title = {ADDQ: Adaptive Distributional Double Q-Learning},
author = {Leif Döring and Benedikt Wille and Maximilian Birr and Mihail Bîrsan and Martin Slowik},
journal= {arXiv preprint arXiv:2506.19478},
year = {2025}
}