The Mean-Squared Error of Double Q-Learning
Abstract
In this paper, we establish a theoretical comparison between the asymptotic mean-squared error of Double Q-learning and Q-learning. Our result builds upon an analysis for linear stochastic approximation based on Lyapunov equations and applies to both tabular setting and with linear function approximation, provided that the optimal policy is unique and the algorithms converge. We show that the asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning if Double Q-learning uses twice the learning rate of Q-learning and outputs the average of its two estimators. We also present some practical implications of this theoretical observation using simulations.
Cite
@article{arxiv.2007.05034,
title = {The Mean-Squared Error of Double Q-Learning},
author = {Wentao Weng and Harsh Gupta and Niao He and Lei Ying and R. Srikant},
journal= {arXiv preprint arXiv:2007.05034},
year = {2022}
}
Comments
An earlier verision of this paper appeared in NeurIPS 2020. This verision updated an incorrect equation and several typos