Central Limit Theorems for Asynchronous Averaged Q-Learning

Xingtu Liu

Central Limit Theorems for Asynchronous Averaged Q-Learning

Machine Learning 2026-04-21 v3 Optimization and Control Machine Learning

Authors: Xingtu Liu

Abstract

This paper establishes central limit theorems for Polyak-Ruppert averaged Q-learning under asynchronous updates. We prove a non-asymptotic central limit theorem, where the convergence rate in Wasserstein distance explicitly reflects the dependence on the number of iterations, state-action space size, the discount factor, and the quality of exploration. In addition, we derive a functional central limit theorem, showing that the partial-sum process converges weakly to a Brownian motion.

Keywords

probability theory

Cite

@article{arxiv.2509.18964,
  title  = {Central Limit Theorems for Asynchronous Averaged Q-Learning},
  author = {Xingtu Liu},
  journal= {arXiv preprint arXiv:2509.18964},
  year   = {2026}
}