Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learning
Abstract
This paper investigates the stability and convergence properties of asynchronous stochastic approximation (SA) algorithms, with a focus on extensions relevant to average-reward reinforcement learning. We first extend a stability proof method of Borkar and Meyn to accommodate more general noise conditions than previously considered, thereby yielding broader convergence guarantees for asynchronous SA. To sharpen the convergence analysis, we further examine the shadowing properties of asynchronous SA, building on a dynamical systems approach of Hirsch and Bena\"{i}m. These results provide a theoretical foundation for a class of relative value iteration-based reinforcement learning algorithms -- developed and analyzed in a companion paper -- for solving average-reward Markov and semi-Markov decision processes.
Cite
@article{arxiv.2409.03915,
title = {Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learning},
author = {Huizhen Yu and Yi Wan and Richard S. Sutton},
journal= {arXiv preprint arXiv:2409.03915},
year = {2025}
}
Comments
34 pages. This version contains only the asynchronous stochastic approximation material from version 2 of the original report; the reinforcement-learning material has been moved to a separate, stand-alone paper (arXiv:2512.06218). Minor corrections and additional remarks have been incorporated. A shorter version of this paper is to appear in the SIAM Journal on Control and Optimization