A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms
Abstract
In this paper, we introduce a unified framework for analyzing a large family of Q-learning algorithms, based on switching system perspectives and ODE-based stochastic approximation. We show that the nonlinear ODE models associated with these Q-learning algorithms can be formulated as switched linear systems, and analyze their asymptotic stability by leveraging existing switching system theories. Our approach provides the first O.D.E. analysis of the asymptotic convergence of various Q-learning algorithms, including asynchronous Q-learning and averaging Q-learning. We also extend the approach to analyze Q-learning with linear function approximation and derive a new sufficient condition for its convergence.
Cite
@article{arxiv.1912.02270,
title = {A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms},
author = {Donghwan Lee and Niao He},
journal= {arXiv preprint arXiv:1912.02270},
year = {2021}
}
Comments
This paper has been accepted in NeurIPS2020