Multi-Bellman operator for convergence of $Q$-learning with linear function approximation
Abstract
We study the convergence of -learning with linear function approximation. Our key contribution is the introduction of a novel multi-Bellman operator that extends the traditional Bellman operator. By exploring the properties of this operator, we identify conditions under which the projected multi-Bellman operator becomes contractive, providing improved fixed-point guarantees compared to the Bellman operator. To leverage these insights, we propose the multi -learning algorithm with linear function approximation. We demonstrate that this algorithm converges to the fixed-point of the projected multi-Bellman operator, yielding solutions of arbitrary accuracy. Finally, we validate our approach by applying it to well-known environments, showcasing the effectiveness and applicability of our findings.
Cite
@article{arxiv.2309.16819,
title = {Multi-Bellman operator for convergence of $Q$-learning with linear function approximation},
author = {Diogo S. Carvalho and Pedro A. Santos and Francisco S. Melo},
journal= {arXiv preprint arXiv:2309.16819},
year = {2023}
}