New Versions of Gradient Temporal Difference Learning
Machine Learning
2024-01-23 v4
Abstract
Sutton, Szepesv\'{a}ri and Maei introduced the first gradient temporal-difference (GTD) learning algorithms compatible with both linear function approximation and off-policy training. The goal of this paper is (a) to propose some variants of GTDs with extensive comparative analysis and (b) to establish new theoretical analysis frameworks for the GTDs. These variants are based on convex-concave saddle-point interpretations of GTDs, which effectively unify all the GTDs into a single framework, and provide simple stability analysis based on recent results on primal-dual gradient dynamics. Finally, numerical comparative analysis is given to evaluate these approaches.
Cite
@article{arxiv.2109.04033,
title = {New Versions of Gradient Temporal Difference Learning},
author = {Donghwan Lee and Han-Dong Lim and Jihoon Park and Okyong Choi},
journal= {arXiv preprint arXiv:2109.04033},
year = {2024}
}