Emphatic Temporal-Difference Learning
Abstract
Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps. Recent works by Sutton, Mahmood and White (2015), and Yu (2015) show that by varying the emphasis in a particular way, these algorithms become stable and convergent under off-policy training with linear function approximation. This paper serves as a unified summary of the available results from both works. In addition, we demonstrate the empirical benefits from the flexibility of emphatic algorithms, including state-dependent discounting, state-dependent bootstrapping, and the user-specified allocation of function approximation resources.
Cite
@article{arxiv.1507.01569,
title = {Emphatic Temporal-Difference Learning},
author = {A. Rupam Mahmood and Huizhen Yu and Martha White and Richard S. Sutton},
journal= {arXiv preprint arXiv:1507.01569},
year = {2015}
}
Comments
9 pages, accepted for presentation at European Workshop on Reinforcement Learning