Emphatic TD Bellman Operator is a Contraction
Machine Learning
2015-08-25 v2 Machine Learning
Abstract
Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a -contraction modulus (where is the discount factor). This allows us to provide error bounds on the approximation error of ETD. To our knowledge, these are the first error bounds for an off-policy evaluation algorithm under general target and behavior policies.
Cite
@article{arxiv.1508.03411,
title = {Emphatic TD Bellman Operator is a Contraction},
author = {Assaf Hallak and Aviv Tamar and Shie Mannor},
journal= {arXiv preprint arXiv:1508.03411},
year = {2015}
}