Faster Policy Learning with Continuous-Time Gradients
Abstract
We study the estimation of policy gradients for continuous-time systems with known dynamics. By reframing policy learning in continuous-time, we show that it is possible construct a more efficient and accurate gradient estimator. The standard back-propagation through time estimator (BPTT) computes exact gradients for a crude discretization of the continuous-time system. In contrast, we approximate continuous-time gradients in the original system. With the explicit goal of estimating continuous-time gradients, we are able to discretize adaptively and construct a more efficient policy gradient estimator which we call the Continuous-Time Policy Gradient (CTPG). We show that replacing BPTT policy gradients with more efficient CTPG estimates results in faster and more robust learning in a variety of control tasks and simulators.
Cite
@article{arxiv.2012.06684,
title = {Faster Policy Learning with Continuous-Time Gradients},
author = {Samuel Ainsworth and Kendall Lowrey and John Thickstun and Zaid Harchaoui and Siddhartha Srinivasa},
journal= {arXiv preprint arXiv:2012.06684},
year = {2021}
}