Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpoint
Abstract
We propose two policy gradient algorithms for solving the problem of control in an off-policy reinforcement learning (RL) context. Both algorithms incorporate a smoothed functional (SF) based gradient estimation scheme. The first algorithm is a straightforward combination of importance sampling-based off-policy evaluation with SF-based gradient estimation. The second algorithm, inspired by the stochastic variance-reduced gradient (SVRG) algorithm, incorporates variance reduction in the update iteration. For both algorithms, we derive non-asymptotic bounds that establish convergence to an approximate stationary point. From these results, we infer that the first algorithm converges at a rate that is comparable to the well-known REINFORCE algorithm in an off-policy RL context, while the second algorithm exhibits an improved rate of convergence.
Cite
@article{arxiv.2101.02137,
title = {Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpoint},
author = {Nithia Vijayan and Prashanth L. A},
journal= {arXiv preprint arXiv:2101.02137},
year = {2024}
}