Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning
Abstract
In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one-step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.
Cite
@article{arxiv.1805.11240,
title = {Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning},
author = {Wen Sun and J. Andrew Bagnell and Byron Boots},
journal= {arXiv preprint arXiv:1805.11240},
year = {2018}
}
Comments
ICLR 2018