Solving goal-oriented tasks is an important but challenging problem in reinforcement learning (RL). For such tasks, the rewards are often sparse, making it difficult to learn a policy effectively. To tackle this difficulty, we propose a new approach called Policy Continuation with Hindsight Inverse Dynamics (PCHID). This approach learns from Hindsight Inverse Dynamics based on Hindsight Experience Replay, enabling the learning process in a self-imitated manner and thus can be trained with supervised learning. This work also extends it to multi-step settings with Policy Continuation. The proposed method is general, which can work in isolation or be combined with other on-policy and off-policy algorithms. On two multi-goal tasks GridWorld and FetchReach, PCHID significantly improves the sample efficiency as well as the final performance.
@article{arxiv.1910.14055,
title = {Policy Continuation with Hindsight Inverse Dynamics},
author = {Hao Sun and Zhizhong Li and Xiaotong Liu and Dahua Lin and Bolei Zhou},
journal= {arXiv preprint arXiv:1910.14055},
year = {2019}
}