Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces
Abstract
Parameterised actions in reinforcement learning are composed of discrete actions with continuous action-parameters. This provides a framework for solving complex domains that require combining high-level actions with flexible control. The recent P-DQN algorithm extends deep Q-networks to learn over such action spaces. However, it treats all action-parameters as a single joint input to the Q-network, invalidating its theoretical foundations. We analyse the issues with this approach and propose a novel method, multi-pass deep Q-networks, or MP-DQN, to address them. We empirically demonstrate that MP-DQN significantly outperforms P-DQN and other previous algorithms in terms of data efficiency and converged policy performance on the Platform, Robot Soccer Goal, and Half Field Offense domains.
Cite
@article{arxiv.1905.04388,
title = {Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces},
author = {Craig J. Bester and Steven D. James and George D. Konidaris},
journal= {arXiv preprint arXiv:1905.04388},
year = {2019}
}
Comments
8 pages, 4 figures