Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn policies. We thus investigate the presence of a common weakness in single-agent RL, namely value function overestimation bias, in the multi-agent setting. Based on our findings, we propose an approach that reduces this bias by using double centralized critics. We evaluate it on six mixed cooperative-competitive tasks, showing a significant advantage over current methods. Finally, we investigate the application of multi-agent methods to high-dimensional robotic tasks and show that our approach can be used to learn decentralized policies in this domain.
@article{arxiv.1910.01465,
title = {Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics},
author = {Johannes Ackermann and Volker Gabler and Takayuki Osa and Masashi Sugiyama},
journal= {arXiv preprint arXiv:1910.01465},
year = {2019}
}
Comments
Accepted for the Deep RL Workshop at NeurIPS 2019; Changes for v2: Changed Figures 3,4, due to an error in the implementation of MATD3. Please refer to this version for fair evaluation