In this research, some of the issues that arise from the scalarization of the multi-objective optimization problem in the Advantage Actor Critic (A2C) reinforcement learning algorithm are investigated. The paper shows how a naive scalarization can lead to gradients overlapping. Furthermore, the possibility that the entropy regularization term can be a source of uncontrolled noise is discussed. With respect to the above issues, a technique to avoid gradient overlapping is proposed, while keeping the same loss formulation. Moreover, a method to avoid the uncontrolled noise, by sampling the actions from distributions with a desired minimum entropy, is investigated. Pilot experiments have been carried out to show how the proposed method speeds up the training. The proposed approach can be applied to any Advantage-based Reinforcement Learning algorithm.
@article{arxiv.2004.04120,
title = {Solving the scalarization issues of Advantage-based Reinforcement Learning Algorithms},
author = {Federico A. Galatolo and Mario G. C. A. Cimino and Gigliola Vaglini},
journal= {arXiv preprint arXiv:2004.04120},
year = {2021}
}