Modularity benefits reinforcement learning agents with competing homeostatic drives
Abstract
The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.
Cite
@article{arxiv.2204.06608,
title = {Modularity benefits reinforcement learning agents with competing homeostatic drives},
author = {Zack Dulberg and Rachit Dubey and Isabel M. Berwian and Jonathan D. Cohen},
journal= {arXiv preprint arXiv:2204.06608},
year = {2022}
}
Comments
4 pages, accepted paper at the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM) 2022