We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.
Cite
@article{arxiv.2104.06159,
title = {Muesli: Combining Improvements in Policy Optimization},
author = {Matteo Hessel and Ivo Danihelka and Fabio Viola and Arthur Guez and Simon Schmitt and Laurent Sifre and Theophane Weber and David Silver and Hado van Hasselt},
journal= {arXiv preprint arXiv:2104.06159},
year = {2022}
}