How Should We Meta-Learn Reinforcement Learning Algorithms?
Abstract
The process of meta-learning algorithms from data, instead of relying on manual design, is growing in popularity as a paradigm for improving the performance of machine learning systems. Meta-learning shows particular promise for reinforcement learning (RL), where algorithms are often adapted from supervised or unsupervised learning despite their suboptimality for RL. However, until now there has been a severe lack of comparison between different meta-learning algorithms, such as using evolution to optimise over black-box functions or LLMs to propose code. In this paper, we carry out this empirical comparison of the different approaches when applied to a range of meta-learned algorithms which target different parts of the RL pipeline. In addition to meta-train and meta-test performance, we also investigate factors including the interpretability, sample cost and train time for each meta-learning algorithm. Based on these findings, we propose several guidelines for meta-learning new RL algorithms which will help ensure that future learned algorithms are as performant as possible.
Cite
@article{arxiv.2507.17668,
title = {How Should We Meta-Learn Reinforcement Learning Algorithms?},
author = {Alexander David Goldie and Zilin Wang and Jaron Cohen and Jakob Nicolaus Foerster and Shimon Whiteson},
journal= {arXiv preprint arXiv:2507.17668},
year = {2025}
}
Comments
Accepted paper at Reinforcement Learning Conference (RLC) 2025