A Generalizable Approach to Learning Optimizers
Machine Learning
2021-06-09 v2 Artificial Intelligence
Neural and Evolutionary Computing
Machine Learning
Abstract
A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.
Cite
@article{arxiv.2106.00958,
title = {A Generalizable Approach to Learning Optimizers},
author = {Diogo Almeida and Clemens Winter and Jie Tang and Wojciech Zaremba},
journal= {arXiv preprint arXiv:2106.00958},
year = {2021}
}