Quasi-hyperbolic momentum and Adam for deep learning
Abstract
Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.
Keywords
Cite
@article{arxiv.1810.06801,
title = {Quasi-hyperbolic momentum and Adam for deep learning},
author = {Jerry Ma and Denis Yarats},
journal= {arXiv preprint arXiv:1810.06801},
year = {2019}
}
Comments
Published as a conference paper at ICLR 2019. This version corrects one typological error in the published text