Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
Machine Learning
2020-02-10 v3 Machine Learning
Abstract
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.
Keywords
Cite
@article{arxiv.1905.11286,
title = {Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks},
author = {Boris Ginsburg and Patrice Castonguay and Oleksii Hrinchuk and Oleksii Kuchaiev and Vitaly Lavrukhin and Ryan Leary and Jason Li and Huyen Nguyen and Yang Zhang and Jonathan M. Cohen},
journal= {arXiv preprint arXiv:1905.11286},
year = {2020}
}
Comments
Preprint, under review