SWALP : Stochastic Weight Averaging in Low-Precision Training
Machine Learning
2019-05-21 v2 Artificial Intelligence
Machine Learning
Abstract
Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.
Cite
@article{arxiv.1904.11943,
title = {SWALP : Stochastic Weight Averaging in Low-Precision Training},
author = {Guandao Yang and Tianyi Zhang and Polina Kirichenko and Junwen Bai and Andrew Gordon Wilson and Christopher De Sa},
journal= {arXiv preprint arXiv:1904.11943},
year = {2019}
}
Comments
Published at ICML 2019