Linearly convergent stochastic heavy ball method for minimizing generalization error
Optimization and Control
2017-12-27 v2 Machine Learning
Numerical Analysis
Machine Learning
Abstract
In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder problem. While in the analysis we constrain ourselves to quadratic loss, the overall objective is not necessarily strongly convex.
Cite
@article{arxiv.1710.10737,
title = {Linearly convergent stochastic heavy ball method for minimizing generalization error},
author = {Nicolas Loizou and Peter Richtárik},
journal= {arXiv preprint arXiv:1710.10737},
year = {2017}
}
Comments
NIPS 2017, Workshop on Optimization for Machine Learning (camera ready version)