A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
Machine Learning
2015-10-29 v2 Classical Analysis and ODEs
Optimization and Control
Abstract
We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.
Cite
@article{arxiv.1503.01243,
title = {A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights},
author = {Weijie Su and Stephen Boyd and Emmanuel J. Candes},
journal= {arXiv preprint arXiv:1503.01243},
year = {2015}
}
Comments
To appear in Journal of Machine Learning Research. Added more simulation studies. Preliminary version appeared in NIPS 2014