Gradient Descent Can Take Exponential Time to Escape Saddle Points
Abstract
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, gradient descent with perturbations [Ge et al., 2015, Jin et al., 2017] is not slowed down by saddle points - it can find an approximate local minimizer in polynomial time. This result implies that GD is inherently slower than perturbed GD, and justifies the importance of adding perturbations for efficient non-convex optimization. While our focus is theoretical, we also present experiments that illustrate our theoretical findings.
Cite
@article{arxiv.1705.10412,
title = {Gradient Descent Can Take Exponential Time to Escape Saddle Points},
author = {Simon S. Du and Chi Jin and Jason D. Lee and Michael I. Jordan and Barnabas Poczos and Aarti Singh},
journal= {arXiv preprint arXiv:1705.10412},
year = {2017}
}
Comments
Accepted by NIPS 2017