Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
Dynamical Systems
2016-06-08 v2 Machine Learning
Abstract
Given a non-convex twice differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where \nabla^2 f has at least one strictly negative eigenvalue has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.
Keywords
Cite
@article{arxiv.1605.00405,
title = {Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions},
author = {Ioannis Panageas and Georgios Piliouras},
journal= {arXiv preprint arXiv:1605.00405},
year = {2016}
}
Comments
2 figures