SGD with a Constant Large Learning Rate Can Converge to Local Maxima
Abstract
Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regimes that the previous works often assume, SGD can exhibit many strange and potentially undesirable behaviors. Specifically, we construct landscapes and data distributions such that (1) SGD converges to local maxima, (2) SGD escapes saddle points arbitrarily slowly, (3) SGD prefers sharp minima over flat ones, and (4) AMSGrad converges to local maxima. We also realize results in a minimal neural network-like example. Our results highlight the importance of simultaneously analyzing the minibatch sampling, discrete-time updates rules, and realistic landscapes to understand the role of SGD in deep learning.
Keywords
Cite
@article{arxiv.2107.11774,
title = {SGD with a Constant Large Learning Rate Can Converge to Local Maxima},
author = {Liu Ziyin and Botao Li and James B. Simon and Masahito Ueda},
journal= {arXiv preprint arXiv:2107.11774},
year = {2023}
}
Comments
Fixed typos