Gradient Descent Happens in a Tiny Subspace
Machine Learning
2018-12-13 v1 Artificial Intelligence
Machine Learning
Abstract
We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.
Keywords
Cite
@article{arxiv.1812.04754,
title = {Gradient Descent Happens in a Tiny Subspace},
author = {Guy Gur-Ari and Daniel A. Roberts and Ethan Dyer},
journal= {arXiv preprint arXiv:1812.04754},
year = {2018}
}
Comments
9 pages + appendices, 12 figures