A geometric interpretation of stochastic gradient descent using diffusion metrics
Machine Learning
2020-02-19 v1 General Relativity and Quantum Cosmology
Differential Geometry
Machine Learning
Abstract
Stochastic gradient descent (SGD) is a key ingredient in the training of deep neural networks and yet its geometrical significance appears elusive. We study a deterministic model in which the trajectories of our dynamical systems are described via geodesics of a family of metrics arising from the diffusion matrix. These metrics encode information about the highly non-isotropic gradient noise in SGD. We establish a parallel with General Relativity models, where the role of the electromagnetic field is played by the gradient of the loss function. We compute an example of a two layer network.
Cite
@article{arxiv.1910.12194,
title = {A geometric interpretation of stochastic gradient descent using diffusion metrics},
author = {R. Fioresi and P. Chaudhari and S. Soatto},
journal= {arXiv preprint arXiv:1910.12194},
year = {2020}
}