A note on diffusion limits for stochastic gradient descent
Machine Learning
2022-10-21 v1 Optimization and Control
Probability
Abstract
In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in stochastic gradient algorithms, has widely approximated stochastic gradient descent by a stochastic differential equation with Gaussian noise. We provide a novel rigorous theoretical justification for this practice that showcases how the Gaussianity of the noise arises naturally.
Cite
@article{arxiv.2210.11257,
title = {A note on diffusion limits for stochastic gradient descent},
author = {Alberto Lanconelli and Christopher S. A. Lauria},
journal= {arXiv preprint arXiv:2210.11257},
year = {2022}
}
Comments
8 pages