Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis
Machine Learning
2021-09-16 v2 Analysis of PDEs
Machine Learning
Abstract
The representation of functions by artificial neural networks depends on a large number of parameters in a non-linear fashion. Suitable parameters of these are found by minimizing a 'loss functional', typically by stochastic gradient descent (SGD) or an advanced SGD-based algorithm. In a continuous time model for SGD with noise that follows the 'machine learning scaling', we show that in a certain noise regime, the optimization algorithm prefers 'flat' minima of the objective function in a sense which is different from the flat minimum selection of continuous time SGD with homogeneous noise.
Cite
@article{arxiv.2106.02588,
title = {Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis},
author = {Stephan Wojtowytsch},
journal= {arXiv preprint arXiv:2106.02588},
year = {2021}
}