Stochastic modified equations and adaptive stochastic gradient algorithms
Machine Learning
2017-06-21 v3 Machine Learning
Abstract
We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.
Cite
@article{arxiv.1511.06251,
title = {Stochastic modified equations and adaptive stochastic gradient algorithms},
author = {Qianxiao Li and Cheng Tai and Weinan E},
journal= {arXiv preprint arXiv:1511.06251},
year = {2017}
}
Comments
Major changes including a proof of the weak approximation, asymptotic expansions and application-oriented adaptive algorithms