Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent
Optimization and Control
2023-09-06 v1 Numerical Analysis
Numerical Analysis
Machine Learning
Abstract
Stochastic optimization methods have been hugely successful in making large-scale optimization problems feasible when computing the full gradient is computationally prohibitive. Using the theory of modified equations for numerical integrators, we propose a class of stochastic differential equations that approximate the dynamics of general stochastic optimization methods more closely than the original gradient flow. Analyzing a modified stochastic differential equation can reveal qualitative insights about the associated optimization method. Here, we study mean-square stability of the modified equation in the case of stochastic coordinate descent.
Cite
@article{arxiv.2309.02082,
title = {Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent},
author = {Stefano Di Giovacchino and Desmond J. Higham and Konstantinos Zygalakis},
journal= {arXiv preprint arXiv:2309.02082},
year = {2023}
}
Comments
15 pages; 3 figures