Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations
Machine Learning
2025-10-14 v2 Numerical Analysis
Numerical Analysis
Optimization and Control
Abstract
In this paper, we propose a continuous-time formulation for the AdaGrad, RMSProp, and Adam optimization algorithms by modeling them as first-order integro-differential equations. We perform numerical simulations of these equations, along with stability and convergence analyses, to demonstrate their validity as accurate approximations of the original algorithms. Our results indicate a strong agreement between the behavior of the continuous-time models and the discrete implementations, thus providing a new perspective on the theoretical understanding of adaptive optimization methods.
Keywords
Cite
@article{arxiv.2411.09734,
title = {Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations},
author = {Carlos Heredia},
journal= {arXiv preprint arXiv:2411.09734},
year = {2025}
}
Comments
47 pages, 15 figures