A Full Adagrad algorithm with O(Nd) operations
Statistics Theory
2025-02-28 v2 Machine Learning
Statistics Theory
Abstract
A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By developing a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective optimization processes. Moreover, the convergence rates of the proposed estimators and their asymptotic efficiency are given. Their effectiveness is demonstrated through numerical studies.
Cite
@article{arxiv.2405.01908,
title = {A Full Adagrad algorithm with O(Nd) operations},
author = {Antoine Godichon-Baggioni and Wei Lu and Bruno Portier},
journal= {arXiv preprint arXiv:2405.01908},
year = {2025}
}