Random-reshuffled SARAH does not need a full gradient computations
Machine Learning
2024-01-17 v2 Optimization and Control
Abstract
The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time. In this paper, we remove the necessity of a full gradient computation. This is achieved by using a randomized reshuffling strategy and aggregating stochastic gradients obtained in each epoch. The aggregated stochastic gradients serve as an estimate of a full gradient in the SARAH algorithm. We provide a theoretical analysis of the proposed approach and conclude the paper with numerical experiments that demonstrate the efficiency of this approach.
Cite
@article{arxiv.2111.13322,
title = {Random-reshuffled SARAH does not need a full gradient computations},
author = {Aleksandr Beznosikov and Martin Takáč},
journal= {arXiv preprint arXiv:2111.13322},
year = {2024}
}
Comments
20 pages, 2 algorithms, 5 figures, 3 tables