Variance Reduced Distributed Non-Convex Optimization Using Matrix Stepsizes
Abstract
Matrix-stepsized gradient descent algorithms have been shown to have superior performance in non-convex optimization problems compared to their scalar counterparts. The det-CGD algorithm, as introduced by Li et al. (2023), leverages matrix stepsizes to perform compressed gradient descent for non-convex objectives and matrix-smooth problems in a federated manner. The authors establish the algorithm's convergence to a neighborhood of a weighted stationarity point under a convex condition for the symmetric and positive-definite matrix stepsize. In this paper, we propose two variance-reduced versions of the det-CGD algorithm, incorporating MARINA and DASHA methods. Notably, we establish theoretically and empirically, that det-MARINA and det-DASHA outperform MARINA, DASHA and the distributed det-CGD algorithms in terms of iteration and communication complexities.
Cite
@article{arxiv.2310.04614,
title = {Variance Reduced Distributed Non-Convex Optimization Using Matrix Stepsizes},
author = {Hanmin Li and Avetik Karagulyan and Peter Richtárik},
journal= {arXiv preprint arXiv:2310.04614},
year = {2024}
}
Comments
Major update: The paper now includes an analysis of det-DASHA, which is another variance reduction extension of det-CGD. 63 pages, 12 figures