Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization
Optimization and Control
2021-10-01 v1 Machine Learning
Machine Learning
Abstract
We show that stochastic acceleration can be achieved under the perturbed iterate framework (Mania et al., 2017) in asynchronous lock-free optimization, which leads to the optimal incremental gradient complexity for finite-sum objectives. We prove that our new accelerated method requires the same linear speed-up condition as the existing non-accelerated methods. Our core algorithmic discovery is a new accelerated SVRG variant with sparse updates. Empirical results are presented to verify our theoretical findings.
Cite
@article{arxiv.2109.15292,
title = {Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization},
author = {Kaiwen Zhou and Anthony Man-Cho So and James Cheng},
journal= {arXiv preprint arXiv:2109.15292},
year = {2021}
}
Comments
21 pages, 22 figures