Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

Cong Xie; Oluwasanmi Koyejo; Indranil Gupta; Haibin Lin

Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

Machine Learning 2020-12-08 v2 Distributed, Parallel, and Cluster Computing Machine Learning

Authors: Cong Xie , Oluwasanmi Koyejo , Indranil Gupta , Haibin Lin

Abstract

When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed algorithm significantly reduces the communication overhead, which, in turn, reduces the training time by up to 30% for the 1B word dataset.

Keywords

distributed training stochastic gradient descent optimization

Cite

@article{arxiv.1911.09030,
  title  = {Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates},
  author = {Cong Xie and Oluwasanmi Koyejo and Indranil Gupta and Haibin Lin},
  journal= {arXiv preprint arXiv:1911.09030},
  year   = {2020}
}

Related papers

View all related →