An introduction to decentralized stochastic optimization with gradient tracking
Machine Learning
2019-11-14 v2 Systems and Control
Systems and Control
Optimization and Control
Machine Learning
Abstract
Decentralized solutions to finite-sum minimization are of significant importance in many signal processing, control, and machine learning applications. In such settings, the data is distributed over a network of arbitrarily-connected nodes and raw data sharing is prohibitive often due to communication or privacy constraints. In this article, we review decentralized stochastic first-order optimization methods and illustrate some recent improvements based on gradient tracking and variance reduction, focusing particularly on smooth and strongly-convex objective functions. We provide intuitive illustrations of the main technical ideas as well as applications of the algorithms in the context of decentralized training of machine learning models.
Cite
@article{arxiv.1907.09648,
title = {An introduction to decentralized stochastic optimization with gradient tracking},
author = {Ran Xin and Soummya Kar and Usman A. Khan},
journal= {arXiv preprint arXiv:1907.09648},
year = {2019}
}