Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization

Zhouyuan Huo; Heng Huang

Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization

Machine Learning 2016-12-21 v4 Optimization and Control

Authors: Zhouyuan Huo , Heng Huang

Abstract

We provide the first theoretical analysis on the convergence rate of the asynchronous stochastic variance reduced gradient (SVRG) descent algorithm on non-convex optimization. Recent studies have shown that the asynchronous stochastic gradient descent (SGD) based algorithms with variance reduction converge with a linear convergent rate on convex problems. However, there is no work to analyze asynchronous SGD with variance reduction technique on non-convex problem. In this paper, we study two asynchronous parallel implementations of SVRG: one is on a distributed memory system and the other is on a shared memory system. We provide the theoretical analysis that both algorithms can obtain a convergence rate of $O(1/T)$ , and linear speed up is achievable if the number of workers is upper bounded. V1,v2,v3 have been withdrawn due to reference issue, please refer the newest version v4.

Keywords

stochastic gradient descent distributed training convex optimization

Cite

@article{arxiv.1604.03584,
  title  = {Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization},
  author = {Zhouyuan Huo and Heng Huang},
  journal= {arXiv preprint arXiv:1604.03584},
  year   = {2016}
}

Comments

V1,v2,v3 have been withdrawn due to reference issue, because arXiv policy, we can't delete them. Please refer the newest version v4

Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization

Abstract

Keywords

Cite

Comments

Related papers