Decoupled Asynchronous Proximal Stochastic Gradient Descent with Variance Reduction
Abstract
In the era of big data, optimizing large scale machine learning problems becomes a challenging task and draws significant attention. Asynchronous optimization algorithms come out as a promising solution. Recently, decoupled asynchronous proximal stochastic gradient descent (DAP-SGD) is proposed to minimize a composite function. It is claimed to be able to off-loads the computation bottleneck from server to workers by allowing workers to evaluate the proximal operators, therefore, server just need to do element-wise operations. However, it still suffers from slow convergence rate because of the variance of stochastic gradient is nonzero. In this paper, we propose a faster method, decoupled asynchronous proximal stochastic variance reduced gradient descent method (DAP-SVRG). We prove that our method has linear convergence for strongly convex problem. Large-scale experiments are also conducted in this paper, and results demonstrate our theoretical analysis.
Cite
@article{arxiv.1609.06804,
title = {Decoupled Asynchronous Proximal Stochastic Gradient Descent with Variance Reduction},
author = {Zhouyuan Huo and Bin Gu and Heng Huang},
journal= {arXiv preprint arXiv:1609.06804},
year = {2016}
}