Semi-Cyclic Stochastic Gradient Descent
Machine Learning
2019-04-24 v1 Machine Learning
Abstract
We consider convex SGD updates with a block-cyclic structure, i.e. where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different characteristics. We show that such block-cyclic structure can significantly deteriorate the performance of SGD, but propose a simple approach that allows prediction with the same performance guarantees as for i.i.d., non-cyclic, sampling.
Cite
@article{arxiv.1904.10120,
title = {Semi-Cyclic Stochastic Gradient Descent},
author = {Hubert Eichner and Tomer Koren and H. Brendan McMahan and Nathan Srebro and Kunal Talwar},
journal= {arXiv preprint arXiv:1904.10120},
year = {2019}
}