A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method
Machine Learning
2012-12-21 v2 Optimization and Control
Machine Learning
Abstract
In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.
Cite
@article{arxiv.1212.2002,
title = {A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method},
author = {Simon Lacoste-Julien and Mark Schmidt and Francis Bach},
journal= {arXiv preprint arXiv:1212.2002},
year = {2012}
}
Comments
8 pages, 6 figures. Changes with previous version: Added reference to concurrently submitted work arXiv:1212.1824v1; clarifications added; typos corrected; title changed to 'subgradient method' as 'subgradient descent' is misnomer