Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection
Abstract
There has been significant recent work on the theory and application of randomized coordinate descent algorithms, beginning with the work of Nesterov [SIAM J. Optim., 22(2), 2012], who showed that a random-coordinate selection rule achieves the same convergence rate as the Gauss-Southwell selection rule. This result suggests that we should never use the Gauss-Southwell rule, as it is typically much more expensive than random selection. However, the empirical behaviours of these algorithms contradict this theoretical result: in applications where the computational costs of the selection rules are comparable, the Gauss-Southwell selection rule tends to perform substantially better than random coordinate selection. We give a simple analysis of the Gauss-Southwell rule showing that---except in extreme cases---its convergence rate is faster than choosing random coordinates. Further, in this work we (i) show that exact coordinate optimization improves the convergence rate for certain sparse problems, (ii) propose a Gauss-Southwell-Lipschitz rule that gives an even faster convergence rate given knowledge of the Lipschitz constants of the partial derivatives, (iii) analyze the effect of approximate Gauss-Southwell rules, and (iv) analyze proximal-gradient variants of the Gauss-Southwell rule.
Keywords
Cite
@article{arxiv.1506.00552,
title = {Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection},
author = {Julie Nutini and Mark Schmidt and Issam H. Laradji and Michael Friedlander and Hoyt Koepke},
journal= {arXiv preprint arXiv:1506.00552},
year = {2018}
}
Comments
ICML 2015. v2: Updated the Gauss-Southwell-q result in Section 8 and Appendix H, to remove the part depending on mu_1 (the proof had an error). Added Section 8.1, which discusses conditions under which a rate depending on mu_1 does hold