A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
Machine Learning
2017-06-21 v1 Optimization and Control
Machine Learning
Abstract
We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework captures and unifies much of the existing literature on adaptive online methods, including the AdaGrad and Online Newton Step algorithms as well as their diagonal versions. As a result, we obtain new convergence proofs for these algorithms that are substantially simpler than previous analyses. Our framework also exposes the rationale for the different preconditioned updates used in common stochastic optimization methods.
Cite
@article{arxiv.1706.06569,
title = {A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization},
author = {Vineet Gupta and Tomer Koren and Yoram Singer},
journal= {arXiv preprint arXiv:1706.06569},
year = {2017}
}