Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
Optimization and Control
2020-06-15 v5
Abstract
In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated and non-accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.
Cite
@article{arxiv.1911.08380,
title = {Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization},
author = {Darina Dvinskikh and Aleksandr Ogaltsov and Alexander Gasnikov and Pavel Dvurechensky and Alexander Tyurin and Vladimir Spokoiny},
journal= {arXiv preprint arXiv:1911.08380},
year = {2020}
}
Comments
18 pages