Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
Optimization and Control
2024-09-19 v2 Machine Learning
Abstract
This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product.
Cite
@article{arxiv.2311.16956,
title = {Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent},
author = {Frederik Köhne and Leonie Kreis and Anton Schiela and Roland Herzog},
journal= {arXiv preprint arXiv:2311.16956},
year = {2024}
}