Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search
Machine Learning
2023-08-08 v1 Optimization and Control
Abstract
This paper explores two recent methods for learning rate optimisation in stochastic gradient descent: D-Adaptation (arXiv:2301.07733) and probabilistic line search (arXiv:1502.02846). These approaches aim to alleviate the burden of selecting an initial learning rate by incorporating distance metrics and Gaussian process posterior estimates, respectively. In this report, I provide an intuitive overview of both methods, discuss their shared design goals, and devise scope for merging the two algorithms.
Cite
@article{arxiv.2308.03102,
title = {Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search},
author = {Max McGuinness},
journal= {arXiv preprint arXiv:2308.03102},
year = {2023}
}