Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning
Abstract
In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or semi-deterministic to stochastic quadratic regularization methods. We leverage the two-phase nature of stochastic optimization to propose a novel first-order algorithm with adaptive sampling and adaptive step size. In the second-order case, we propose a novel stochastic damped L-BFGS method that improves on previous algorithms in the highly nonconvex context of deep learning. Both algorithms are evaluated on well-known deep learning datasets and exhibit promising performance.
Cite
@article{arxiv.2111.14761,
title = {Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning},
author = {Sanae Lotfi and Tiphaine Bonniot de Ruisselet and Dominique Orban and Andrea Lodi},
journal= {arXiv preprint arXiv:2111.14761},
year = {2021}
}
Comments
29 pages, 8 figures. arXiv admin note: text overlap with arXiv:2012.05783