A survey of deep learning optimizers -- first and second order methods
Machine Learning
2023-09-28 v2 Computer Vision and Pattern Recognition
Optimization and Control
Abstract
Deep Learning optimization involves minimizing a high-dimensional loss function in the weight space which is often perceived as difficult due to its inherent difficulties such as saddle points, local minima, ill-conditioning of the Hessian and limited compute resources. In this paper, we provide a comprehensive review of standard optimization methods successfully used in deep learning research and a theoretical assessment of the difficulties in numerical optimization from the optimization literature.
Cite
@article{arxiv.2211.15596,
title = {A survey of deep learning optimizers -- first and second order methods},
author = {Rohan Kashyap},
journal= {arXiv preprint arXiv:2211.15596},
year = {2023}
}