ISAAC Newton: Input-based Approximate Curvature for Newton's Method
Machine Learning
2023-05-02 v1 Computer Vision and Pattern Recognition
Optimization and Control
Machine Learning
Abstract
We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.
Cite
@article{arxiv.2305.00604,
title = {ISAAC Newton: Input-based Approximate Curvature for Newton's Method},
author = {Felix Petersen and Tobias Sutter and Christian Borgelt and Dongsung Huh and Hilde Kuehne and Yuekai Sun and Oliver Deussen},
journal= {arXiv preprint arXiv:2305.00604},
year = {2023}
}
Comments
Published at ICLR 2023, Code @ https://github.com/Felix-Petersen/isaac, Video @ https://youtu.be/7RKRX-MdwqM