A second-order-like optimizer with adaptive gradient scaling for deep learning
Machine Learning
2024-12-13 v2 Artificial Intelligence
Optimization and Control
Abstract
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After giving geometrical insights, we evaluate INNAprop on CIFAR-10, Food101, and ImageNet with ResNets, VGG, DenseNet, and ViT, and on GPT-2 (OpenWebText) train from scratch and with LoRA fine-tuning (E2E). INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.
Cite
@article{arxiv.2410.05871,
title = {A second-order-like optimizer with adaptive gradient scaling for deep learning},
author = {Jérôme Bolte and Ryan Boustany and Edouard Pauwels and Andrei Purica},
journal= {arXiv preprint arXiv:2410.05871},
year = {2024}
}