Adaptive Hierarchical Hyper-gradient Descent
Machine Learning
2021-05-12 v3 Neural and Evolutionary Computing
Abstract
In this study, we investigate learning rate adaption at different levels based on the hyper-gradient descent framework and propose a method that adaptively learns the optimizer parameters by combining multiple levels of learning rates with hierarchical structures. Meanwhile, we show the relationship between regularizing over-parameterized learning rates and building combinations of adaptive learning rates at different levels. The experiments on several network architectures, including feed-forward networks, LeNet-5 and ResNet-18/34, show that the proposed multi-level adaptive approach can outperform baseline adaptive methods in a variety of circumstances.
Cite
@article{arxiv.2008.07277,
title = {Adaptive Hierarchical Hyper-gradient Descent},
author = {Renlong Jie and Junbin Gao and Andrey Vasnev and Minh-Ngoc Tran},
journal= {arXiv preprint arXiv:2008.07277},
year = {2021}
}