LALR: Theoretical and Experimental validation of Lipschitz Adaptive Learning Rate in Regression and Neural Networks
Machine Learning
2020-06-25 v1 Machine Learning
Abstract
We propose a theoretical framework for an adaptive learning rate policy for the Mean Absolute Error loss function and Quantile loss function and evaluate its effectiveness for regression tasks. The framework is based on the theory of Lipschitz continuity, specifically utilizing the relationship between learning rate and Lipschitz constant of the loss function. Based on experimentation, we have found that the adaptive learning rate policy enables up to 20x faster convergence compared to a constant learning rate policy.
Keywords
Cite
@article{arxiv.2006.13307,
title = {LALR: Theoretical and Experimental validation of Lipschitz Adaptive Learning Rate in Regression and Neural Networks},
author = {Snehanshu Saha and Tejas Prashanth and Suraj Aralihalli and Sumedh Basarkod and T. S. B Sudarshan and Soma S Dhavala},
journal= {arXiv preprint arXiv:2006.13307},
year = {2020}
}
Comments
Accepted in IJCNN 2020