On the distance between two neural networks and the stability of learning
Machine Learning
2021-01-11 v3 Numerical Analysis
Neural and Evolutionary Computing
Numerical Analysis
Machine Learning
Abstract
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.
Keywords
Cite
@article{arxiv.2002.03432,
title = {On the distance between two neural networks and the stability of learning},
author = {Jeremy Bernstein and Arash Vahdat and Yisong Yue and Ming-Yu Liu},
journal= {arXiv preprint arXiv:2002.03432},
year = {2021}
}