On Connected Sublevel Sets in Deep Learning
Machine Learning
2019-05-15 v2 Machine Learning
Abstract
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.
Keywords
Cite
@article{arxiv.1901.07417,
title = {On Connected Sublevel Sets in Deep Learning},
author = {Quynh Nguyen},
journal= {arXiv preprint arXiv:1901.07417},
year = {2019}
}
Comments
Accepted at ICML 2019. More discussions and visualizations are added