On the Implicit Bias of Dropout
Machine Learning
2018-06-27 v1 Artificial Intelligence
Machine Learning
Abstract
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.
Cite
@article{arxiv.1806.09777,
title = {On the Implicit Bias of Dropout},
author = {Poorya Mianjy and Raman Arora and Rene Vidal},
journal= {arXiv preprint arXiv:1806.09777},
year = {2018}
}
Comments
17 pages, 3 figures, In Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML), 2018