Characteristics of Monte Carlo Dropout in Wide Neural Networks
Abstract
Monte Carlo (MC) dropout is one of the state-of-the-art approaches for uncertainty estimation in neural networks (NNs). It has been interpreted as approximately performing Bayesian inference. Based on previous work on the approximation of Gaussian processes by wide and deep neural networks with random weights, we study the limiting distribution of wide untrained NNs under dropout more rigorously and prove that they as well converge to Gaussian processes for fixed sets of weights and biases. We sketch an argument that this property might also hold for infinitely wide feed-forward networks that are trained with (full-batch) gradient descent. The theory is contrasted by an empirical analysis in which we find correlations and non-Gaussian behaviour for the pre-activations of finite width NNs. We therefore investigate how (strongly) correlated pre-activations can induce non-Gaussian behavior in NNs with strongly correlated weights.
Keywords
Cite
@article{arxiv.2007.05434,
title = {Characteristics of Monte Carlo Dropout in Wide Neural Networks},
author = {Joachim Sicking and Maram Akila and Tim Wirtz and Sebastian Houben and Asja Fischer},
journal= {arXiv preprint arXiv:2007.05434},
year = {2020}
}
Comments
Accepted at the ICML 2020 workshop for Uncertainty and Robustness in Deep Learning