Structured Weight Priors for Convolutional Neural Networks
Abstract
Selection of an architectural prior well suited to a task (e.g. convolutions for image data) is crucial to the success of deep neural networks (NNs). Conversely, the weight priors within these architectures are typically left vague, e.g.~independent Gaussian distributions, which has led to debate over the utility of Bayesian deep learning. This paper explores the benefits of adding structure to weight priors. It initially considers first-layer filters of a convolutional NN, designing a prior based on random Gabor filters. Second, it considers adding structure to the prior of final-layer weights by estimating how each hidden feature relates to each class. Empirical results suggest that these structured weight priors lead to more meaningful functional priors for image data. This contributes to the ongoing discussion on the importance of weight priors.
Cite
@article{arxiv.2007.14235,
title = {Structured Weight Priors for Convolutional Neural Networks},
author = {Tim Pearce and Andrew Y. K. Foong and Alexandra Brintrup},
journal= {arXiv preprint arXiv:2007.14235},
year = {2020}
}
Comments
Presented at the ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning