Regularising Deep Networks with Deep Generative Models
Abstract
We develop a new method for regularising neural networks. We learn a probability distribution over the activations of all layers of the model and then insert imputed values into the network during training. We obtain a posterior for an arbitrary subset of activations conditioned on the remainder. This is a generalisation of data augmentation to the hidden layers of a network, and a form of data-aware dropout. We demonstrate that our training method leads to higher test accuracy and lower test-set cross-entropy for neural networks trained on CIFAR-10 and SVHN compared to standard regularisation baselines: our approach leads to networks with better calibrated uncertainty over the class posteriors all the while delivering greater test-set accuracy.
Cite
@article{arxiv.1909.11507,
title = {Regularising Deep Networks with Deep Generative Models},
author = {Matthew Willetts and Alexander Camuto and Stephen Roberts and Chris Holmes},
journal= {arXiv preprint arXiv:1909.11507},
year = {2019}
}
Comments
8 pages plus appendix