Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods
Abstract
We introduce a novel framework for the estimation of the posterior distribution over the weights of a neural network, based on a new probabilistic interpretation of adaptive optimisation algorithms such as AdaGrad and Adam. We demonstrate the effectiveness of our Bayesian Adam method, Badam, by experimentally showing that the learnt uncertainties correctly relate to the weights' predictive capabilities by weight pruning. We also demonstrate the quality of the derived uncertainty measures by comparing the performance of Badam to standard methods in a Thompson sampling setting for multi-armed bandits, where good uncertainty measures are required for an agent to balance exploration and exploitation.
Cite
@article{arxiv.1811.03679,
title = {Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods},
author = {Samuel Kessler and Arnold Salas and Vincent W. C. Tan and Stefan Zohren and Stephen Roberts},
journal= {arXiv preprint arXiv:1811.03679},
year = {2020}
}
Comments
Presented at the ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning