Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
Abstract
The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We also prove non-asymptotic convergence of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the scalability and effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.
Keywords
Cite
@article{arxiv.1902.03932,
title = {Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning},
author = {Ruqi Zhang and Chunyuan Li and Jianyi Zhang and Changyou Chen and Andrew Gordon Wilson},
journal= {arXiv preprint arXiv:1902.03932},
year = {2020}
}
Comments
Published at ICLR 2020