A PAC-Bayes oracle inequality for sparse neural networks
Statistics Theory
2026-01-09 v2 Machine Learning
Statistics Theory
Abstract
We study the Gibbs posterior distribution for sparse deep neural nets in a nonparametric regression setting. The posterior can be accessed via Metropolis-adjusted Langevin algorithms. Using a mixture over uniform priors on sparse sets of network weights, we prove an oracle inequality which shows that the method adapts to the unknown regularity and hierarchical structure of the regression function. The estimator achieves the minimax-optimal rate of convergence (up to a logarithmic factor).
Cite
@article{arxiv.2204.12392,
title = {A PAC-Bayes oracle inequality for sparse neural networks},
author = {Maximilian F. Steffen and Mathias Trabs},
journal= {arXiv preprint arXiv:2204.12392},
year = {2026}
}