How good is PAC-Bayes at explaining generalisation?
Machine Learning
2025-03-12 v1 Machine Learning
Abstract
We discuss necessary conditions for a PAC-Bayes bound to provide a meaningful generalisation guarantee. Our analysis reveals that the optimal generalisation guarantee depends solely on the distribution of the risk induced by the prior distribution. In particular, achieving a target generalisation level is only achievable if the prior places sufficient mass on high-performing predictors. We relate these requirements to the prevalent practice of using data-dependent priors in deep learning PAC-Bayes applications, and discuss the implications for the claim that PAC-Bayes ``explains'' generalisation.
Cite
@article{arxiv.2503.08231,
title = {How good is PAC-Bayes at explaining generalisation?},
author = {Antoine Picard-Weibel and Eugenio Clerico and Roman Moscoviz and Benjamin Guedj},
journal= {arXiv preprint arXiv:2503.08231},
year = {2025}
}