Large Deviations of Gaussian Neural Networks with ReLU activation
Machine Learning
2026-02-10 v3 Machine Learning
Probability
Abstract
We prove a large deviation principle for deep neural networks with Gaussian weights and at most linearly growing activation functions, such as ReLU. This generalises earlier work, in which bounded and continuous activation functions were considered. In practice, linearly growing activation functions such as ReLU are most commonly used. We furthermore simplify previous expressions for the rate function and provide a power-series expansions for the ReLU case.
Keywords
Cite
@article{arxiv.2405.16958,
title = {Large Deviations of Gaussian Neural Networks with ReLU activation},
author = {Quirin Vogel},
journal= {arXiv preprint arXiv:2405.16958},
year = {2026}
}
Comments
typo corrected from a previous version