Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints
Machine Learning
2025-05-27 v3 Artificial Intelligence
Machine Learning
Abstract
Many state-of-the-art models trained on long-range sequences, for example S4, S5 or LRU, are made of sequential blocks combining State-Space Models (SSMs) with neural networks. In this paper we provide a PAC bound that holds for these kind of architectures with \emph{stable} SSM blocks and does not depend on the length of the input sequence. Imposing stability of the SSM blocks is a standard practice in the literature, and it is known to help performance. Our results provide a theoretical justification for the use of stable SSM blocks as the proposed PAC bound decreases as the degree of stability of the SSM blocks increases.
Cite
@article{arxiv.2405.20278,
title = {Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints},
author = {Dániel Rácz and Mihály Petreczky and Bálint Daróczy},
journal= {arXiv preprint arXiv:2405.20278},
year = {2025}
}
Comments
preliminary version accepted at ICML 2024 Next Generation of Sequence Modeling Architectures Workshop