Transformers are Universal Predictors
Machine Learning
2023-07-18 v1 Computation and Language
Abstract
We find limits to the Transformer architecture for language modeling and show it has a universal prediction property in an information-theoretic sense. We further analyze performance in non-asymptotic data regimes to understand the role of various components of the Transformer architecture, especially in the context of data-efficient training. We validate our theoretical analysis with experiments on both synthetic and real datasets.
Cite
@article{arxiv.2307.07843,
title = {Transformers are Universal Predictors},
author = {Sourya Basu and Moulik Choraria and Lav R. Varshney},
journal= {arXiv preprint arXiv:2307.07843},
year = {2023}
}
Comments
Neural Compression Workshop (ICML 2023)