Lower bounds on transformers with infinite precision
Machine Learning
2024-12-31 v1 Artificial Intelligence
Machine Learning
Abstract
In this note, we use the VC dimension technique to prove the first lower bound against one-layer softmax transformers with infinite precision. We do so for two tasks: function composition, considered by Peng, Narayanan, and Papadimitriou, and the SUM task, considered by Sanford, Hsu, and Telgarsky.
Cite
@article{arxiv.2412.20195,
title = {Lower bounds on transformers with infinite precision},
author = {Alexander Kozachinskiy},
journal= {arXiv preprint arXiv:2412.20195},
year = {2024}
}