English

One-layer transformers fail to solve the induction heads task

Machine Learning 2024-08-27 v1 Machine Learning
Authors: Clayton Sanford , Daniel Hsu , Matus Telgarsky
View on arXiv ↗ PDF ↗

Abstract

A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.

Cite

@article{arxiv.2408.14332,
  title  = {One-layer transformers fail to solve the induction heads task},
  author = {Clayton Sanford and Daniel Hsu and Matus Telgarsky},
  journal= {arXiv preprint arXiv:2408.14332},
  year   = {2024}
}

Related papers

View all related →
Machine Learning · Computer Science
Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)
Lena Strobl, Dana Angluin, Robert Frank
2025-03-31
Machine Learning · Computer Science
Theoretical limitations of multi-layer Transformer
Lijie Chen, Binghui Peng, Hongxun Wu
2024-12-05
Machine Learning · Computer Science
Parity, Sensitivity, and Transformers
Alexander Kozachinskiy, Tomasz Steifer, Przemysław Wałȩga
2026-05-08
Machine Learning · Computer Science
Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
Tiberiu Musat
2025-10-29
Computation and Language · Computer Science
Transformers Struggle to Learn to Search
Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi +5
2025-03-18
Machine Learning · Computer Science
What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks
Xingwu Chen, Difan Zou
2024-04-03
Computation and Language · Computer Science
Faith and Fate: Limits of Transformers on Compositionality
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li +12
2023-11-01
Machine Learning · Computer Science
Two Heads Are Better than One: Simulating Large Transformers with Small Ones
Hantao Yu, Josh Alman
2025-06-23
Computation and Language · Computer Science
Dissecting Multiplication in Transformers: Insights into LLMs
Luyu Qiu, Jianing Li, Chi Su, Chen Jason Zhang +1
2024-07-23
Machine Learning · Computer Science
A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
William Merrill, Ashish Sabharwal
2025-11-07
Machine Learning · Computer Science
Attention-Only Transformers and Implementing MLPs with Attention Heads
Robert Huben, Valerie Morris
2023-09-18
Computation and Language · Computer Science
Transformers Can Learn Connectivity in Some Graphs but Not Others
Amit Roy, Abulhair Saparov
2026-04-23
Machine Learning · Computer Science
Lower bounds for one-layer transformers that compute parity
Daniel Hsu
2026-05-13
Machine Learning · Computer Science
Representational Strengths and Limitations of Transformers
Clayton Sanford, Daniel Hsu, Matus Telgarsky
2023-11-17
Machine Learning · Computer Science
Transformers Linearly Represent Highly Structured World Models
Roman Kniazev, Nathanaël Fijalkow
2026-05-20
Machine Learning · Computer Science
Why are Sensitive Functions Hard for Transformers?
Michael Hahn, Mark Rofin
2024-05-28
Computation and Language · Computer Science
Transformer Layers as Painters
Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones
2025-02-14
Computational Complexity · Computer Science
On the Computational Hardness of Transformers
Barna Saha, Yinzhan Xu, Christopher Ye, Hantao Yu
2026-03-13
Machine Learning · Computer Science
Transformers Provably Learn Algorithmic Solutions for Graph Connectivity, But Only with the Right Data
Qilin Ye, Deqing Fu, Robin Jia, Vatsal Sharan
2026-02-19
Machine Learning · Computer Science
Transformers converge to invariant algorithmic cores
Joshua S. Schiffman
2026-02-27
Machine Learning · Computer Science
Strassen Attention, Split VC Dimension and Compositionality in Transformers
Alexander Kozachinskiy, Felipe Urrutia, Hector Jimenez, Tomasz Steifer +5
2025-09-26
Computation and Language · Computer Science
An Efficient Transformer Decoder with Compressed Sub-layers
Yanyang Li, Ye Lin, Tong Xiao, Jingbo Zhu
2023-05-12
Machine Learning · Computer Science
Looped Transformers are Better at Learning Learning Algorithms
Liu Yang, Kangwook Lee, Robert Nowak, Dimitris Papailiopoulos
2024-03-19
Machine Learning · Computer Science
The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It
Gabriel Garcia
2026-05-18
Machine Learning · Computer Science
Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
Hanseul Cho, Jaeyoung Cha, Srinadh Bhojanapalli, Chulhee Yun
2025-04-18
R2 v1 2026-06-28T18:24:04.767Z