Transformers, parallel computation, and logarithmic depth
Machine Learning
2024-02-15 v1
Abstract
We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation. As a consequence, we show that logarithmic depth is sufficient for transformers to solve basic computational tasks that cannot be efficiently solved by several other neural sequence models and sub-quadratic transformer approximations. We thus establish parallelism as a key distinguishing property of transformers.
Cite
@article{arxiv.2402.09268,
title = {Transformers, parallel computation, and logarithmic depth},
author = {Clayton Sanford and Daniel Hsu and Matus Telgarsky},
journal= {arXiv preprint arXiv:2402.09268},
year = {2024}
}
Comments
58 pages, 19 figures, code available at https://github.com/chsanford/hop-induction-heads