We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.
Cite
@article{arxiv.2604.24878,
title = {Transformer Approximations from ReLUs},
author = {Jerry Yao-Chieh Hu and Mingcheng Lu and Yi-Chen Lee and Han Liu},
journal= {arXiv preprint arXiv:2604.24878},
year = {2026}
}