Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context pretraining by 36% while matching baseline loss, requiring no architectural changes. MuSe clusters queries and keys separately in representation space. This yields query-specific summaries that substantially outperform spatial blocking at matched sparsity, while also enabling drop-in compatibility with existing pretrained models; we validate on Llama 3.1-8B and 3.2-1B without retraining. We pretrain language models up to 1B parameters at 64k context on code and scientific documents, confirming that MuSe preserves quality and long-context utilization during training.
@article{arxiv.2509.10406,
title = {Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining},
author = {Rupert Mitchell and Kristian Kersting},
journal= {arXiv preprint arXiv:2509.10406},
year = {2026}
}