Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.
@article{arxiv.2512.13921,
title = {Sliding Window Recurrences for Sequence Models},
author = {Dragos Secrieru and Garyk Brixi and Yoshua Bengio and Taiji Suzuki and Michael Poli and Stefano Massaroli},
journal= {arXiv preprint arXiv:2512.13921},
year = {2025}
}