Sequence-Aware Split Heuristic to Mitigate SM Underutilization in FlashAttention-3 Low-Head-Count Decoding

Martí Llopart Font; Javier Hernando; Cristina España-Bonet

Sequence-Aware Split Heuristic to Mitigate SM Underutilization in FlashAttention-3 Low-Head-Count Decoding

Hardware Architecture 2026-04-02 v1 Distributed, Parallel, and Cluster Computing

Authors: Martí Llopart Font , Javier Hernando , Cristina España-Bonet

Abstract

The standard FlashAttention-3 heuristic exhibits a GPU occupancy bottleneck in low-head-count decoding configurations because it disables sequence splitting based on sequence length alone, underutilizing the Streaming Multiprocessors of Hopper GPUs. Our proposed sequence-aware split policy mitigates this by allowing sequence-level parallelism in low-head-count regimes, improving hardware utilization to deliver roughly a 21 to 24% improvement in decoder kernel efficiency on metadata-enabled inference paths, with no observed regressions.

Keywords

memory hierarchy computer architecture concurrent algorithm

Cite

@article{arxiv.2604.00028,
  title  = {Sequence-Aware Split Heuristic to Mitigate SM Underutilization in FlashAttention-3 Low-Head-Count Decoding},
  author = {Martí Llopart Font and Javier Hernando and Cristina España-Bonet},
  journal= {arXiv preprint arXiv:2604.00028},
  year   = {2026}
}

Related papers

View all related →