English

Precomputing the Future-Offset Average in TriAttention

Data Structures and Algorithms 2026-07-01 v1 Machine Learning

Abstract

TriAttention is a recent method for shrinking the KV cache of long-reasoning LLMs: it scores each cached key by how much attention it is likely to receive and evicts the lowest-scoring ones. Because a key does not know how far away its future queries will sit, the score is averaged over a ladder of 17 possible future distances. We point out that this average is free: the future distance enters the score only through the position-dependent rotation, so the whole 17-fold average collapses--exactly, by a one-line algebraic identity--into a single per-band weight that is computed once, offline. Scoring a key then costs one evaluation instead of seventeen, with no change to which keys get pruned. The saving is modest and lives entirely in TriAttention's pruning-score computation, not in the attention kernel; we present it as a small, exact complement to their method, and we confirm the identity numerically.

Cite

@article{arxiv.2607.13051,
  title  = {Precomputing the Future-Offset Average in TriAttention},
  author = {Amarnath Mukherjee},
  journal= {arXiv preprint arXiv:2607.13051},
  year   = {2026}
}

Comments

8 pages, 2 figures