Autoregressive video generation enables streaming and open-ended long video synthesis, but still suffers from long-term degradation caused by accumulated errors. Existing KVCache strategies usually apply unified historical-frame retention, implicitly assuming homogeneous historical dependencies across attention heads. We revisit historical-frame attention and reveal three distinct head types: Anchor Heads require broad long-range context, Wave Heads exhibit periodic temporal dependencies, and Veil Heads focus on initial and adjacent frames. Based on this finding, we propose Pyramid Forcing, a head-aware pyramidal KVCache framework that identifies head types offline, assigns behavior-specific cache policies, and supports heterogeneous cache lengths via efficient ragged-cache attention. Experiments on Self Forcing and Causal Forcing show that Pyramid Forcing consistently improves long-horizon generation quality on VBench-Long, increasing the 60-second Self Forcing score from 77.87 to 81.21 while enhancing motion dynamics, visual fidelity, and semantic consistency. Project: https://if-lab-pku.github.io/Pyramid-Forcing/.
Cite
@article{arxiv.2605.13111,
title = {Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation},
author = {Jiayu Chen and Junbei Tang and Wenbiao Zhao and Maoliang Li and Jiayi Luo and Zihao Zheng and Jiawei Yang and Guojie Luo and Xiang Chen},
journal= {arXiv preprint arXiv:2605.13111},
year = {2026}
}