Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen
@article{arxiv.2502.01776,
title = {Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity},
author = {Haocheng Xi and Shuo Yang and Yilong Zhao and Chenfeng Xu and Muyang Li and Xiuyu Li and Yujun Lin and Han Cai and Jintao Zhang and Dacheng Li and Jianfei Chen and Ion Stoica and Kurt Keutzer and Song Han},
journal= {arXiv preprint arXiv:2502.01776},
year = {2025}
}