English

FullDiT: Multi-Task Video Generative Foundation Model with Full Attention

Computer Vision and Pattern Recognition 2025-03-26 v1

Abstract

Current video generative foundation models primarily focus on text-to-video tasks, providing limited control for fine-grained video content creation. Although adapter-based approaches (e.g., ControlNet) enable additional controls with minimal fine-tuning, they encounter challenges when integrating multiple conditions, including: branch conflicts between independently trained adapters, parameter redundancy leading to increased computational cost, and suboptimal performance compared to full fine-tuning. To address these challenges, we introduce FullDiT, a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. By fusing multi-task conditions into a unified sequence representation and leveraging the long-context learning ability of full self-attention to capture condition dynamics, FullDiT reduces parameter overhead, avoids conditions conflict, and shows scalability and emergent ability. We further introduce FullBench for multi-task video generation evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation.

Keywords

Cite

@article{arxiv.2503.19907,
  title  = {FullDiT: Multi-Task Video Generative Foundation Model with Full Attention},
  author = {Xuan Ju and Weicai Ye and Quande Liu and Qiulin Wang and Xintao Wang and Pengfei Wan and Di Zhang and Kun Gai and Qiang Xu},
  journal= {arXiv preprint arXiv:2503.19907},
  year   = {2025}
}

Comments

Project Page: https://fulldit.github.io/

R2 v1 2026-06-28T22:34:12.138Z