Recent studies have identified Direct Preference Optimization (DPO) as an efficient and reward-free approach to improving video generation quality. However, existing methods largely follow image-domain paradigms and are mainly developed on small-scale models (approximately 2B parameters), limiting their ability to address the unique challenges of video tasks, such as costly data construction, unstable training, and heavy memory consumption. To overcome these limitations, we introduce a GT-Pair that automatically builds high-quality preference pairs by using real videos as positives and model-generated videos as negatives, eliminating the need for any external annotation. We further present Reg-DPO, which incorporates the SFT loss as a regularization term into the DPO loss to enhance training stability and generation fidelity. Additionally, by combining the FSDP framework with multiple memory optimization techniques, our approach achieves nearly three times higher training capacity than using FSDP alone. Extensive experiments on both I2V and T2V tasks across multiple datasets demonstrate that our method consistently outperforms existing approaches, delivering superior video generation quality.
@article{arxiv.2511.01450,
title = {Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation},
author = {Jie Du and Xinyu Gong and Qingshan Tan and Wen Li and Yangming Cheng and Weitao Wang and Chenlu Zhan and Suhui Wu and Hao Zhang and Jun Zhang},
journal= {arXiv preprint arXiv:2511.01450},
year = {2025}
}
Comments
The paper is withdrawn due to the need for further revision and verification of experimental results. A revised version will be resubmitted once the updates are completed