Direct Preference Optimization (DPO) is successful for alignment in LLMs but still faces challenges in text-to-image generation. Existing studies are confined to denoising diffusion models while overlooking flow-matching, and suffer from an objective mismatch when applying discrete NLP-based DPO to regression-based generative tasks.\ In this paper, we derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework, and point out from a gradient perspective that the standard DPO objective is suboptimal for text-to-image generation. Consequently, we propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model. Qualitative and quantitative experiments on diffusion models (SD1.5, SDXL) and flow-matching model (SD3-Medium) demonstrate the superiority of our approach over existing baselines.
@article{arxiv.2605.21123,
title = {Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models},
author = {Kesong Li and Yixuan Xu and Kuo-kun Tseng and Weiyi Lu and Kan Liu and Tao Lan},
journal= {arXiv preprint arXiv:2605.21123},
year = {2026}
}
Comments
Code and models are available at: https://github.com/Whynot0101/Linear-DPO . Work done during an internship at Alibaba Group