English

A Reason-then-Describe Instruction Interpreter for Controllable Video Generation

Computer Vision and Pattern Recognition 2025-11-26 v1

Abstract

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.

Keywords

Cite

@article{arxiv.2511.20563,
  title  = {A Reason-then-Describe Instruction Interpreter for Controllable Video Generation},
  author = {Shengqiong Wu and Weicai Ye and Yuanxing Zhang and Jiahao Wang and Quande Liu and Xintao Wang and Pengfei Wan and Kun Gai and Hao Fei and Tat-Seng Chua},
  journal= {arXiv preprint arXiv:2511.20563},
  year   = {2025}
}

Comments

27 pages, 13 figures, 13 tables, Project Page: https://sqwu.top/ReaDe/

R2 v1 2026-07-01T07:54:39.743Z