English

Do Joint Audio-Video Generation Models Understand Physics?

Sound 2026-05-11 v1 Artificial Intelligence Computer Vision and Pattern Recognition Multimedia

Abstract

Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency? We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench tests models across three scene categories: Steady State, Event Transition, and Environment Transition. It covers physics-grounded subcategories drawn from real-world scenes, plus Anti-AV-Physics prompts that deliberately request physically inconsistent audio-video behavior. Each generation is evaluated along five dimensions: visual semantic adherence, audio semantic adherence, visual physical commonsense, audio physical commonsense, and cross-modal physical commonsense. Across three proprietary and four open-source models, we find that Seedance 2.0 performs best overall, but all models remain far from robust physical understanding. Performance drops sharply on event-driven and environment-driven transitions, and even strong proprietary systems collapse on Anti-AV-Physics prompts. We further introduce AV-Phys Agent, a ReAct-style evaluator that combines a multimodal language model with deterministic acoustic measurement tools, producing rankings that closely align with human ratings. Our results identify cross-modal physical consistency and transition-driven scene dynamics as key open challenges for joint audio-video generation.

Keywords

Cite

@article{arxiv.2605.07061,
  title  = {Do Joint Audio-Video Generation Models Understand Physics?},
  author = {Zijun Cui and Xiulong Liu and Hao Fang and Mingwei Xu and Jiageng Liu and Zexin Xu and Weiguo Pian and Shijian Deng and Feiyu Du and Chenming Ge and Yapeng Tian},
  journal= {arXiv preprint arXiv:2605.07061},
  year   = {2026}
}

Comments

Preprint. Full abstract appears in the PDF

R2 v1 2026-07-01T12:56:34.717Z