English

PhyCo: Learning Controllable Physical Priors for Generative Motion

Computer Vision and Pattern Recognition 2026-05-01 v1 Artificial Intelligence Machine Learning

Abstract

Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.

Keywords

Cite

@article{arxiv.2604.28169,
  title  = {PhyCo: Learning Controllable Physical Priors for Generative Motion},
  author = {Sriram Narayanan and Ziyu Jiang and Srinivasa Narasimhan and Manmohan Chandraker},
  journal= {arXiv preprint arXiv:2604.28169},
  year   = {2026}
}

Comments

CVPR 2026. Project Page: https://phyco-video.github.io/

R2 v1 2026-07-01T12:44:06.889Z