We introduce the challenging problem of multi-object system identification from videos, for which prior methods are ill-suited due to their focus on single-object scenes or discrete material classification with a fixed set of material prototypes. To address this, we propose MOSIV, a new framework that directly optimizes for continuous, per-object material parameters using a differentiable simulator guided by geometric objectives derived from video. We also present a new synthetic benchmark with contact-rich, multi-object interactions to facilitate evaluation. On this benchmark, MOSIV substantially improves grounding accuracy and long-horizon simulation fidelity over adapted baselines, establishing it as a strong baseline for this new task. Our analysis shows that object-level fine-grained supervision and geometry-aligned objectives are critical for stable optimization in these complex, multi-object settings. The source code and dataset will be released.
@article{arxiv.2603.06022,
title = {MOSIV: Multi-Object System Identification from Videos},
author = {Chunjiang Liu and Xiaoyuan Wang and Qingran Lin and Albert Xiao and Haoyu Chen and Shizheng Wen and Hao Zhang and Lu Qi and Ming-Hsuan Yang and Laszlo A. Jeni and Min Xu and Yizhou Zhao},
journal= {arXiv preprint arXiv:2603.06022},
year = {2026}
}