The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.
@article{arxiv.2605.08452,
title = {NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics},
author = {Jian Lan and Zhicheng Liu and Xinpeng Wang and Yuhao Zhou and Haokun Chen and Jiancheng Lv and Barbara Plank and Thomas Seidl},
journal= {arXiv preprint arXiv:2605.08452},
year = {2026}
}