Towards Visual Foundational Models of Physical Scenes
Abstract
We describe a first step towards learning general-purpose visual representations of physical scenes using only image prediction as a training criterion. To do so, we first define "physical scene" and show that, even though different agents may maintain different representations of the same scene, the underlying physical scene that can be inferred is unique. Then, we show that NeRFs cannot represent the physical scene, as they lack extrapolation mechanisms. Those, however, could be provided by Diffusion Models, at least in theory. To test this hypothesis empirically, NeRFs can be combined with Diffusion Models, a process we refer to as NeRF Diffusion, used as unsupervised representations of the physical scene. Our analysis is limited to visual data, without external grounding mechanisms that can be provided by independent sensory modalities.
Cite
@article{arxiv.2306.03727,
title = {Towards Visual Foundational Models of Physical Scenes},
author = {Chethan Parameshwara and Alessandro Achille and Matthew Trager and Xiaolong Li and Jiawei Mo and Matthew Trager and Ashwin Swaminathan and CJ Taylor and Dheera Venkatraman and Xiaohan Fei and Stefano Soatto},
journal= {arXiv preprint arXiv:2306.03727},
year = {2023}
}
Comments
TLDR: Physical scenes are equivalence classes of sufficient statistics, and can be inferred uniquely by any agent measuring the same finite data; We formalize and implement an approach to representation learning that overturns "naive realism" in favor of an analytical approach of Russell and Koenderink. NeRFs cannot capture the physical scenes, but combined with Diffusion Models they can