迈向物理场景的视觉基础模型
计算机视觉与模式识别
2023-06-07 v1 人工智能
机器学习
机器人学
摘要
我们描述了仅以图像预测作为训练准则来学习物理场景通用视觉表征的第一步。为此，我们首先定义“物理场景”，并表明尽管不同智能体对同一场景可能维持不同表征，但可推断出的底层物理场景是唯一的。接着，我们表明 NeRF 无法表征物理场景，因其缺乏外推机制。然而，这些机制至少理论上可由扩散模型（Diffusion Models）提供。为实证检验该假设，可将 NeRF 与扩散模型结合，我们称此过程为 NeRF Diffusion，用作物理场景的无监督表征。我们的分析仅限于视觉数据，不含可由独立感官模态提供的外部 grounding 机制。
引用
@article{arxiv.2306.03727,
title = {Towards Visual Foundational Models of Physical Scenes},
author = {Chethan Parameshwara and Alessandro Achille and Matthew Trager and Xiaolong Li and Jiawei Mo and Matthew Trager and Ashwin Swaminathan and CJ Taylor and Dheera Venkatraman and Xiaohan Fei and Stefano Soatto},
journal= {arXiv preprint arXiv:2306.03727},
year = {2023}
}
备注
TLDR: Physical scenes are equivalence classes of sufficient statistics, and can be inferred uniquely by any agent measuring the same finite data; We formalize and implement an approach to representation learning that overturns "naive realism" in favor of an analytical approach of Russell and Koenderink. NeRFs cannot capture the physical scenes, but combined with Diffusion Models they can