跨视图世界模型
摘要
世界模型使智能体能够通过想象未来状态来进行规划，但现有方法仅从单一视角（通常是第三人称 egocentric 视角）运行，即使其他视角会更容易进行规划；例如，导航受鸟瞰图的帮助。我们引入跨视图世界模型 (Cross-View World Models, XVWM)，通过跨视图预测目标进行训练：给定来自一个视角的帧序列，预测在执行某个动作后，从同一视角或不同视角获得的未来状态。通过跨视图一致性约束实现几何正则化：由于输入和输出视角可能共享很少或没有视觉重叠，为了跨视角进行预测，模型必须学习环境 3D 结构的视角不变表示。我们在来自 Aimlabs 的同步多视图游戏数据上进行训练，Aimlabs 是一个提供精确对齐的多摄像头录像和高频动作标签的 aim 训练平台。 resulting model gives agents parallel imagination streams across viewpoints, enabling planning in whichever frame of reference best suits the task while executing from the egocentric view. Our results show that multi-view consistency provides a strong learning signal for spatially grounded representations. Finally, predicting the consequences of one's actions from another viewpoint may offer a foundation for perspective-taking in multi-agent settings.
引用
@article{arxiv.2602.07277,
title = {Cross-View World Models},
author = {Rishabh Sharma and Gijs Hogervorst and Wayne E. Mackey and David J. Heeger and Stefano Martiniani},
journal= {arXiv preprint arXiv:2602.07277},
year = {2026}
}
备注
12 pages, 7 figures