ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Abstract
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.
Keywords
Cite
@article{arxiv.2007.04954,
title = {ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation},
author = {Chuang Gan and Jeremy Schwartz and Seth Alter and Damian Mrowca and Martin Schrimpf and James Traer and Julian De Freitas and Jonas Kubilius and Abhishek Bhandwaldar and Nick Haber and Megumi Sano and Kuno Kim and Elias Wang and Michael Lingelbach and Aidan Curtis and Kevin Feigelis and Daniel M. Bear and Dan Gutfreund and David Cox and Antonio Torralba and James J. DiCarlo and Joshua B. Tenenbaum and Josh H. McDermott and Daniel L. K. Yamins},
journal= {arXiv preprint arXiv:2007.04954},
year = {2021}
}
Comments
Oral Presentation at NeurIPS 21 Datasets and Benchmarks Track. Project page: http://www.threedworld.org