Scaling Instructable Agents Across Many Simulated Worlds
Abstract
Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructions across a diverse range of virtual 3D environments, including curated research environments as well as open-ended, commercial video games. Our goal is to develop an instructable agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface: the inputs are image observations and language instructions and the outputs are keyboard-and-mouse actions. This general approach is challenging, but it allows agents to ground language across many visually complex and semantically rich environments while also allowing us to readily run agents in new environments. In this paper we describe our motivation and goal, the initial progress we have made, and promising preliminary results on several diverse research environments and a variety of commercial video games.
Cite
@article{arxiv.2404.10179,
title = {Scaling Instructable Agents Across Many Simulated Worlds},
author = {SIMA Team and Maria Abi Raad and Arun Ahuja and Catarina Barros and Frederic Besse and Andrew Bolt and Adrian Bolton and Bethanie Brownfield and Gavin Buttimore and Max Cant and Sarah Chakera and Stephanie C. Y. Chan and Jeff Clune and Adrian Collister and Vikki Copeman and Alex Cullum and Ishita Dasgupta and Dario de Cesare and Julia Di Trapani and Yani Donchev and Emma Dunleavy and Martin Engelcke and Ryan Faulkner and Frankie Garcia and Charles Gbadamosi and Zhitao Gong and Lucy Gonzales and Kshitij Gupta and Karol Gregor and Arne Olav Hallingstad and Tim Harley and Sam Haves and Felix Hill and Ed Hirst and Drew A. Hudson and Jony Hudson and Steph Hughes-Fitt and Danilo J. Rezende and Mimi Jasarevic and Laura Kampis and Rosemary Ke and Thomas Keck and Junkyung Kim and Oscar Knagg and Kavya Kopparapu and Rory Lawton and Andrew Lampinen and Shane Legg and Alexander Lerchner and Marjorie Limont and Yulan Liu and Maria Loks-Thompson and Joseph Marino and Kathryn Martin Cussons and Loic Matthey and Siobhan Mcloughlin and Piermaria Mendolicchio and Hamza Merzic and Anna Mitenkova and Alexandre Moufarek and Valeria Oliveira and Yanko Oliveira and Hannah Openshaw and Renke Pan and Aneesh Pappu and Alex Platonov and Ollie Purkiss and David Reichert and John Reid and Pierre Harvey Richemond and Tyson Roberts and Giles Ruscoe and Jaume Sanchez Elias and Tasha Sandars and Daniel P. Sawyer and Tim Scholtes and Guy Simmons and Daniel Slater and Hubert Soyer and Heiko Strathmann and Peter Stys and Allison C. Tam and Denis Teplyashin and Tayfun Terzi and Davide Vercelli and Bojan Vujatovic and Marcus Wainwright and Jane X. Wang and Zhengdong Wang and Daan Wierstra and Duncan Williams and Nathaniel Wong and Sarah York and Nick Young},
journal= {arXiv preprint arXiv:2404.10179},
year = {2024}
}