In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts, or conversely to ignore contact dynamics, focusing on grounding high-level movement in vision and language. In this work, we show that advances in generative modeling, photorealistic rendering, and procedural generation allow us to tackle tasks requiring both. By generating contact-rich trajectories with accurate physics in semantically-diverse simulations, we can distill behaviors into large multimodal models that directly transfer to the real world: a system we call Proc4Gem. Specifically, we show that a foundation model, Gemini, fine-tuned on only simulation data, can be instructed in language to control a quadruped robot to push an object with its body to unseen targets in unseen real-world environments. Our real-world results demonstrate the promise of using simulation to imbue foundation models with physical agency. Videos can be found at our website: https://sites.google.com/view/proc4gem
@article{arxiv.2503.08593,
title = {Proc4Gem: Foundation models for physical agency through procedural generation},
author = {Yixin Lin and Jan Humplik and Sandy H. Huang and Leonard Hasenclever and Francesco Romano and Stefano Saliceti and Daniel Zheng and Jose Enrique Chen and Catarina Barros and Adrian Collister and Matt Young and Adil Dostmohamed and Ben Moran and Ken Caluwaerts and Marissa Giustina and Joss Moore and Kieran Connell and Francesco Nori and Nicolas Heess and Steven Bohez and Arunkumar Byravan},
journal= {arXiv preprint arXiv:2503.08593},
year = {2025}
}