Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
@article{arxiv.2603.03596,
title = {MEM: Multi-Scale Embodied Memory for Vision Language Action Models},
author = {Marcel Torne and Karl Pertsch and Homer Walke and Kyle Vedder and Suraj Nair and Brian Ichter and Allen Z. Ren and Haohuan Wang and Jiaming Tang and Kyle Stachowicz and Karan Dhabalia and Michael Equi and Quan Vuong and Jost Tobias Springenberg and Sergey Levine and Chelsea Finn and Danny Driess},
journal= {arXiv preprint arXiv:2603.03596},
year = {2026}
}