We present EgoAllo, a system for human motion estimation from a head-mounted device. Using only egocentric SLAM poses and images, EgoAllo guides sampling from a conditional diffusion model to estimate 3D body pose, height, and hand parameters that capture a device wearer's actions in the allocentric coordinate frame of the scene. To achieve this, our key insight is in representation: we propose spatial and temporal invariance criteria for improving model performance, from which we derive a head motion conditioning parameterization that improves estimation by up to 18%. We also show how the bodies estimated by our system can improve hand estimation: the resulting kinematic and temporal constraints can reduce world-frame errors in single-frame estimates by 40%. Project page: https://egoallo.github.io/
Cite
@article{arxiv.2410.03665,
title = {Estimating Body and Hand Motion in an Ego-sensed World},
author = {Brent Yi and Vickie Ye and Maya Zheng and Yunqi Li and Lea Müller and Georgios Pavlakos and Yi Ma and Jitendra Malik and Angjoo Kanazawa},
journal= {arXiv preprint arXiv:2410.03665},
year = {2024}
}