3-Dimensional Embodied Reference Understanding (3D-ERU) combines a language description and an accompanying pointing gesture to identify the most relevant target object in a 3D scene. Although prior work has explored pure language-based 3D grounding, there has been limited exploration of 3D-ERU, which also incorporates human pointing gestures. To address this gap, we introduce a data augmentation framework-Imputer, and use it to curate a new benchmark dataset-ImputeRefer for 3D-ERU, by incorporating human pointing gestures into existing 3D scene datasets that only contain language instructions. We also propose Ges3ViG, a novel model for 3D-ERU that achieves ~30% improvement in accuracy as compared to other 3D-ERU models and ~9% compared to other purely language-based 3D grounding models. Our code and dataset are available at https://github.com/AtharvMane/Ges3ViG.
@article{arxiv.2504.09623,
title = {Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding},
author = {Atharv Mahesh Mane and Dulanga Weerakoon and Vigneshwaran Subbaraju and Sougata Sen and Sanjay E. Sarma and Archan Misra},
journal= {arXiv preprint arXiv:2504.09623},
year = {2025}
}
Comments
Accepted to the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025