LLM-Driven Augmented Reality Puppeteer: Controller-Free Voice-Commanded Robot Teleoperation
Abstract
The integration of robotics and augmented reality (AR) presents transformative opportunities for advancing human-robot interaction (HRI) by improving usability, intuitiveness, and accessibility. This work introduces a controller-free, LLM-driven voice-commanded AR puppeteering system, enabling users to teleoperate a robot by manipulating its virtual counterpart in real time. By leveraging natural language processing (NLP) and AR technologies, our system -- prototyped using Meta Quest 3 -- eliminates the need for physical controllers, enhancing ease of use while minimizing potential safety risks associated with direct robot operation. A preliminary user demonstration successfully validated the system's functionality, demonstrating its potential for safer, more intuitive, and immersive robotic control.
Keywords
Cite
@article{arxiv.2502.09142,
title = {LLM-Driven Augmented Reality Puppeteer: Controller-Free Voice-Commanded Robot Teleoperation},
author = {Yuchong Zhang and Bastian Orthmann and Michael C. Welle and Jonne Van Haastregt and Danica Kragic},
journal= {arXiv preprint arXiv:2502.09142},
year = {2025}
}
Comments
Accepted as conference proceeding in International Conference on Human-Computer Interaction 2025 (HCI International 2025)