Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
Abstract
Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains. Notably, in the realm of robot task planning, LLMs harness their advanced reasoning and language comprehension capabilities to formulate precise and efficient action plans based on natural language instructions. However, for embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception. This study provides a comprehensive overview of the emerging integration of LLMs and multimodal LLMs into various robotic tasks. Additionally, we propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions. Our results, based on diverse datasets, indicate that GPT-4V effectively enhances robot performance in embodied tasks. This extensive survey and evaluation of LLMs and multimodal LLMs across a variety of robotic tasks enriches the understanding of LLM-centric embodied intelligence and provides forward-looking insights toward bridging the gap in Human-Robot-Environment interaction.
Cite
@article{arxiv.2401.04334,
title = {Large Language Models for Robotics: Opportunities, Challenges, and Perspectives},
author = {Jiaqi Wang and Zihao Wu and Yiwei Li and Hanqi Jiang and Peng Shu and Enze Shi and Huawen Hu and Chong Ma and Yiheng Liu and Xuhui Wang and Yincheng Yao and Xuan Liu and Huaqin Zhao and Zhengliang Liu and Haixing Dai and Lin Zhao and Bao Ge and Xiang Li and Tianming Liu and Shu Zhang},
journal= {arXiv preprint arXiv:2401.04334},
year = {2024}
}