What Foundation Models can Bring for Robot Learning in Manipulation : A Survey
Abstract
The realization of universal robots is an ultimate goal of researchers. However, a key hurdle in achieving this goal lies in the robots' ability to manipulate objects in their unstructured surrounding environments according to different tasks. The learning-based approach is considered an effective way to address generalization. The impressive performance of foundation models in the fields of computer vision and natural language suggests the potential of embedding foundation models into manipulation tasks as a viable path toward achieving general manipulation capability. However, we believe achieving general manipulation capability requires an overarching framework akin to auto driving. This framework should encompass multiple functional modules, with different foundation models assuming distinct roles in facilitating general manipulation capability. This survey focuses on the contributions of foundation models to robot learning for manipulation. We propose a comprehensive framework and detail how foundation models can address challenges in each module of the framework. What's more, we examine current approaches, outline challenges, suggest future research directions, and identify potential risks associated with integrating foundation models into this domain.
Cite
@article{arxiv.2404.18201,
title = {What Foundation Models can Bring for Robot Learning in Manipulation : A Survey},
author = {Dingzhe Li and Yixiang Jin and Yuhao Sun and Yong A and Hongze Yu and Jun Shi and Xiaoshuai Hao and Peng Hao and Huaping Liu and Xiang Li and Xinde Li and Fuchun Sun and Jianwei Zhang and Bin Fang},
journal= {arXiv preprint arXiv:2404.18201},
year = {2025}
}