将 LMM 规划器与 3D 技能策略集成以实现可泛化的操纵
摘要
近期大型多模态模型（LMM）在视觉推理能力方面的突破，以及 3D 特征场的语义丰富化，正在拓展机器人能力的边界。这些发展为连接 LMM 的高层推理与利用 3D 特征场的低层控制策略具有巨大潜力。本文引入了 LMM-3DP 框架，用于集成 LMM 规划器和 3D 技能策略。我们的 метод consists of three key perspectives: high-level planning, low-level control, and effective integration。 For high-level planning, LMM-3DP supports dynamic scene understanding for environment disturbances, a critic agent with self-feedback, history policy memorization, and reattempts after failures。 For low-level control, LMM-3DP utilizes a semantic-aware 3D feature field for accurate manipulation。 In aligning high-level and low-level control for robot actions, language embeddings representing the high-level policy are jointly attended with the 3D feature field in the 3D transformer for seamless integration。我们在真实厨房环境中对多种技能和长期限任务进行了广泛评估。结果表明，低层控制成功率提升了约 1.45 倍，高层规划准确率相对于基于 LLM 的基线提升约 1.5 倍。演示视频及 LMM-3DP概览可在 https://lmm-3dp-release.github.io 查看。
引用
@article{arxiv.2501.18733,
title = {Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation},
author = {Yuelei Li and Ge Yan and Annabella Macaluso and Mazeyu Ji and Xueyan Zou and Xiaolong Wang},
journal= {arXiv preprint arXiv:2501.18733},
year = {2025}
}