CLIP-引导的 SAM：面向可提示分割的参数高效语义条件

可提示基座模型如 Segment Anything Model (SAM) 产生高质量掩码，但仍然在语义上感知不足，依赖外部提示来指定类别。现有的视觉-语言方法通过使用外部提示耦合来解决这一限制，其中视觉-语言模型生成的空间提示作为独立阶段供 SAM 使用。我们提出 CLIP-引导的 SAM，一个基于内部语义条件的参数高效分割框架。不使用仅用于生成提示的语义信号，而是将 CLIP 派生的文本、视觉和相似度特征直接注入 SAM 的图像编码器通过轻量级多模态语义适配器。这使得这些适配器能够影响 SAM 内部特征表示，允许语义信息影响掩码预测，同时保持 SAM 原始的可提示界面. Our framework is designed for low labeled-data settings and applies to both general-domain benchmarks and specialized downstream tasks. It supports two operating modes: Manual mode, for interactive segmentation with both text and spatial prompts, and Semi-Automatic text-only mode, for applications that require concept-specific segmentation using only textual input. We show that robustness depends on aligning training with the type of prompts used at inference, making train-test prompt consistency an important design principle. Through extensive experiments and ablations, we evaluate our method against SAM+PEFT baselines without semantic conditioning, vision-language + SAM pipelines, SAM 3, and strong semi-supervised segmentation methods that rely on large amounts of unlabeled data. Across these settings, CLIP-Guided SAM consistently achieves superior or competitive performance while remaining parameter-efficient in both training and deployment。