Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
Abstract
From image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which prove effective for tasks like visual question answering. However, leveraging the learned association for open-vocabulary semantic segmentation remains a challenge. In this paper, we propose a simple, yet extremely effective, training-free technique, Plug-and-Play Open-Vocabulary Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with direct text-to-image cross-attention and an image-text matching loss. To balance between over-segmentation and under-segmentation, we introduce Salience Dropout; by iteratively dropping patches that the model is most attentive to, we are able to better resolve the entire extent of the segmentation mask. PnP-OVSS does not require any neural network training and performs hyperparameter tuning without the need for any segmentation annotations, even for a validation set. PnP-OVSS demonstrates substantial improvements over comparable baselines (+26.2% mIoU on Pascal VOC, +20.5% mIoU on MS COCO, +3.1% mIoU on COCO Stuff and +3.0% mIoU on ADE20K). Our codebase is at https://github.com/letitiabanana/PnP-OVSS.
Keywords
Cite
@article{arxiv.2311.17095,
title = {Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models},
author = {Jiayun Luo and Siddhesh Khandelwal and Leonid Sigal and Boyang Li},
journal= {arXiv preprint arXiv:2311.17095},
year = {2024}
}
Comments
Accepted to CVPR 2024; Earlier version of this paper contained an unintentional error stemming from a bug in the code. This version corrects this error, which had to do with filtering of class names. In consultation with CVPR Program Chairs it was suggested errata be submitted as the updated (fixed) code reinforced original findings (albeit with slightly different final numbers)