English

Exploring Open-Vocabulary Object Recognition in Images using CLIP

Computer Vision and Pattern Recognition 2026-03-09 v1

Abstract

To address the limitations of existing open-vocabulary object recognition methods, specifically high system complexity, substantial training costs, and limited generalization, this paper proposes a novel Open-Vocabulary Object Recognition (OVOR) framework based on a streamlined two-stage strategy: object segmentation followed by recognition. The framework eliminates the need for complex retraining and labor-intensive annotation. After cropping object regions, we generate object-level image embeddings alongside category-level text embeddings using CLIP, which facilitates arbitrary vocabularies. To reduce reliance on CLIP and enhance encoding flexibility, we further introduce a CNN/MLP-based method that extracts convolutional neural network (CNN) feature maps and utilizes a multilayer perceptron (MLP) to align visual features with text embeddings. These embeddings are concatenated and processed via Singular Value Decomposition (SVD) to construct a shared representation space. Finally, recognition is performed through embedding similarity matching. Experiments on COCO, Pascal VOC, and ADE20K demonstrate that training-free, CLIP-based encoding without SVD achieves the highest average AP, outperforming current state-of-the-art methods. Simultaneously, the results highlight the potential of CNN/MLP-based image encoding for OVOR.

Keywords

Cite

@article{arxiv.2603.05962,
  title  = {Exploring Open-Vocabulary Object Recognition in Images using CLIP},
  author = {Wei Yu Chen and Ying Dai},
  journal= {arXiv preprint arXiv:2603.05962},
  year   = {2026}
}
R2 v1 2026-07-01T11:06:17.065Z