With its powerful visual-language alignment capability, CLIP performs well in zero-shot and few-shot learning tasks. However, we found in experiments that CLIP's logits suffer from serious inter-class confusion problems in downstream tasks, and the ambiguity between categories seriously affects the accuracy. To address this challenge, we propose a novel method called Logits DeConfusion, which effectively learns and eliminates inter-class confusion in logits by combining our Multi-level Adapter Fusion (MAF) module with our Inter-Class Deconfusion (ICD) module. Our MAF extracts features from different levels and fuses them uniformly to enhance feature representation. Our ICD learnably eliminates inter-class confusion in logits with a residual structure. Experimental results show that our method can significantly improve the classification performance and alleviate the inter-class confusion problem. The code is available at https://github.com/LiShuo1001/LDC.
@article{arxiv.2504.12104,
title = {Logits DeConfusion with CLIP for Few-Shot Learning},
author = {Shuo Li and Fang Liu and Zehua Hao and Xinyi Wang and Lingling Li and Xu Liu and Puhua Chen and Wenping Ma},
journal= {arXiv preprint arXiv:2504.12104},
year = {2025}
}