Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.
@article{arxiv.2309.09496,
title = {CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval},
author = {Yating Liu and Yaowei Li and Zimo Liu and Wenming Yang and Yaowei Wang and Qingmin Liao},
journal= {arXiv preprint arXiv:2309.09496},
year = {2024}
}
Comments
ICASSP2024(accepted). minor typos revision compared to version 1 in arxiv