通过硬负样本增强多模态对比学习中的概念理解
摘要
当前利用对比学习的多模态模型面临在发展细粒度概念理解方面的局限。这源于在预训练期间使用随机负样本，导致损失函数中几乎只比较极不相似的概念。 Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives permute terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across a wide range of vision-language datasets, including our InpaintCOCO dataset.
引用
@article{arxiv.2403.02875,
title = {Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples},
author = {Philipp J. Rösch and Norbert Oswald and Michaela Geierhos and Jindřich Libovický},
journal= {arXiv preprint arXiv:2403.02875},
year = {2024}
}