ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO
Abstract
Image-Text matching (ITM) is a common task for evaluating the quality of Vision and Language (VL) models. However, existing ITM benchmarks have a significant limitation. They have many missing correspondences, originating from the data construction process itself. For example, a caption is only matched with one image although the caption can be matched with other similar images and vice versa. To correct the massive false negatives, we construct the Extended COCO Validation (ECCV) Caption dataset by supplying the missing associations with machine and human annotators. We employ five state-of-the-art ITM models with diverse properties for our annotation process. Our dataset provides x3.6 positive image-to-caption associations and x8.5 caption-to-image associations compared to the original MS-COCO. We also propose to use an informative ranking-based metric mAP@R, rather than the popular Recall@K (R@K). We re-evaluate the existing 25 VL models on existing and proposed benchmarks. Our findings are that the existing benchmarks, such as COCO 1K R@K, COCO 5K R@K, CxC R@1 are highly correlated with each other, while the rankings change when we shift to the ECCV mAP@R. Lastly, we delve into the effect of the bias introduced by the choice of machine annotator. Source code and dataset are available at https://github.com/naver-ai/eccv-caption
Cite
@article{arxiv.2204.03359,
title = {ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO},
author = {Sanghyuk Chun and Wonjae Kim and Song Park and Minsuk Chang and Seong Joon Oh},
journal= {arXiv preprint arXiv:2204.03359},
year = {2024}
}
Comments
Published in ECCV 2022; 32 pages (2.3MB); Code and dataset: https://github.com/naver-ai/eccv-caption; v5 fixes errors in Table 4: the COCO 1K R@1 numbers were incorrect. All other tables and figures are correct. v5 also adds RSUM scores in Tab 4 and 5: RSUM has a high correlation with COCO 1K recalls; v4 fixes errors in v3 -- see the v4 comment for details