Vision-language pre-training (VLP) on large-scale datasets has shown premier performance on various downstream tasks. In contrast to plenty of available benchmarks with English corpus, large-scale pre-training datasets and downstream datasets with Chinese corpus remain largely unexplored. In this work, we build a large-scale high-quality Chinese Cross-Modal Benchmark named CCMB for the research community, which contains the currently largest public pre-training dataset Zero and five human-annotated fine-tuning datasets for downstream tasks. Zero contains 250 million images paired with 750 million text descriptions, plus two of the five fine-tuning datasets are also currently the largest ones for Chinese cross-modal downstream tasks. Along with the CCMB, we also develop a VLP framework named R2D2, applying a pre-Ranking + Ranking strategy to learn powerful vision-language representations and a two-way distillation method (i.e., target-guided Distillation and feature-guided Distillation) to further enhance the learning capability. With the Zero and the R2D2 VLP framework, we achieve state-of-the-art performance on twelve downstream datasets from five broad categories of tasks including image-text retrieval, image-text matching, image caption, text-to-image generation, and zero-shot image classification. The datasets, models, and codes are available at https://github.com/yuxie11/R2D2
@article{arxiv.2205.03860,
title = {CCMB: A Large-scale Chinese Cross-modal Benchmark},
author = {Chunyu Xie and Heng Cai and Jincheng Li and Fanjing Kong and Xiaoyu Wu and Jianfei Song and Henrique Morimitsu and Lin Yao and Dexin Wang and Xiangzheng Zhang and Dawei Leng and Baochang Zhang and Xiangyang Ji and Yafeng Deng},
journal= {arXiv preprint arXiv:2205.03860},
year = {2023}
}