GEM: A General Evaluation Benchmark for Multimodal Tasks

Lin Su; Nan Duan; Edward Cui; Lei Ji; Chenfei Wu; Huaishao Luo; Yongfei Liu; Ming Zhong; Taroon Bharti; Arun Sacheti

GEM: A General Evaluation Benchmark for Multimodal Tasks

Computation and Language 2021-06-21 v1 Computer Vision and Pattern Recognition Multimedia

Authors: Lin Su , Nan Duan , Edward Cui , Lei Ji , Chenfei Wu , Huaishao Luo , Yongfei Liu , Ming Zhong , Taroon Bharti , Arun Sacheti

View on arXiv ↗ PDF ↗

Abstract

In this paper, we present GEM as a General Evaluation benchmark for Multimodal tasks. Different from existing datasets such as GLUE, SuperGLUE, XGLUE and XTREME that mainly focus on natural language tasks, GEM is a large-scale vision-language benchmark, which consists of GEM-I for image-language tasks and GEM-V for video-language tasks. Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages. We also provide two baseline models for this benchmark. We will release the dataset, code and baseline models, aiming to advance the development of multilingual multimodal research.

Keywords

multimodal learning machine translation large language model

Cite

@article{arxiv.2106.09889,
  title  = {GEM: A General Evaluation Benchmark for Multimodal Tasks},
  author = {Lin Su and Nan Duan and Edward Cui and Lei Ji and Chenfei Wu and Huaishao Luo and Yongfei Liu and Ming Zhong and Taroon Bharti and Arun Sacheti},
  journal= {arXiv preprint arXiv:2106.09889},
  year   = {2021}
}

Comments

Accepted by Findings of ACL 2021

GEM: A General Evaluation Benchmark for Multimodal Tasks

Abstract

Keywords

Cite

Comments

Related papers