Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.
@article{arxiv.2409.06851,
title = {LIME: Less Is More for MLLM Evaluation},
author = {King Zhu and Qianbo Zang and Shian Jia and Siwei Wu and Feiteng Fang and Yizhi Li and Shawn Gavin and Tuney Zheng and Jiawei Guo and Bo Li and Haoning Wu and Xingwei Qu and Jian Yang and Zachary Liu and Xiang Yue and J. H. Liu and Chenghua Lin and Min Yang and Shiwen Ni and Wenhao Huang and Ge Zhang},
journal= {arXiv preprint arXiv:2409.06851},
year = {2024}
}