Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for fine-grained details. Although GPT-4V has shown promising results in various multi-modal tasks, leveraging GPT-4V as a generalist evaluator for these tasks has not yet been systematically explored. We comprehensively validate GPT-4V's capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. We employ two evaluation methods, single-answer grading and pairwise comparison, using GPT-4V. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators. Despite limitations like restricted visual clarity grading and real-world complex reasoning, its ability to provide human-aligned scores enriched with detailed explanations is promising for universal automatic evaluator.
@article{arxiv.2311.01361,
title = {GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks},
author = {Xinlu Zhang and Yujie Lu and Weizhi Wang and An Yan and Jun Yan and Lianke Qin and Heng Wang and Xifeng Yan and William Yang Wang and Linda Ruth Petzold},
journal= {arXiv preprint arXiv:2311.01361},
year = {2023}
}