Kaleidoscope：面向大规模多语言视觉评估的内部语言考试
计算与语言
2025-04-30 v2 计算机视觉与模式识别
摘要
视觉语言模型（VLM）的评估主要依赖英文基准测试，在多语言和多文化覆盖方面存在显著不足。虽然多语言基准测试在规模和语言方面有所扩展，但许多依赖英文数据集的翻译版本，无法捕捉文化细微差别。在本工作中，我们提出Kaleidoscope，作为目前最全面的多语言视觉语言模型评估基准。Kaleidoscope是一个大规模、面向多语言评估的多模态基准，旨在评估VLM在不同语言和视觉输入下的表现。Kaleidoscope覆盖18种语言和14个不同学科，总计20,911个多选题。通过与全球 diverse 研究者合作的开放科学合作构建，Kaleidoscope确保语言和文化的真实性。我们评估了顶尖的多语言视觉语言模型，发现它们在低资源语言和复杂多模态情境下的表现较差。我们的结果凸显了在文化包容性多模态评估框架方面的需求。
引用
@article{arxiv.2504.07072,
title = {Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation},
author = {Israfel Salazar and Manuel Fernández Burda and Shayekh Bin Islam and Arshia Soltani Moakhar and Shivalika Singh and Fabian Farestam and Angelika Romanou and Danylo Boiko and Dipika Khullar and Mike Zhang and Dominik Krzemiński and Jekaterina Novikova and Luísa Shimabucoro and Joseph Marvin Imperial and Rishabh Maheshwary and Sharad Duwal and Alfonso Amayuelas and Swati Rajwal and Jebish Purbey and Ahmed Ruby and Nicholas Popovič and Marek Suppa and Azmine Toushik Wasi and Ram Mohan Rao Kadiyala and Olga Tsymboi and Maksim Kostritsya and Bardia Soltani Moakhar and Gabriel da Costa Merlin and Otávio Ferracioli Coletti and Maral Jabbari Shiviari and MohammadAmin farahani fard and Silvia Fernandez and María Grandury and Dmitry Abulkhanov and Drishti Sharma and Andre Guarnier De Mitri and Leticia Bossatto Marchezi and Setayesh Heydari and Johan Obando-Ceron and Nazar Kohut and Beyza Ermis and Desmond Elliott and Enzo Ferrante and Sara Hooker and Marzieh Fadaee},
journal= {arXiv preprint arXiv:2504.07072},
year = {2025}
}
备注
v2: corrected the author list