INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Abstract
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.
Cite
@article{arxiv.2411.19799,
title = {INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge},
author = {Angelika Romanou and Negar Foroutan and Anna Sotnikova and Zeming Chen and Sree Harsha Nelaturu and Shivalika Singh and Rishabh Maheshwary and Micol Altomare and Mohamed A. Haggag and Snegha A and Alfonso Amayuelas and Azril Hafizi Amirudin and Viraat Aryabumi and Danylo Boiko and Michael Chang and Jenny Chim and Gal Cohen and Aditya Kumar Dalmia and Abraham Diress and Sharad Duwal and Daniil Dzenhaliou and Daniel Fernando Erazo Florez and Fabian Farestam and Joseph Marvin Imperial and Shayekh Bin Islam and Perttu Isotalo and Maral Jabbarishiviari and Börje F. Karlsson and Eldar Khalilov and Christopher Klamm and Fajri Koto and Dominik Krzemiński and Gabriel Adriano de Melo and Syrielle Montariol and Yiyang Nan and Joel Niklaus and Jekaterina Novikova and Johan Samir Obando Ceron and Debjit Paul and Esther Ploeger and Jebish Purbey and Swati Rajwal and Selvan Sunitha Ravi and Sara Rydell and Roshan Santhosh and Drishti Sharma and Marjana Prifti Skenduli and Arshia Soltani Moakhar and Bardia Soltani Moakhar and Ran Tamir and Ayush Kumar Tarun and Azmine Toushik Wasi and Thenuka Ovin Weerasinghe and Serhan Yilmaz and Mike Zhang and Imanol Schlag and Marzieh Fadaee and Sara Hooker and Antoine Bosselut},
journal= {arXiv preprint arXiv:2411.19799},
year = {2024}
}