English

Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network

Computation and Language 2026-07-20 v1
Authors: Pilar Sánchez-Gijón , Susana Valdez , Sofía Calvo Del Barrio , Florence Bellemont , Anna Kokkinidou , Mihai Cristian Brasoveanu
View on arXiv ↗ PDF ↗ DOI ↗

Abstract

This paper reports on a collaboration between the Directorate-General for Translation (DGT) and the European Master's in Translation (EMT) to localise the MMLU dataset into 11 European languages. Beyond creating a more inclusive benchmark for LLM evaluation, the project offers master's students authentic, project-based professional training in translation, revision, project management, and multilingual coordination, while highlighting key methodological, administrative, and workflow challenges.

Keywords

MMLU localization multilingual benchmark LLM evaluation translation training European languages dataset creation

Cite

@article{arxiv.2607.18432,
  title  = {Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network},
  author = {Pilar Sánchez-Gijón and Susana Valdez and Sofía Calvo Del Barrio and Florence Bellemont and Anna Kokkinidou and Mihai Cristian Brasoveanu},
  journal= {arXiv preprint arXiv:2607.18432},
  year   = {2026}
}