Improving Methodologies for LLM Evaluations Across Global Languages
Abstract
As frontier AI models are deployed globally, it is essential that their behaviour remains safe and reliable across diverse linguistic and cultural contexts. To examine how current model safeguards hold up in such settings, participants from the International Network for Advanced AI Measurement, Evaluation and Science, including representatives from Singapore, Japan, Australia, Canada, the EU, France, Kenya, South Korea and the UK conducted a joint multilingual evaluation exercise. Led by Singapore AISI, two open-weight models were tested across ten languages spanning high and low resourced groups: Cantonese English, Farsi, French, Japanese, Korean, Kiswahili, Malay, Mandarin Chinese and Telugu. Over 6,000 newly translated prompts were evaluated across five harm categories (privacy, non-violent crime, violent crime, intellectual property and jailbreak robustness), using both LLM-as-a-judge and human annotation. The exercise shows how safety behaviours can vary across languages. These include differences in safeguard robustness across languages and harm types and variation in evaluator reliability (LLM-as-judge vs. human review). Further, it also generated methodological insights for improving multilingual safety evaluations, such as the need for culturally contextualised translations, stress-tested evaluator prompts and clearer human annotation guidelines. This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems and calls for continued collaboration with the wider research community and industry.
Cite
@article{arxiv.2601.15706,
title = {Improving Methodologies for LLM Evaluations Across Global Languages},
author = {Akriti Vij and Benjamin Chua and Darshini Ramiah and En Qi Ng and Mahran Morsidi and Naga Nikshith Gangarapu and Sharmini Johnson and Vanessa Wilfred and Vikneswaran Kumaran and Wan Sie Lee and Wenzhuo Yang and Yongsen Zheng and Bill Black and Boming Xia and Frank Sun and Hao Zhang and Qinghua Lu and Suyu Ma and Yue Liu and Chi-kiu Lo and Fatemeh Azadi and Isar Nejadgholi and Sowmya Vajjala and Agnes Delaborde and Nicolas Rolin and Tom Seimandi and Akiko Murakami and Haruto Ishi and Satoshi Sekine and Takayuki Semitsu and Tasuku Sasaki and Angela Kinuthia and Jean Wangari and Michael Michie and Stephanie Kasaon and Hankyul Baek and Jaewon Noh and Kihyuk Nam and Sang Seo and Sungpil Shin and Taewhi Lee and Yongsu Kim and Daisy Newbold-Harrop and Jessica Wang and Mahmoud Ghanem and Vy Hong},
journal= {arXiv preprint arXiv:2601.15706},
year = {2026}
}
Comments
Author names have been organised by country, and in alphabetical order within countries