ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding
Abstract
With the rapid development of evaluation datasets to assess LLMs understanding across a wide range of subjects and domains, identifying a suitable language understanding benchmark has become increasingly challenging. In this work, we explore LLM evaluation challenges for low-resource language understanding and introduce \proverbeval, LLM evaluation benchmark for low-resource languages, focusing on low-resource language understanding in culture-specific scenarios. We benchmark various LLMs and explore factors that create variability in the benchmarking process. We observed performance variances of up to 50\%, depending on the order in which answer choices were presented in multiple-choice tasks. Native language proverb descriptions significantly improve tasks such as proverb generation, contributing to improved outcomes. Additionally, monolingual evaluations consistently outperformed their cross-lingual counterparts in generation tasks. We argue that special attention must be given to the order of choices, the choice of prompt language, task variability, and generation tasks when creating LLM evaluation benchmarks. Evaluation data available at https://huggingface.co/datasets/israel/ProverbEval, evaluation code https://github.com/EthioNLP/EthioProverbEval.
Cite
@article{arxiv.2411.05049,
title = {ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding},
author = {Israel Abebe Azime and Atnafu Lambebo Tonja and Tadesse Destaw Belay and Yonas Chanie and Bontu Fufa Balcha and Negasi Haile Abadi and Henok Biadglign Ademtew and Mulubrhan Abebe Nerea and Debela Desalegn Yadeta and Derartu Dagne Geremew and Assefa Atsbiha tesfau and Philipp Slusallek and Thamar Solorio and Dietrich Klakow},
journal= {arXiv preprint arXiv:2411.05049},
year = {2025}
}