AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages
Abstract
Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).
Keywords
Cite
@article{arxiv.2311.09828,
title = {AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages},
author = {Jiayi Wang and David Ifeoluwa Adelani and Sweta Agrawal and Marek Masiak and Ricardo Rei and Eleftheria Briakou and Marine Carpuat and Xuanli He and Sofia Bourhim and Andiswa Bukula and Muhidin Mohamed and Temitayo Olatoye and Tosin Adewumi and Hamam Mokayed and Christine Mwase and Wangui Kimotho and Foutse Yuehgoh and Anuoluwapo Aremu and Jessica Ojo and Shamsuddeen Hassan Muhammad and Salomey Osei and Abdul-Hakeem Omotayo and Chiamaka Chukwuneke and Perez Ogayo and Oumaima Hourrane and Salma El Anigri and Lolwethu Ndolela and Thabiso Mangwana and Shafie Abdi Mohamed and Ayinde Hassan and Oluwabusayo Olufunke Awoyomi and Lama Alkhaled and Sana Al-Azzawi and Naome A. Etori and Millicent Ochieng and Clemencia Siro and Samuel Njoroge and Eric Muchiri and Wangari Kimotho and Lyse Naomi Wamba Momo and Daud Abolade and Simbiat Ajao and Iyanuoluwa Shode and Ricky Macharm and Ruqayya Nasir Iro and Saheed S. Abdullahi and Stephen E. Moore and Bernard Opoku and Zainab Akinjobi and Abeeb Afolabi and Nnaemeka Obiefuna and Onyekachi Raphael Ogbu and Sam Brian and Verrah Akinyi Otiende and Chinedu Emmanuel Mbonu and Sakayo Toadoum Sari and Yao Lu and Pontus Stenetorp},
journal= {arXiv preprint arXiv:2311.09828},
year = {2024}
}
Comments
Accepted by NAACL 2024