CommonLID：重新评估网络数据上的状态-of-the-art 语言识别性能
计算与语言
2026-01-27 v1
摘要
语言识别（LID）是构建多语言语料库的基本步骤。然而，LID 模型在许多语言上仍表现不佳，尤其是在用于训练多语言语言模型的噪声且异构的网络数据上。在本文中，我们介绍了 CommonLID，这是一个面向网络领域的、由社区驱动的、人工标注的 LID 基准数据集，覆盖 109 种语言。其中许多语言之前都未得到充分服务，使得 CommonLID 是开发更具代表性的高质量文本语料库的关键资源。我们通过使用 CommonLID 以及五个其他常用评估数据集，对八个流行的 LID 模型进行测试，展示了 CommonLID 的价值。我们分析了结果，以定位我们的贡献，并提供该领域研究现状的概览。特别是，我们指出，现有评估在网络领域的许多语言上高估了 LID 的准确率。我们在采用开放、宽松许可证的前提下，公开 CommonLID 以及用于创建其的代码。
引用
@article{arxiv.2601.18026,
title = {CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data},
author = {Pedro Ortiz Suarez and Laurie Burchell and Catherine Arnett and Rafael Mosquera-Gómez and Sara Hincapie-Monsalve and Thom Vaughan and Damian Stewart and Malte Ostendorff and Idris Abdulmumin and Vukosi Marivate and Shamsuddeen Hassan Muhammad and Atnafu Lambebo Tonja and Hend Al-Khalifa and Nadia Ghezaiel Hammouda and Verrah Otiende and Tack Hwa Wong and Jakhongir Saydaliev and Melika Nobakhtian and Muhammad Ravi Shulthan Habibi and Chalamalasetti Kranti and Carol Muchemi and Khang Nguyen and Faisal Muhammad Adam and Luis Frentzen Salim and Reem Alqifari and Cynthia Amol and Joseph Marvin Imperial and Ilker Kesen and Ahmad Mustafid and Pavel Stepachev and Leshem Choshen and David Anugraha and Hamada Nayel and Seid Muhie Yimam and Vallerie Alexandra Putra and My Chiffon Nguyen and Azmine Toushik Wasi and Gouthami Vadithya and Rob van der Goot and Lanwenn ar C'horr and Karan Dua and Andrew Yates and Mithil Bangera and Yeshil Bangera and Hitesh Laxmichand Patel and Shu Okabe and Fenal Ashokbhai Ilasariya and Dmitry Gaynullin and Genta Indra Winata and Yiyuan Li and Juan Pablo Martínez and Amit Agarwal and Ikhlasul Akmal Hanif and Raia Abu Ahmad and Esther Adenuga and Filbert Aurelian Tjiaranata and Weerayut Buaphet and Michael Anugraha and Sowmya Vajjala and Benjamin Rice and Azril Hafizi Amirudin and Jesujoba O. Alabi and Srikant Panda and Yassine Toughrai and Bruhan Kyomuhendo and Daniel Ruffinelli and Akshata A and Manuel Goulão and Ej Zhou and Ingrid Gabriela Franco Ramirez and Cristina Aggazzotti and Konstantin Dobler and Jun Kevin and Quentin Pagès and Nicholas Andrews and Nuhu Ibrahim and Mattes Ruckdeschel and Amr Keleg and Mike Zhang and Casper Muziri and Saron Samuel and Sotaro Takeshita and Kun Kerdthaisong and Luca Foppiano and Rasul Dent and Tommaso Green and Ahmad Mustapha Wali and Kamohelo Makaaka and Vicky Feliren and Inshirah Idris and Hande Celikkanat and Abdulhamid Abubakar and Jean Maillard and Benoît Sagot and Thibault Clérice and Kenton Murray and Sarah Luger},
journal= {arXiv preprint arXiv:2601.18026},
year = {2026}
}
备注
17 pages, 7 tables, 5 figures