Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Abstract
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.
Keywords
Cite
@article{arxiv.2103.12028,
title = {Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets},
author = {Julia Kreutzer and Isaac Caswell and Lisa Wang and Ahsan Wahab and Daan van Esch and Nasanbayar Ulzii-Orshikh and Allahsera Tapo and Nishant Subramani and Artem Sokolov and Claytone Sikasote and Monang Setyawan and Supheakmungkol Sarin and Sokhar Samb and Benoît Sagot and Clara Rivera and Annette Rios and Isabel Papadimitriou and Salomey Osei and Pedro Ortiz Suarez and Iroro Orife and Kelechi Ogueji and Andre Niyongabo Rubungo and Toan Q. Nguyen and Mathias Müller and André Müller and Shamsuddeen Hassan Muhammad and Nanda Muhammad and Ayanda Mnyakeni and Jamshidbek Mirzakhalov and Tapiwanashe Matangira and Colin Leong and Nze Lawson and Sneha Kudugunta and Yacine Jernite and Mathias Jenny and Orhan Firat and Bonaventure F. P. Dossou and Sakhile Dlamini and Nisansa de Silva and Sakine Çabuk Ballı and Stella Biderman and Alessia Battisti and Ahmed Baruwa and Ankur Bapna and Pallavi Baljekar and Israel Abebe Azime and Ayodele Awokoya and Duygu Ataman and Orevaoghene Ahia and Oghenefego Ahia and Sweta Agrawal and Mofetoluwa Adeyemi},
journal= {arXiv preprint arXiv:2103.12028},
year = {2022}
}
Comments
Accepted at TACL; pre-MIT Press publication version