BigScience ROOTS语料库：一个1.6TB的复合多语言数据集
计算与语言
2023-03-08 v1 人工智能
摘要
随着语言模型变得越来越大，对大规模高质量文本数据集的需求前所未有的迫切，尤其是在多语言环境下。BigScience研讨会是一个为期1年的国际多学科倡议，其目标是将研究和训练大语言模型作为一项以价值观为驱动的事业，将伦理、危害和治理问题置于前沿。本文记录了BigScience为构建责任开放科学开放协作文本源（ROOTS）语料库所进行的数据创建和整理工作，该语料库是一个跨越59种语言、规模1.6TB的数据集，用于训练1760亿参数的BigScience大型开放科学开放访问多语言（BLOOM）语言模型。我们进一步发布了该语料库的一个大型初始子集及其分析，希望以数据和处理工具赋能大规模单语言和多语言建模项目，并刺激围绕这一大型多语言语料库的研究。
引用
@article{arxiv.2303.03915,
title = {The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset},
author = {Hugo Laurençon and Lucile Saulnier and Thomas Wang and Christopher Akiki and Albert Villanova del Moral and Teven Le Scao and Leandro Von Werra and Chenghao Mou and Eduardo González Ponferrada and Huu Nguyen and Jörg Frohberg and Mario Šaško and Quentin Lhoest and Angelina McMillan-Major and Gerard Dupont and Stella Biderman and Anna Rogers and Loubna Ben allal and Francesco De Toni and Giada Pistilli and Olivier Nguyen and Somaieh Nikpoor and Maraim Masoud and Pierre Colombo and Javier de la Rosa and Paulo Villegas and Tristan Thrush and Shayne Longpre and Sebastian Nagel and Leon Weber and Manuel Muñoz and Jian Zhu and Daniel Van Strien and Zaid Alyafeai and Khalid Almubarak and Minh Chien Vu and Itziar Gonzalez-Dios and Aitor Soroa and Kyle Lo and Manan Dey and Pedro Ortiz Suarez and Aaron Gokaslan and Shamik Bose and David Adelani and Long Phan and Hieu Tran and Ian Yu and Suhas Pai and Jenny Chim and Violette Lepercq and Suzana Ilic and Margaret Mitchell and Sasha Alexandra Luccioni and Yacine Jernite},
journal= {arXiv preprint arXiv:2303.03915},
year = {2023}
}
备注
NeurIPS 2022, Datasets and Benchmarks Track