DataComp-LM：为语言模型寻找下一代训练集
摘要
我们引入语言模型的数据比较框架DataComp for Language Models (DCLM)，用于受控数据集实验以改进语言模型。作为DCLM的一部分，我们提供了来自Common Crawl提取的240T标记的标准化语料库，基于OpenLM框架的有效预训练配方，以及53项广泛的下游评估套件。DCLM基准的参与者可在412M至70亿参数的模型规模范围内实验数据清洗策略，如去重、过滤和数据混合。作为DCLM的基线，我们进行了大量实验，发现基于模型的过滤是组装高质量训练集的关键。 resulting dataset, DCLM-Baseline, enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.
引用
@article{arxiv.2406.11794,
title = {DataComp-LM: In search of the next generation of training sets for language models},
author = {Jeffrey Li and Alex Fang and Georgios Smyrnis and Maor Ivgi and Matt Jordan and Samir Gadre and Hritik Bansal and Etash Guha and Sedrick Keh and Kushal Arora and Saurabh Garg and Rui Xin and Niklas Muennighoff and Reinhard Heckel and Jean Mercat and Mayee Chen and Suchin Gururangan and Mitchell Wortsman and Alon Albalak and Yonatan Bitton and Marianna Nezhurina and Amro Abbas and Cheng-Yu Hsieh and Dhruba Ghosh and Josh Gardner and Maciej Kilian and Hanlin Zhang and Rulin Shao and Sarah Pratt and Sunny Sanyal and Gabriel Ilharco and Giannis Daras and Kalyani Marathe and Aaron Gokaslan and Jieyu Zhang and Khyathi Chandu and Thao Nguyen and Igor Vasiljevic and Sham Kakade and Shuran Song and Sujay Sanghavi and Fartash Faghri and Sewoong Oh and Luke Zettlemoyer and Kyle Lo and Alaaeldin El-Nouby and Hadi Pouransari and Alexander Toshev and Stephanie Wang and Dirk Groeneveld and Luca Soldaini and Pang Wei Koh and Jenia Jitsev and Thomas Kollar and Alexandros G. Dimakis and Yair Carmon and Achal Dave and Ludwig Schmidt and Vaishaal Shankar},
journal= {arXiv preprint arXiv:2406.11794},
year = {2025}
}
备注
Project page: https://www.datacomp.ai/dclm/