English

Datasheet for the Pile

Computation and Language 2022-01-20 v1
Authors: Stella Biderman , Kieran Bicheno , Leo Gao
View on arXiv ↗ PDF ↗

Abstract

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.

Cite

@article{arxiv.2201.07311,
  title  = {Datasheet for the Pile},
  author = {Stella Biderman and Kieran Bicheno and Leo Gao},
  journal= {arXiv preprint arXiv:2201.07311},
  year   = {2022}
}

Comments

Accompanies "The Pile: An 800GB Dataset of Diverse Text for Language Modeling" arXiv:2101.00027

Related papers

View all related →
Computation and Language · Computer Science
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding +8
2021-01-05
Computation and Language · Computer Science
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic +23
2025-06-06
Computation and Language · Computer Science
The MiniPile Challenge for Data-Efficient Language Models
Jean Kaddour
2023-04-18
Computation and Language · Computer Science
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha +3
2022-11-30
Computation and Language · Computer Science
TabLib: A Dataset of 627M Tables with Context
Gus Eggert, Kevin Huo, Mike Biven, Justin Waugh
2023-10-13
Sound · Computer Science
Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li +10
2025-10-09
Software Engineering · Computer Science
A Versatile Dataset of Agile Open Source Software Projects
Vali Tawosi, Afnan Al-Subaihin, Rebecca Moussa, Federica Sarro
2022-02-03
Computation and Language · Computer Science
IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus
Honghao Gui, Lin Yuan, Hongbin Ye, Ningyu Zhang +3
2024-05-28
Software Engineering · Computer Science
A Dataset of Dockerfiles
Jordan Henkel, Christian Bird, Shuvendu K. Lahiri, Thomas Reps
2020-03-31
Computation and Language · Computer Science
MathPile: A Billion-Token-Scale Pretraining Corpus for Math
Zengzhi Wang, Xuefeng Li, Rui Xia, Pengfei Liu
2024-10-30
Computation and Language · Computer Science
MultiLegalPile: A 689GB Multilingual Legal Corpus
Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis +1
2024-05-21
Computation and Language · Computer Science
The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling
Joey Öhman, Severine Verlinden, Ariel Ekgren, Amaru Cuba Gyllensten +4
2023-03-31
Machine Learning · Computer Science
The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón +6
2021-11-19
Artificial Intelligence · Computer Science
Speech Wikimedia: A 77 Language Multilingual Speech Dataset
Rafael Mosquera Gómez, Julián Eusse, Juan Ciro, Daniel Galvez +3
2023-08-31
Social and Information Networks · Computer Science
Making the complete OpenAIRE citation graph easily accessible through compact data representation
Joakim Skarding, Pavel Sanda
2026-05-19
Machine Learning · Computer Science
ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models
Adrian Mirza, Nawaf Alampara, Martiño Ríos-García, Mohamed Abdelalim +11
2025-05-20
Software Engineering · Computer Science
NICHE: A Curated Dataset of Engineered Machine Learning Projects in Python
Ratnadira Widyasari, Zhou Yang, Ferdian Thung, Sheng Qin Sim +7
2023-03-14
Software Engineering · Computer Science
The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large
Sumon Biswas, Mohammad Wardat, Hridesh Rajan
2022-02-15
Audio and Speech Processing · Electrical Eng. & Systems
Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li +10
2024-09-10
Computation and Language · Computer Science
NarrativeXL: A Large-scale Dataset For Long-Term Memory Models
Arseny Moskvichev, Ky-Vinh Mai
2023-12-11
Computation and Language · Computer Science
The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen +3
2023-10-31
Computation and Language · Computer Science
Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models
Hyunbyung Park, Sukyung Lee, Gyoungjin Gim, Yungi Kim +2
2025-03-05
Computation and Language · Computer Science
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen, Maliheh Izadi
2025-12-30
Machine Learning · Computer Science
Open Data on GitHub: Unlocking the Potential of AI
Anthony Cintron Roman, Kevin Xu, Arfon Smith, Jehu Torres Vega +2
2023-06-13
Cryptography and Security · Computer Science
ProPILE: Probing Privacy Leakage in Large Language Models
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri +2
2023-07-06
R2 v1 2026-06-24T08:54:33.454Z