English

FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text

Computation and Language 2025-12-29 v2

Abstract

We present FineFreq, a large-scale multilingual character frequency dataset derived from the FineWeb and FineWeb2 corpora, covering over 1900 languages and spanning 2013-2025. The dataset contains frequency counts for 96 trillion characters processed from 57 TB of compressed text. For each language, FineFreq provides per-character statistics with aggregate and year-level frequencies, allowing fine-grained temporal analysis. The dataset preserves naturally occurring multilingual features such as cross-script borrowings, emoji, and acronyms without applying artificial filtering. Each character entry includes Unicode metadata (category, script, block), enabling domain-specific or other downstream filtering and analysis. The full dataset is released in both CSV and Parquet formats, with associated metadata, available on GitHub and HuggingFace. https://github.com/Bin-2/FineFreq

Keywords

Cite

@article{arxiv.2512.09701,
  title  = {FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text},
  author = {Binbin Xu},
  journal= {arXiv preprint arXiv:2512.09701},
  year   = {2025}
}
R2 v1 2026-07-01T08:18:55.894Z