How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices.
@article{arxiv.2403.00686,
title = {A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages},
author = {Catherine Arnett and Tyler A. Chang and Benjamin K. Bergen},
journal= {arXiv preprint arXiv:2403.00686},
year = {2024}
}