Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Abstract
Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with <UNK> placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE leads to more equitable token counts across languages, with negligible impact on global compression rate and no substantial effect on language-model performance in downstream tasks.
Keywords
Cite
@article{arxiv.2508.04796,
title = {Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization},
author = {Negar Foroutan and Clara Meister and Debjit Paul and Joel Niklaus and Sina Ahmadi and Antoine Bosselut and Rico Sennrich},
journal= {arXiv preprint arXiv:2508.04796},
year = {2025}
}