Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
Abstract
Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers.
Cite
@article{arxiv.2510.09947,
title = {Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation},
author = {Mir Tafseer Nayeem and Sawsan Alqahtani and Md Tahmid Rahman Laskar and Tasnim Mohiuddin and M Saiful Bari},
journal= {arXiv preprint arXiv:2510.09947},
year = {2025}
}
Comments
NeurIPS 2025 Workshop