Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation

Mir Tafseer Nayeem, Sawsan Alqahtani, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, M Saiful Bari
10/11/2025
cs.CLcs.AIcs.LG

No official code found. Best implementation: tafseer-nayeem/STRR (1 star, last commit 2025-12-07).

Abstract

Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers.

View on arXivView PDF

Code Implementations(1)

tafseer-nayeem/STRRCommunity61%

[NeurIPS 2025 Workshop] - Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation

10Jupyter NotebookSep 26, 202510 months ago

Similar Papers

Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency

87% match2025

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design

86% match2025

IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs

86% match2025

The Token Tax: Systematic Bias in Multilingual Tokenization

86% match2025

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

85% match2024

Beyond Text Compression: Evaluating Tokenizers Across Scales

84% match2025

Cite this paper

@article{nayeem2025beyond,
  title  = {Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation},
  author = {Mir Tafseer Nayeem and Sawsan Alqahtani and Md Tahmid Rahman Laskar and Tasnim Mohiuddin and M Saiful Bari},
  year   = {2025},
  eprint = {2510.09947},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.09947v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.09947.svg)](https://codeofpaper.com/paper/2510.09947)

Discussion