Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights

N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, Maunendra Sankar Desarkar
6/21/2025
cs.CL

No confident code implementation found for this paper.

Abstract

Tokenization plays a pivotal role in multilingual NLP. However, existing tokenizers are often skewed towards high-resource languages, limiting their effectiveness for linguistically diverse and morphologically rich languages such as those in the Indian subcontinent. This paper presents a comprehensive intrinsic evaluation of tokenization strategies across 17 Indian languages. We quantify the trade-offs between bottom-up and top-down tokenizer algorithms (BPE and Unigram LM), effects of vocabulary sizes, and compare strategies of multilingual vocabulary construction such as joint and cluster-based training. We also show that extremely low-resource languages can benefit from tokenizers trained on related high-resource languages. Our study provides practical insights for building more fair, efficient, and linguistically informed tokenizers for multilingual NLP.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs

89% match2025

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

89% match2024

Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha

89% match2025

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design

88% match2025

Comparative analysis of subword tokenization approaches for Indian languages

87% match2025

Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing

87% match2024

Cite this paper

@article{karthika2025multilingual,
  title  = {Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights},
  author = {N J Karthika and Maharaj Brahma and Rohit Saluja and Ganesh Ramakrishnan and Maunendra Sankar Desarkar},
  year   = {2025},
  eprint = {2506.17789},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.17789v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.17789.svg)](https://codeofpaper.com/paper/2506.17789)

Discussion