VerChol -- Grammar-First Tokenization for Agglutinative Languages

Prabhu Raja
3/6/2026
cs.CL

No confident code implementation found for this paper.

Abstract

Tokenization is the foundational step in all large language model (LLM) pipelines, yet the dominant approach Byte Pair Encoding (BPE) and its variants is inherently script agnostic and optimized for English like morphology. For agglutinative languages a typological class encompassing the Dravidian family (Tamil, Kannada, Telugu, Malayalam), Turkic languages (Turkish, Azerbaijani, Uzbek), Uralic languages (Finnish, Hungarian, Estonian), Korean, Japanese, Swahili, Basque, and others, a single word may encode root, tense, aspect, person, number, gender agreement, case, and postpositions into one orthographic unit. Statistical tokenizers fragment these words into byte pair chunks that sever morpheme boundaries and inflate token counts.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{raja2026verchol,
  title  = {VerChol -- Grammar-First Tokenization for Agglutinative Languages},
  author = {Prabhu Raja},
  year   = {2026},
  eprint = {2603.05883},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.05883v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.05883.svg)](https://codeofpaper.com/paper/2603.05883)

Discussion