Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

Abishek Stephen, Jindřich Libovický
1/26/2026
cs.CL

No official code found. Best implementation: abishekjs/morph-tok-eval (0 stars, last commit 2025-10-29).

Abstract

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentation data that is either unavailable or of inconsistent quality across many languages, our approach utilizes morpho-syntactic features. These are available in resources such as Universal Dependencies or UniMorph for a much wider range of languages. The metric works by probabilistically aligning subwords with morphological features through an IBM Model 1. Our experiments show that the metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems.

View on arXivView PDF

Code Implementations(1)

abishekjs/morph-tok-evalCommunity71%

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

01PythonMay 6, 202511 months ago

Similar Papers

Lexically Grounded Subword Segmentation

87% match2 repos22024

Evaluating Morphological Alignment of Tokenizers in 70 Languages

87% match1 repo192025

Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization

86% match2024

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

84% match2025

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages

82% match1 repo32025

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay

82% match2026

Cite this paper

@article{stephen2026evaluating,
  title  = {Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features},
  author = {Abishek Stephen and Jindřich Libovický},
  year   = {2026},
  eprint = {2601.18536},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.18536v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.18536.svg)](https://codeofpaper.com/paper/2601.18536)

Discussion