Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets

Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov
1/26/2026
cs.CLcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing 'glottosets' from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Measuring cross-language intelligibility between Romance languages with computational tools

81% match2026

Advancing the Database of Cross-Linguistic Colexifications with New Workflows and Data

81% match2025

Sampling the Swadesh List to Identify Similar Languages with Tree Spaces

80% match1 repo12024

From Isolates to Families: Using Neural Networks for Automated Language Affiliation

80% match2025

Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation

80% match2026

Patterns of Persistence and Diffusibility across the World's Languages

80% match2024

Cite this paper

@article{chelombitko2026subwordbased,
  title  = {Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets},
  author = {Iaroslav Chelombitko and Mika Hämäläinen and Aleksey Komissarov},
  year   = {2026},
  eprint = {2601.18791},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.18791v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.18791.svg)](https://codeofpaper.com/paper/2601.18791)

Discussion