What changes when you randomly choose BPE merge operations? Not much

Jonne Sälevä, Constantine Lignos
5/4/2023
cs.CLcs.AI

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We introduce three simple randomized variants of byte pair encoding (BPE) and explore whether randomizing the selection of merge operations substantially affects a downstream machine translation task. We focus on translation into morphologically rich languages, hypothesizing that this task may show sensitivity to the method of choosing subwords. Analysis using a Bayesian linear model indicates that two of the variants perform nearly indistinguishably compared to standard BPE while the other degrades performance less than we anticipated. We conclude that although standard BPE is widely used, there exists an interesting universe of potential variations on it worth investigating. Our code is available at: https://github.com/bltlab/random-bpe.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance

83% match2025

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models

80% match2025

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

80% match1 repo202025

Theoretical Analysis of Byte-Pair Encoding

78% match2024

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

78% match1 repo32023

Entropy-Driven Pre-Tokenization for Byte-Pair Encoding

77% match2025

Cite this paper

@article{slev2023what,
  title  = {What changes when you randomly choose BPE merge operations? Not much},
  author = {Jonne Sälevä and Constantine Lignos},
  year   = {2023},
  eprint = {2305.03029},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.03029v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.03029.svg)](https://codeofpaper.com/paper/2305.03029)

Discussion