BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, Ivan P. Yamshchikov
9/6/2024
cs.CL

No official code found. Best implementation: pchizhov/picky_bpe (27 stars, last commit 2024-11-25).

Abstract

Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.

View on arXivView PDF

Code Implementations(1)

pchizhov/picky_bpeCommunity100%

BPE modification that implements removing of the intermediate tokens during tokenizer training.

274PythonMay 25, 20241 years ago

Similar Papers

Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models

89% match2025

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

87% match2026

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

87% match2024

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models

86% match1 repo92024

AdaptBPE: From General Purpose to Specialized Tokenizers

86% match2026

Tokenization Is More Than Compression

86% match2 repos142024

Cite this paper

@article{chizhov2024gets,
  title  = {BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training},
  author = {Pavel Chizhov and Catherine Arnett and Elizaveta Korotkova and Ivan P. Yamshchikov},
  year   = {2024},
  eprint = {2409.04599},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.04599v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.04599.svg)](https://codeofpaper.com/paper/2409.04599)

Discussion