No confident code implementation found for this paper. 1 possible match can be shown below.
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most $δ$ symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).
No confident code match yet
We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.
No code implementations found yet.
Know of an implementation? Let us know in the comments below!
Tokenisation over Bounded Alphabets is Hard
A Formal Perspective on Byte-Pair Encoding
Two Counterexamples to Tokenization and the Noiseless Channel
Frequency-Ordered Tokenization for Better Text Compression
Hierarchical Token Merging with Recursive Autoencoders for Unsupervised Grammar Induction
Tokenization and the Noiseless Channel
@article{whittington2024tokenisation,
title = {Tokenisation is NP-Complete},
author = {Philip Whittington and Gregor Bachmann and Tiago Pimentel},
year = {2024},
eprint = {2412.15210},
archivePrefix = {arXiv},
url = {http://arxiv.org/abs/2412.15210v1}
}Shows whether official code is linked for this paper.
[](https://codeofpaper.com/paper/2412.15210)