Training and Evaluation of a Multilingual Tokenizer for GPT-SW3

Felix Stollenwerk
4/28/2023
cs.CLcs.AI

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

This paper provides a detailed discussion of the multilingual tokenizer used for GPT-SW3. It was trained on the Nordic Pile using the SentencePiece library and the BPE algorithm. We outline the tokenizer's most important features and share details on its learned vocabulary. In addition, we systematically analyze the properties and evaluate the performance of the tokenizer with regard to the different languages present in the data.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

LangMAP: A Language-Adaptive Approach to Tokenization

87% match2026

MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

85% match2026

On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi

84% match2023

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

84% match2026

MorphPiece : A Linguistic Tokenizer for Large Language Models

83% match2023

Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights

83% match2025

Cite this paper

@article{stollenwerk2023training,
  title  = {Training and Evaluation of a Multilingual Tokenizer for GPT-SW3},
  author = {Felix Stollenwerk},
  year   = {2023},
  eprint = {2304.14780},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.14780v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.14780.svg)](https://codeofpaper.com/paper/2304.14780)

Discussion