Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Taku Kudo
4/29/2018
cs.CL

Official code: google/sentencepiece (12,144 stars, Apache-2.0, last commit 2026-10-20).

Abstract

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

View on arXivView PDF

Code Implementations(1)

google/sentencepieceOfficial100%
Apache-2.0

Unsupervised text tokenizer for Neural Network-based text generation.

12,1441,388C, C++Mar 7, 20174 days agoApache-2.0
natural-language-processingneural-machine-translationword-segmentation

Similar Papers

BPE-Dropout: Simple and Effective Subword Regularization

89% match1 repo542019

Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation

88% match2023

SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation

85% match2023

Neural Machine Translation of Rare Words with Subword Units

83% match1 repo2.3k2015

Synthetic and Natural Noise Both Break Neural Machine Translation

82% match1 repo292017

Lexically Grounded Subword Segmentation

81% match2 repos22024

Cite this paper

@article{kudo2018subword,
  title  = {Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates},
  author = {Taku Kudo},
  year   = {2018},
  eprint = {1804.10959},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1804.10959}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1804.10959.svg)](https://codeofpaper.com/paper/1804.10959)

Discussion