Mask-Predict: Parallel Decoding of Conditional Masked Language Models

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer
4/19/2019
cs.CLcs.AIcs.LG

Official code: facebookresearch/Mask-Predict (247 stars, last commit 2021-09-17).

Abstract

Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves state-of-the-art performance levels for non-autoregressive and parallel decoding translation models by over 4 BLEU on average. It is also able to reach within about 1 BLEU point of a typical left-to-right transformer model, while decoding significantly faster.

View on arXivView PDF

Code Implementations(1)

facebookresearch/Mask-PredictOfficial100%

A masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation.

24741C++, LuaAug 29, 20195 years agoNOASSERTION

Similar Papers

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

87% match1 repo42023

RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer

86% match2023

Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation

86% match1 repo182019

Non-Autoregressive Neural Machine Translation

86% match1 repo2712017

Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

85% match1 repo132019

INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion

85% match2023

Cite this paper

@article{ghazvininejad2019maskpredict,
  title  = {Mask-Predict: Parallel Decoding of Conditional Masked Language Models},
  author = {Marjan Ghazvininejad and Omer Levy and Yinhan Liu and Luke Zettlemoyer},
  year   = {2019},
  eprint = {1904.09324},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.09324}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.09324.svg)](https://codeofpaper.com/paper/1904.09324)

Discussion