Sequence-to-Sequence Learning as Beam-Search Optimization

Sam Wiseman, Alexander M. Rush
6/9/2016
cs.CLcs.LGcs.NE

Official code: harvardnlp/BSO (54 stars, last commit 2017-01-23).

Abstract

Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work, we introduce a model and beam-search training scheme, based on the work of Daume III and Marcu (2005), that extends seq2seq to learn global sequence scores. This structured approach avoids classical biases associated with local training and unifies the training loss with the test-time usage, while preserving the proven model architecture of seq2seq and its efficient training approach. We show that our system outperforms a highly-optimized attention-based seq2seq system and other baselines on three different sequence to sequence tasks: word ordering, parsing, and machine translation.

View on arXivView PDF

Code Implementations(1)

harvardnlp/BSOOfficial100%
5420Lua, ShellOct 18, 20169 years ago

Similar Papers

Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models

87% match1 repo292023

Sequence Level Training with Recurrent Neural Networks

83% match1 repo3322015

AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

83% match2022

Glancing Future for Simultaneous Machine Translation

83% match1 repo32023

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

82% match2 repos642020

Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

82% match1 repo1.6k2018

Cite this paper

@article{wiseman2016sequencetosequence,
  title  = {Sequence-to-Sequence Learning as Beam-Search Optimization},
  author = {Sam Wiseman and Alexander M. Rush},
  year   = {2016},
  eprint = {1606.02960},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1606.02960}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1606.02960.svg)](https://codeofpaper.com/paper/1606.02960)

Discussion