Convolutional Sequence to Sequence Learning

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin
5/8/2017
cs.CL

Official code: facebookresearch/fairseq (32,213 stars, MIT, PyTorch, last commit 2025-09-30).

Abstract

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

View on arXivView PDF

Code Implementations(1)

facebookresearch/fairseqOfficial100%
PyTorchMIT

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

32,2136,667C++, LuaAug 29, 20171 years agoMIT
artificial-intelligencepythonpytorch

Similar Papers

Universal Transformers

89% match1 repo17.5k2018

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

89% match1 repo4.6k2018

Attention Is All You Need

87% match1 repo17.5k2017

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

86% match1 repo452017

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

86% match1 repo4182015

RWKV: Reinventing RNNs for the Transformer Era

85% match2 repos14.7k2023

Cite this paper

@article{gehring2017convolutional,
  title  = {Convolutional Sequence to Sequence Learning},
  author = {Jonas Gehring and Michael Auli and David Grangier and Denis Yarats and Yann N. Dauphin},
  year   = {2017},
  eprint = {1705.03122},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1705.03122}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1705.03122.svg)](https://codeofpaper.com/paper/1705.03122)

Discussion