Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction

Maha Elbayad, Laurent Besacier, Jakob Verbeek
8/11/2018
cs.CL

Official code: elbayadm/attn2d (496 stars, MIT, PyTorch, last commit 2021-05-08).

Abstract

Current state-of-the-art machine translation systems are based on encoder-decoder architectures, that first encode the input sequence, and then generate an output sequence based on the input encoding. Both are interfaced with an attention mechanism that recombines a fixed encoding of the source tokens based on the decoder state. We propose an alternative approach which instead relies on a single 2D convolutional neural network across both sequences. Each layer of our network re-codes source tokens on the basis of the output sequence produced so far. Attention-like properties are therefore pervasive throughout the network. Our model yields excellent results, outperforming state-of-the-art encoder-decoder systems, while being conceptually simpler and having fewer parameters.

View on arXivView PDF

Code Implementations(1)

elbayadm/attn2dOfficial100%
PyTorchMIT

Pervasive Attention: 2D Convolutional Networks for Sequence-to-Sequence Prediction

49669C++, LuaAug 10, 20185 years agoMIT
fairseqneural-machine-translationnlpnmtpytorch

Similar Papers

Attention Is All You Need

90% match1 repo17.5k2017

Joint Source-Target Self Attention with Locality Constraints

89% match1 repo202019

Fully Character-Level Neural Machine Translation without Explicit Segmentation

87% match1 repo1432016

Asynchronous Bidirectional Decoding for Neural Machine Translation

86% match1 repo312018

GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation

85% match1 repo32022

Neural Machine Translation

85% match5 repos17.5k2017

Cite this paper

@article{elbayad2018pervasive,
  title  = {Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction},
  author = {Maha Elbayad and Laurent Besacier and Jakob Verbeek},
  year   = {2018},
  eprint = {1808.03867},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1808.03867}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1808.03867.svg)](https://codeofpaper.com/paper/1808.03867)

Discussion