Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Sandeep Subramanian, Adam Trischler, Yoshua Bengio, Christopher J Pal
3/30/2018
cs.CL

Official code: facebookresearch/SentEval (2,111 stars, last commit 2024-03-19). 1 more implementation below.

Abstract

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

View on arXivView PDF

Code Implementations(2)

facebookresearch/SentEvalOfficial100%

A python tool for evaluating the quality of sentence embeddings.

2,111306sed, ShellMay 18, 20172 years agoNOASSERTION
Maluuba/gensenOfficial100%

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

31143Shell, PythonFeb 22, 20186 years agoNOASSERTION

Similar Papers

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

88% match2 repos2.3k2017

Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features

86% match1 repo1.2k2017

JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applications

85% match1 repo82023

Multi-Task Deep Neural Networks for Natural Language Understanding

84% match1 repo2.3k2019

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

83% match2 repos6.6k2021

An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understanding

83% match1 repo62025

Cite this paper

@article{subramanian2018learning,
  title  = {Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning},
  author = {Sandeep Subramanian and Adam Trischler and Yoshua Bengio and Christopher J Pal},
  year   = {2018},
  eprint = {1804.00079},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1804.00079}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1804.00079.svg)](https://codeofpaper.com/paper/1804.00079)

Discussion