What you can cram into a single vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni
5/3/2018
cs.CL

Official code: facebookresearch/SentEval (2,111 stars, last commit 2024-03-19).

Abstract

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.

View on arXivView PDF

Code Implementations(1)

facebookresearch/SentEvalOfficial100%

A python tool for evaluating the quality of sentence embeddings.

2,111306sed, ShellMay 18, 20172 years agoNOASSERTION

Similar Papers

On the Dimensionality of Sentence Embeddings

86% match2023

Finding Meaning in Embeddings: Concept Separation Curves

85% match2026

Are there identifiable structural parts in the sentence embedding whole?

85% match2024

Translation Aligned Sentence Embeddings for Turkish Language

85% match2023

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

84% match2 repos2.3k2017

Mechanistic Decomposition of Sentence Representations

84% match2025

Cite this paper

@article{conneau2018what,
  title  = {What you can cram into a single vector: Probing sentence embeddings for linguistic properties},
  author = {Alexis Conneau and German Kruszewski and Guillaume Lample and Loïc Barrault and Marco Baroni},
  year   = {2018},
  eprint = {1805.01070},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1805.01070}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1805.01070.svg)](https://codeofpaper.com/paper/1805.01070)

Discussion