SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

Paul-Ambroise Duquenne, Holger Schwenk, Benoît Sagot
8/22/2023
cs.CL

No official code found. Best implementation: facebookresearch/SONAR (913 stars, last commit 2025-10-10). 2 more below.

Abstract

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks. We also provide a text decoder for 200 languages, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. Our text-to-text results are competitive compared to the state-of-the-art NLLB~1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.

View on arXivView PDF

Code Implementations(3)

facebookresearch/SONARCommunity100%

SONAR, a new multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders.

913103PythonAug 1, 20231 years agoNOASSERTION
raxtemur/SonarTransformersCommunity67%

Port of SONAR (Sentence-level multimodal and language-agnostic representations) from fairseq2 into HuggingFace Transformers.

00Python, Jupyter NotebookFeb 5, 20268 months ago
tutur90/Open-SONARCommunity64%

Reproduction of the training pipeline "SONAR: Sentence-Level Multimodal and Language-Agnostic Representations"

10Shell, PythonSep 13, 20251 years ago

Similar Papers

Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

92% match2026

UR-BERT: Scaling Text Encoders for Massively Multilingual TTS Through Universal Romanization and Speech Token Prediction

85% match2026

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

85% match1 repo12026

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

84% match2026

Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

84% match2026

Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer

84% match2023

Cite this paper

@article{duquenne2023sonar,
  title  = {SONAR: Sentence-Level Multimodal and Language-Agnostic Representations},
  author = {Paul-Ambroise Duquenne and Holger Schwenk and Benoît Sagot},
  year   = {2023},
  eprint = {2308.11466},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.11466v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.11466.svg)](https://codeofpaper.com/paper/2308.11466)

Discussion