MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei-Ning Hsu, Juan Pino +1 more
3/1/2023
cs.CLeess.AS

Official code: facebookresearch/muavic (406 stars, last commit 2023-09-11).

Abstract

We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https://github.com/facebookresearch/muavic.

View on arXivView PDF

Code Implementations(1)

facebookresearch/muavicOfficial100%

MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

40638Shell, PythonFeb 23, 20233 years agoNOASSERTION

Similar Papers

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

89% match2024

HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation

88% match2023

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

88% match3 repos11.9k2023

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

87% match1 repo482026

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

87% match2023

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

86% match2026

Cite this paper

@article{anwar2023muavic,
  title  = {MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation},
  author = {Mohamed Anwar and Bowen Shi and Vedanuj Goswami and Wei-Ning Hsu and Juan Pino and Changhan Wang},
  year   = {2023},
  eprint = {2303.00628},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.00628v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.00628.svg)](https://codeofpaper.com/paper/2303.00628)

Discussion