Multilingual Speech-to-Speech Translation into Multiple Target Languages

Hongyu Gong, Ning Dong, Sravya Popuri, Vedanuj Goswami, Ann Lee +1 more
7/17/2023
cs.CLcs.SDeess.AS

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into $16$ target languages.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

87% match1 repo312023

Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer

87% match2023

Towards a Deep Understanding of Multilingual End-to-End Speech Translation

86% match1 repo52023

End-to-End Speech-to-Text Translation: A Survey

85% match2023

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

85% match3 repos11.9k2023

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

84% match2023

Cite this paper

@article{gong2023multilingual,
  title  = {Multilingual Speech-to-Speech Translation into Multiple Target Languages},
  author = {Hongyu Gong and Ning Dong and Sravya Popuri and Vedanuj Goswami and Ann Lee and Juan Pino},
  year   = {2023},
  eprint = {2307.08655},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.08655v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.08655.svg)](https://codeofpaper.com/paper/2307.08655)

Discussion