DUB: Discrete Unit Back-translation for Speech Translation

Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang, Yaqian Zhou
5/19/2023
cs.CLcs.SDeess.AS

Official code: 0nutation/DUB (28 stars, MIT, last commit 2023-06-28).

Abstract

How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST. Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation (DUB) to answer two questions: (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://github.com/0nutation/DUB.

View on arXivView PDF

Code Implementations(1)

0nutation/DUBOfficial100%
MIT

Code and pretrained models for "DUB: Discrete Unit Back-translation for Speech Translation" (ACL 2023 Findings)

283C++, CudaMay 7, 20233 years agoMIT

Similar Papers

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

86% match3 repos11.9k2023

M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation

86% match1 repo82022

Back Translation for Speech-to-text Translation Without Transcripts

86% match1 repo112023

English to Central Kurdish Speech Translation: Corpus Creation, Evaluation, and Orthographic Standardization

86% match2026

PolyVoice: Language Models for Speech to Speech Translation

85% match2026

End-to-End Speech-to-Text Translation: A Survey

84% match2023

Cite this paper

@article{zhang2023dub,
  title  = {DUB: Discrete Unit Back-translation for Speech Translation},
  author = {Dong Zhang and Rong Ye and Tom Ko and Mingxuan Wang and Yaqian Zhou},
  year   = {2023},
  eprint = {2305.11411},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.11411v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.11411.svg)](https://codeofpaper.com/paper/2305.11411)

Discussion