M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation

Jinming Zhao, Hao Yang, Ehsan Shareghi, Gholamreza Haffari
7/3/2022
cs.CLcs.LGcs.SD

Official code: mingzi151/w2v2-st (8 stars, last commit 2022-09-17).

Abstract

End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pre-training and fine-tuning, largely due to the modality differences between speech outputs from the encoder and text inputs to the decoder. In this work, we aim to bridge the modality gap between speech and text to improve translation quality. We propose M-Adapter, a novel Transformer-based module, to adapt speech representations to text. While shrinking the speech sequence, M-Adapter produces features desired for speech-to-text translation via modelling global and local dependencies of a speech sequence. Our experimental results show that our model outperforms a strong baseline by up to 1 BLEU score on the Must-C En$\rightarrow$DE dataset.\footnote{Our code is available at https://github.com/mingzi151/w2v2-st.}

View on arXivView PDF

Code Implementations(1)

mingzi151/w2v2-stOfficial100%
82C++, LuaSep 17, 20224 years ago

Similar Papers

Soft Alignment of Modality Space for End-to-end Speech Translation

88% match2023

Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

87% match1 repo312023

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

86% match2023

DUB: Discrete Unit Back-translation for Speech Translation

86% match1 repo282023

Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing

86% match2023

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

86% match2023

Cite this paper

@article{zhao2022madapter,
  title  = {M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation},
  author = {Jinming Zhao and Hao Yang and Ehsan Shareghi and Gholamreza Haffari},
  year   = {2022},
  eprint = {2207.00952},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.00952v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.00952.svg)](https://codeofpaper.com/paper/2207.00952)

Discussion