Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation

Yuchen Han, Chen Xu, Tong Xiao, Jingbo Zhu
6/13/2023
cs.CLcs.SDeess.AS

Official code: hannlp/TAB (3 stars, MIT, last commit 2023-07-14).

Abstract

Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in end-to-end speech translation (E2E ST). The commonplace "modality gap" between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the "capacity gap": high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the over-fitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for en-fr on the MuST-C dataset. Code and models are available at https://github.com/hannlp/TAB.

View on arXivView PDF

Code Implementations(1)

hannlp/TABOfficial100%
MIT

Code and pretrained models for ACL 2023 short paper: "Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation"

30C++, LuaMay 23, 20233 years agoMIT

Similar Papers

Understanding and Bridging the Modality Gap for Speech Translation

88% match1 repo162023

Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition

87% match2023

ReMova: Fine-tuning LLMs for English to Belarusian translation

86% match2026

Fine-tuning Large Language Models for Adaptive Machine Translation

86% match1 repo902023

Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech

85% match2026

Tuning Large language model for End-to-end Speech Translation

85% match2023

Cite this paper

@article{han2023modality,
  title  = {Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation},
  author = {Yuchen Han and Chen Xu and Tong Xiao and Jingbo Zhu},
  year   = {2023},
  eprint = {2306.07650},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.07650v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.07650.svg)](https://codeofpaper.com/paper/2306.07650)

Discussion