Soft Alignment of Modality Space for End-to-end Speech Translation

Yuhao Zhang, Kaiqi Kou, Bei Li, Chen Xu, Chunliang Zhang +2 more
12/18/2023
cs.CLcs.AIcs.SD

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

End-to-end Speech Translation (ST) aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer. Existing methods typically employ hard alignment (H-Align) of individual speech and text segments, which can degrade textual representations. To address this, we introduce Soft Alignment (S-Align), using adversarial training to align the representation spaces of both modalities. S-Align creates a modality-invariant space while preserving individual modality quality. Experiments on three languages from the MuST-C dataset show S-Align outperforms H-Align across multiple tasks and offers translation capabilities on par with specialized translation models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation

88% match1 repo82022

Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer

87% match2023

Tuning Large language model for End-to-end Speech Translation

87% match2023

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

87% match2023

BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing

87% match1 repo592023

Understanding and Bridging the Modality Gap for Speech Translation

86% match1 repo162023

Cite this paper

@article{zhang2023soft,
  title  = {Soft Alignment of Modality Space for End-to-end Speech Translation},
  author = {Yuhao Zhang and Kaiqi Kou and Bei Li and Chen Xu and Chunliang Zhang and Tong Xiao and Jingbo Zhu},
  year   = {2023},
  eprint = {2312.10952},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.10952v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.10952.svg)](https://codeofpaper.com/paper/2312.10952)

Discussion