Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation

Peidong Wang, Jian Xue, Jinyu Li, Junkun Chen, Aswin Shanmugam Subramanian
6/12/2024
cs.CLcs.SDeess.AS

No confident code implementation found for this paper.

Abstract

Language-agnostic many-to-one end-to-end speech translation models can convert audio signals from different source languages into text in a target language. These models do not need source language identification, which improves user experience. In some cases, the input language can be given or estimated. Our goal is to use this additional language information while preserving the quality of the other languages. We accomplish this by introducing a simple and effective linear input network. The linear input network is initialized as an identity matrix, which ensures that the model can perform as well as, or better than, the original model. Experimental results show that the proposed method can successfully enhance the specified language, while keeping the language-agnostic ability of the many-to-one ST models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data

83% match2024

Zero-resource Speech Translation and Recognition with LLMs

82% match2024

PolyVoice: Language Models for Speech to Speech Translation

81% match2026

Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptation

81% match2025

SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought

81% match2024

End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMs

80% match2025

Cite this paper

@article{wang2024soft,
  title  = {Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation},
  author = {Peidong Wang and Jian Xue and Jinyu Li and Junkun Chen and Aswin Shanmugam Subramanian},
  year   = {2024},
  eprint = {2406.10276},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.10276v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.10276.svg)](https://codeofpaper.com/paper/2406.10276)

Discussion