MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu +2 more
3/19/2024
cs.CLcs.SDeess.AS

No confident code implementation found for this paper.

Abstract

There have been emerging research interest and advances in speech-to-speech translation (S2ST), translating utterances from one language to another. This work proposes Multitask Speech Language Model (MSLM), which is a decoder-only speech language model trained in a multitask setting. Without reliance on text training data, our model is able to support multilingual S2ST with speaker style preserved.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought

85% match2024

Direct Speech to Speech Translation: A Review

84% match2025

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning

81% match1 repo1.3k2024

Investigating Decoder-only Large Language Models for Speech-to-text Translation

81% match2024

MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages

80% match1 repo182025

Using Phonemes in cascaded S2S translation pipeline

80% match1 repo2025

Cite this paper

@article{peng2024mslmsst,
  title  = {MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation},
  author = {Yifan Peng and Ilia Kulikov and Yilin Yang and Sravya Popuri and Hui Lu and Changhan Wang and Hongyu Gong},
  year   = {2024},
  eprint = {2403.12408},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.12408v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.12408.svg)](https://codeofpaper.com/paper/2403.12408)

Discussion