LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura
7/22/2024
cs.CL

Official code: openaudiolab/LLaST (26 stars, Apache-2.0, last commit 2024-08-11).

Abstract

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.

View on arXivView PDF

Code Implementations(1)

openaudiolab/LLaSTOfficial100%
Apache-2.0

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

262PythonMay 14, 20242 years agoApache-2.0

Similar Papers

PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs

88% match2026

Investigating Decoder-only Large Language Models for Speech-to-text Translation

87% match2024

End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMs

87% match2025

Speech Translation Refinement using Large Language Models

86% match1 repo12025

Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?

86% match2024

Zero-resource Speech Translation and Recognition with LLMs

85% match2024

Cite this paper

@article{chen2024llast,
  title  = {LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models},
  author = {Xi Chen and Songyang Zhang and Qibing Bai and Kai Chen and Satoshi Nakamura},
  year   = {2024},
  eprint = {2407.15415},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.15415v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.15415.svg)](https://codeofpaper.com/paper/2407.15415)

Discussion