Soundwave: Less is More for Speech-Text Alignment in LLMs

Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang +1 more
2/18/2025
cs.CLcs.AIcs.SD

Official code: FreedomIntelligence/Soundwave (223 stars, Apache-2.0, last commit 2025-03-16).

Abstract

Existing end-to-end speech large language models (LLMs) usually rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth. We focus on two fundamental problems between speech and text: the representation space gap and sequence length inconsistency. We propose Soundwave, which utilizes an efficient training strategy and a novel architecture to address these issues. Results show that Soundwave outperforms the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks, using only one-fiftieth of the training data. Further analysis shows that Soundwave still retains its intelligence during conversation. The project is available at https://github.com/FreedomIntelligence/Soundwave.

View on arXivView PDF

Code Implementations(1)

FreedomIntelligence/SoundwaveOfficial100%
Apache-2.0

The official Soundwave repository

22320PythonFeb 18, 20251 years agoApache-2.0

Similar Papers

Closing the Gap Between Text and Speech Understanding in LLMs

84% match2025

ALARM: Audio-Language Alignment for Reasoning Models

84% match1 repo162026

Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data

84% match2025

From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data

83% match2025

DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment

82% match1 repo1402025

Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models

82% match2025

Cite this paper

@article{zhang2025soundwave,
  title  = {Soundwave: Less is More for Speech-Text Alignment in LLMs},
  author = {Yuhao Zhang and Zhiheng Liu and Fan Bu and Ruiyu Zhang and Benyou Wang and Haizhou Li},
  year   = {2025},
  eprint = {2502.12900},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.12900v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.12900.svg)](https://codeofpaper.com/paper/2502.12900)

Discussion