SparQLe: Speech Queries to Text Translation Through LLMs

Amirbek Djanibekov, Hanan Aldarmaki
2/13/2025
cs.CLcs.AI

No official code found. Best implementation: djanibekov/rebooting-llm (4 stars, last commit 2025-10-05).

Abstract

With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that combines self-supervised speech representations with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English speech data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising approach for various speech understanding applications.

View on arXivView PDF

Code Implementations(1)

djanibekov/rebooting-llmCommunity100%

Implementation of SparQLe. SparQLe is a speech understanding and processing framework that integrates query-former (Qformer) architecture with large language models for speech-to-text applications.

41PythonApr 20, 20241 years ago

Similar Papers

Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation

84% match2025

Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs

83% match2025

Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024

83% match2024

Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models

83% match2025

Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving

83% match2025

PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs

82% match2026

Cite this paper

@article{djanibekov2025sparqle,
  title  = {SparQLe: Speech Queries to Text Translation Through LLMs},
  author = {Amirbek Djanibekov and Hanan Aldarmaki},
  year   = {2025},
  eprint = {2502.09284},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.09284v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.09284.svg)](https://codeofpaper.com/paper/2502.09284)

Discussion