StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection

Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli
6/10/2024
cs.SDcs.AIcs.CL

No official code found. Best implementation: hlt-mt/FBK-fairseq (60 stars, PyTorch, last commit 2026-03-19).

Abstract

Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.

View on arXivView PDF

Code Implementations(1)

hlt-mt/FBK-fairseqCommunity100%
PyTorch

Repository containing the open source code of works published at the FBK MT unit.

605C++, LuaApr 2, 20227 months agoNOASSERTION
deep-learninggender-biaspytorchsimultaneous-translationspeech-to-text+2 more

Similar Papers

StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model

85% match2025

FASST: Fast LLM-based Simultaneous Speech Translation

84% match2024

Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

84% match2025

Streaming Translation and Transcription Through Speech-to-Text Causal Alignment

83% match2026

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning

82% match1 repo1.3k2024

Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

82% match1 repo322023

Cite this paper

@article{papi2024streamatt,
  title  = {StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection},
  author = {Sara Papi and Marco Gaido and Matteo Negri and Luisa Bentivogli},
  year   = {2024},
  eprint = {2406.06097},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.06097v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.06097.svg)](https://codeofpaper.com/paper/2406.06097)

Discussion