SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli
6/20/2024
cs.CLcs.AIcs.SD

Official code: hlt-mt/FBK-fairseq (60 stars, PyTorch, last commit 2026-03-19).

Abstract

This paper describes the FBK's participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year's submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used "off-the-shelf" and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English->{German, Japanese, Chinese}, and Czech->English), achieving acceptable or even better results compared to last year's submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.

View on arXivView PDF

Code Implementations(1)

hlt-mt/FBK-fairseqOfficial100%
PyTorch

Repository containing the open source code of works published at the FBK MT unit.

605C++, LuaApr 2, 20227 months agoNOASSERTION
deep-learninggender-biaspytorchsimultaneous-translationspeech-to-text+2 more

Cite this paper

@article{papi2024simulseamless,
  title  = {SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation},
  author = {Sara Papi and Marco Gaido and Matteo Negri and Luisa Bentivogli},
  year   = {2024},
  eprint = {2406.14177},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.14177v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.14177.svg)](https://codeofpaper.com/paper/2406.14177)

Discussion