Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection

Dennis Fucci, Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri +1 more
10/24/2023
cs.CLcs.AI

No official code found. Best implementation: hlt-mt/FBK-fairseq (60 stars, PyTorch, last commit 2026-03-19).

Abstract

When translating words referring to the speaker, speech translation (ST) systems should not resort to default masculine generics nor rely on potentially misleading vocal traits. Rather, they should assign gender according to the speakers' preference. The existing solutions to do so, though effective, are hardly feasible in practice as they involve dedicated model re-training on gender-labeled ST data. To overcome these limitations, we propose the first inference-time solution to control speaker-related gender inflections in ST. Our approach partially replaces the (biased) internal language model (LM) implicitly learned by the ST decoder with gender-specific external LMs. Experiments on en->es/fr/it show that our solution outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. The gains are even larger (up to 32.0 and 3.4) in the challenging condition where speakers' vocal traits conflict with their gender.

View on arXivView PDF

Code Implementations(1)

hlt-mt/FBK-fairseqCommunity100%
PyTorch

Repository containing the open source code of works published at the FBK MT unit.

605C++, LuaApr 2, 20227 months agoNOASSERTION
deep-learninggender-biaspytorchsimultaneous-translationspeech-to-text+2 more

Similar Papers

How To Build Competitive Multi-gender Speech Translation Models For Controlling Speaker Gender Translation

94% match1 repo602023

Addressing speaker gender bias in large scale speech translation systems

90% match2025

Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation

88% match2025

Different Speech Translation Models Encode and Translate Speaker Gender Differently

88% match2 repos12025

ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation

88% match2026

Gender-specific Machine Translation with Large Language Models

88% match2023

Cite this paper

@article{fucci2023integrating,
  title  = {Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection},
  author = {Dennis Fucci and Marco Gaido and Sara Papi and Mauro Cettolo and Matteo Negri and Luisa Bentivogli},
  year   = {2023},
  eprint = {2310.15752},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.15752v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.15752.svg)](https://codeofpaper.com/paper/2310.15752)

Discussion