Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

Brian Yan, Vineel Pratap, Shinji Watanabe, Michael Auli
9/27/2024
cs.CLcs.SDeess.AS

No official code found. Best implementation: facebookresearch/fairseq (32,213 stars, MIT, PyTorch, last commit 2025-09-30).

Abstract

Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical settings. Automatic Spoken Language Identification (SLID) models are not perfect and misclassifications have a substantial impact on the final ASR accuracy. In this paper, we present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy for several prominent acoustic models by employing external features such as language models and text-based language identification models. Our results on FLEURS using the MMS and Whisper models show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively and word error rates which are 3.3% and 2.0% lower on these benchmarks.

View on arXivView PDF

Code Implementations(1)

facebookresearch/fairseqCommunity100%
PyTorchMIT

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

32,2136,667C++, LuaAug 29, 20171 years agoMIT
artificial-intelligencepythonpytorch

Cite this paper

@article{yan2024improving,
  title  = {Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking},
  author = {Brian Yan and Vineel Pratap and Shinji Watanabe and Michael Auli},
  year   = {2024},
  eprint = {2409.18428},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.18428v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.18428.svg)](https://codeofpaper.com/paper/2409.18428)

Discussion