Improving Massively Multilingual ASR With Auxiliary CTC Objectives

William Chen, Brian Yan, Jiatong Shi, Yifan Peng, Soumi Maiti +1 more
2/24/2023
cs.CLcs.SDeess.AS

Official code: espnet/espnet (9,991 stars, Apache-2.0, PyTorch, last commit 2026-04-27).

Abstract

Multilingual Automatic Speech Recognition (ASR) models have extended the usability of speech technologies to a wide variety of languages. With how many languages these models have to handle, however, a key to understanding their imbalanced performance across different languages is to examine if the model actually knows which language it should transcribe. In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID). We investigate techniques inspired from recent Connectionist Temporal Classification (CTC) studies to help the model handle the large number of languages, conditioning on the LID predictions of auxiliary tasks. Our experimental results demonstrate the effectiveness of our technique over standard CTC/Attention-based hybrid models. Furthermore, our state-of-the-art systems using self-supervised models with the Conformer architecture improve over the results of prior work on FLEURS by a relative 28.4% CER. Trained models and reproducible recipes are available at https://github.com/espnet/espnet/tree/master/egs2/fleurs/asr1 .

View on arXivView PDF

Code Implementations(1)

espnet/espnetOfficial100%
PyTorchApache-2.0

End-to-End Speech Processing Toolkit

9,9912,444C++, CudaDec 13, 20175 months agoApache-2.0
chainerdeep-learningend-to-endkaldimachine-translation+11 more

Similar Papers

Investigating model performance in language identification: beyond simple error statistics

86% match1 repo32023

ML-SUPERB: Multilingual Speech Universal PERformance Benchmark

86% match2 repos3112023

Benchmarking Transformers-based models on French Spoken Language Understanding tasks

85% match2022

UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

85% match2023

LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families

84% match2026

ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets

84% match2024

Cite this paper

@article{chen2023improving,
  title  = {Improving Massively Multilingual ASR With Auxiliary CTC Objectives},
  author = {William Chen and Brian Yan and Jiatong Shi and Yifan Peng and Soumi Maiti and Shinji Watanabe},
  year   = {2023},
  eprint = {2302.12829},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.12829v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.12829.svg)](https://codeofpaper.com/paper/2302.12829)

Discussion