Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

W. Ronny Huang, Cyril Allauzen, Tongzhou Chen, Kilol Gupta, Ke Hu +5 more
1/23/2024
cs.CLcs.SDeess.AS

No confident code implementation found for this paper. 5 possible matches can be shown below.

Abstract

In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 5 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition

87% match2025

Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM

86% match2025

Massively Multilingual Shallow Fusion with Large Language Models

85% match2023

An Embarrassingly Simple Approach for LLM with Strong ASR Capacity

85% match1 repo1.1k2024

Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration

85% match2025

Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR

85% match2026

Cite this paper

@article{huang2024multilingual,
  title  = {Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study},
  author = {W. Ronny Huang and Cyril Allauzen and Tongzhou Chen and Kilol Gupta and Ke Hu and James Qin and Yu Zhang and Yongqiang Wang and Shuo-Yiin Chang and Tara N. Sainath},
  year   = {2024},
  eprint = {2401.12789},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.12789v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.12789.svg)](https://codeofpaper.com/paper/2401.12789)

Discussion