Exploring the limits of decoder-only models trained on public speech recognition corpora

Ankit Gupta, George Saon, Brian Kingsbury
1/31/2024
cs.CLcs.SDeess.AS

No official code found. Best implementation: ag1988/mel-asr (21 stars, MIT, last commit 2024-10-11).

Abstract

The emergence of industrial-scale speech recognition (ASR) models such as Whisper and USM, trained on 1M hours of weakly labelled and 12M hours of audio only proprietary data respectively, has led to a stronger need for large scale public ASR corpora and competitive open source pipelines. Unlike the said models, large language models are typically based on Transformer decoders, and it remains unclear if decoder-only models trained on public data alone can deliver competitive performance. In this work, we investigate factors such as choice of training datasets and modeling components necessary for obtaining the best performance using public English ASR corpora alone. Our Decoder-Only Transformer for ASR (DOTA) model comprehensively outperforms the encoder-decoder open source replication of Whisper (OWSM) on nearly all English ASR benchmarks and outperforms Whisper large-v3 on 7 out of 15 test sets. We release our codebase and model checkpoints under permissive license.

View on arXivView PDF

Code Implementations(1)

ag1988/mel-asrCommunity64%
MIT

The accompanying code for "Exploring the limits of decoder-only models trained on public speech recognition corpora" (Ankit Gupta, George Saon, Brian Kingsbury. Interspeech 2024).

211PythonFeb 3, 20242 years agoMIT

Similar Papers

Anatomy of Industrial Scale Multilingual ASR

83% match2024

OLMoASR: Open Models and Data for Training Robust Speech Recognition Models

83% match2025

Efficient Compression of Multitask Multilingual Speech Models

83% match2024

Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words

83% match2024

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

82% match1 repo2402026

Less is More: Accurate Speech Recognition & Translation without Web-Scale Data

82% match2024

Cite this paper

@article{gupta2024exploring,
  title  = {Exploring the limits of decoder-only models trained on public speech recognition corpora},
  author = {Ankit Gupta and George Saon and Brian Kingsbury},
  year   = {2024},
  eprint = {2402.00235},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.00235v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.00235.svg)](https://codeofpaper.com/paper/2402.00235)

Discussion