Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition

Dongji Gao, Hainan Xu, Desh Raj, Leibny Paola Garcia Perera, Daniel Povey +1 more
9/26/2023
eess.AScs.CLcs.LG

Official code: k2-fsa/icefall (1,445 stars, Apache-2.0, last commit 2026-04-12).

Abstract

Training automatic speech recognition (ASR) systems requires large amounts of well-curated paired data. However, human annotators usually perform "non-verbatim" transcription, which can result in poorly trained models. In this paper, we propose Omni-temporal Classification (OTC), a novel training criterion that explicitly incorporates label uncertainties originating from such weak supervision. This allows the model to effectively learn speech-text alignments while accommodating errors present in the training transcripts. OTC extends the conventional CTC objective for imperfect transcripts by leveraging weighted finite state transducers. Through experiments conducted on the LibriSpeech and LibriVox datasets, we demonstrate that training ASR models with OTC avoids performance degradation even with transcripts containing up to 70% errors, a scenario where CTC models fail completely. Our implementation is available at https://github.com/k2-fsa/icefall.

View on arXivView PDF

Code Implementations(1)

k2-fsa/icefallOfficial100%
Apache-2.0
1,445416Perl, ShellJun 26, 20216 months agoApache-2.0

Similar Papers

Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcripts

89% match2023

WST: Weakly Supervised Transducer for Automatic Speech Recognition

84% match2025

From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data

81% match2025

Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards

81% match2026

Less Peaky and More Accurate CTC Forced Alignment by Label Priors

80% match1 repo112024

Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM

80% match2025

Cite this paper

@article{gao2023learning,
  title  = {Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition},
  author = {Dongji Gao and Hainan Xu and Desh Raj and Leibny Paola Garcia Perera and Daniel Povey and Sanjeev Khudanpur},
  year   = {2023},
  eprint = {2309.15796},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.15796v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.15796.svg)](https://codeofpaper.com/paper/2309.15796)

Discussion