Label-Looping: Highly Efficient Decoding for Transducers

Vladimir Bataev, Hainan Xu, Daniel Galvez, Vitaly Lavrukhin, Boris Ginsburg
6/10/2024
eess.AScs.AIcs.CL

No official code found. Best implementation: NVIDIA-NeMo/NeMo (17,831 stars, Apache-2.0, last commit 2026-05-02).

Abstract

This paper introduces a highly efficient greedy decoding algorithm for Transducer-based speech recognition models. We redesign the standard nested-loop design for RNN-T decoding, swapping loops over frames and labels: the outer loop iterates over labels, while the inner loop iterates over frames searching for the next non-blank symbol. Additionally, we represent partial hypotheses in a special structure using CUDA tensors, supporting parallelized hypotheses manipulations. Experiments show that the label-looping algorithm is up to 2.0X faster than conventional batched decoding when using batch size 32. It can be further combined with other compiler or GPU call-related techniques to achieve even more speedup. Our algorithm is general-purpose and can work with both conventional Transducers and Token-and-Duration Transducers. We open-source our implementation to benefit the research community.

View on arXivView PDF

Code Implementations(1)

NVIDIA-NeMo/NeMoCommunity100%
Apache-2.0

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

17,8313,521C++, CSSAug 5, 20195 months agoApache-2.0
asrdeeplearninggenerative-aimachine-translationneural-networks+5 more

Similar Papers

WIND: Accelerated RNN-T Decoding with Windowed Inference for Non-blank Detection

80% match2025

Breaking Through the Spike: Spike Window Decoding for Accelerated and Precise Automatic Speech Recognition

79% match2025

EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding

79% match1 repo2052015

NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding

79% match2025

WST: Weakly Supervised Transducer for Automatic Speech Recognition

79% match2025

Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text

79% match2026

Cite this paper

@article{bataev2024labellooping,
  title  = {Label-Looping: Highly Efficient Decoding for Transducers},
  author = {Vladimir Bataev and Hainan Xu and Daniel Galvez and Vitaly Lavrukhin and Boris Ginsburg},
  year   = {2024},
  eprint = {2406.06220},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.06220v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.06220.svg)](https://codeofpaper.com/paper/2406.06220)

Discussion