Word Level Timestamp Generation for Automatic Speech Recognition and Translation

Ke Hu, Krishna Puvvada, Elena Rastorgueva, Zhehuai Chen, He Huang +5 more
5/21/2025
cs.CLcs.SDeess.AS

No official code found. Best implementation: NVIDIA-NeMo/NeMo (17,831 stars, Apache-2.0, last commit 2026-05-02).

Abstract

We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks such as speech content retrieval and timed subtitles. While traditional hybrid systems and end-to-end (E2E) models may employ external modules for timestamp prediction, our approach eliminates the need for separate alignment mechanisms. By leveraging the NeMo Forced Aligner (NFA) as a teacher model, we generate word-level timestamps and train the Canary model to predict timestamps directly. We introduce a new <|timestamp|> token, enabling the Canary model to predict start and end timestamps for each word. Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages, with minimal WER degradation. Additionally, we extend our system to automatic speech translation (AST) tasks, achieving timestamp prediction errors around 200 milliseconds.

View on arXivView PDF

Code Implementations(1)

NVIDIA-NeMo/NeMoCommunity100%
Apache-2.0

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

17,8313,521C++, CSSAug 5, 20195 months agoApache-2.0
asrdeeplearninggenerative-aimachine-translationneural-networks+5 more

Similar Papers

In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

87% match2026

Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

80% match2024

Timestamped Embedding-Matching Acoustic-to-Word CTC ASR

80% match2023

Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model

80% match1 repo19.4k2023

Set the Clock: Temporal Alignment of Pretrained Language Models

79% match1 repo322024

Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST

79% match2025

Cite this paper

@article{hu2025word,
  title  = {Word Level Timestamp Generation for Automatic Speech Recognition and Translation},
  author = {Ke Hu and Krishna Puvvada and Elena Rastorgueva and Zhehuai Chen and He Huang and Shuoyang Ding and Kunal Dhawan and Hainan Xu and Jagadeesh Balam and Boris Ginsburg},
  year   = {2025},
  eprint = {2505.15646},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.15646v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.15646.svg)](https://codeofpaper.com/paper/2505.15646)

Discussion