TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR

Shashi Kumar, Srikanth Madikeri, Juan Zuluaga-Gomez, Iuliia Thorbecke, Esaú Villatoro-Tello +4 more
7/5/2024
cs.CLcs.SDeess.AS

Official code: idiap/tokenverse-unifying-speech-nlp (4 stars, last commit 2024-10-04).

Abstract

In traditional conversational intelligence from speech, a cascaded pipeline is used, involving tasks such as voice activity detection, diarization, transcription, and subsequent processing with different NLP models for tasks like semantic endpointing and named entity recognition (NER). Our paper introduces TokenVerse, a single Transducer-based model designed to handle multiple tasks. This is achieved by integrating task-specific tokens into the reference text during ASR model training, streamlining the inference and eliminating the need for separate NLP models. In addition to ASR, we conduct experiments on 3 different tasks: speaker change detection, endpointing, and NER. Our experiments on a public and a private dataset show that the proposed method improves ASR by up to 7.7% in relative WER while outperforming the cascaded pipeline approach in individual task performance. Our code is publicly available: https://github.com/idiap/tokenverse-unifying-speech-nlp

View on arXivView PDF

Code Implementations(1)

idiap/tokenverse-unifying-speech-nlpOfficial100%

Coming soon

40Oct 4, 20242 years ago

Similar Papers

TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation

86% match2025

TokenChain: A Discrete Speech Chain via Semantic Token Modeling

82% match2025

Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction

80% match1 repo602024

WhisperNER: Unified Open Named Entity and Speech Recognition

79% match1 repo1992024

Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts

79% match2025

Mind the Gap: Entity-Preserved Context-Aware ASR Structured Transcriptions

79% match2025

Cite this paper

@article{kumar2024tokenverse,
  title  = {TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR},
  author = {Shashi Kumar and Srikanth Madikeri and Juan Zuluaga-Gomez and Iuliia Thorbecke and Esaú Villatoro-Tello and Sergio Burdisso and Petr Motlicek and Karthik Pandia and Aravind Ganapathiraju},
  year   = {2024},
  eprint = {2407.04444},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.04444v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.04444.svg)](https://codeofpaper.com/paper/2407.04444)

Discussion