Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator

Vladimir Bataev, Roman Korostik, Evgeny Shabalin, Vitaly Lavrukhin, Boris Ginsburg
2/27/2023
eess.AScs.CLcs.LG

No official code found. Best implementation: NVIDIA-NeMo/NeMo (17,831 stars, Apache-2.0, last commit 2026-05-02).

Abstract

We propose an end-to-end Automatic Speech Recognition (ASR) system that can be trained on transcribed speech data, text-only data, or a mixture of both. The proposed model uses an integrated auxiliary block for text-based training. This block combines a non-autoregressive multi-speaker text-to-mel-spectrogram generator with a GAN-based enhancer to improve the spectrogram quality. The proposed system can generate a mel-spectrogram dynamically during training. It can be used to adapt the ASR model to a new domain by using text-only data from this domain. We demonstrate that the proposed training method significantly improves ASR accuracy compared to the system trained on transcribed speech only. It also surpasses cascade TTS systems with the vocoder in the adaptation quality and training speed.

View on arXivView PDF

Code Implementations(1)

NVIDIA-NeMo/NeMoCommunity100%
Apache-2.0

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

17,8313,521C++, CSSAug 5, 20195 months agoApache-2.0
asrdeeplearninggenerative-aimachine-translationneural-networks+5 more

Similar Papers

Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer

84% match2023

Unsupervised Pre-Training For Data-Efficient Text-to-Speech On Low Resource Languages

82% match1 repo272023

DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift

82% match2026

Instruction Data Generation and Unsupervised Adaptation for Speech Language Models

80% match2024

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers

80% match2025

Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Data

79% match1 repo2132020

Cite this paper

@article{bataev2023textonly,
  title  = {Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator},
  author = {Vladimir Bataev and Roman Korostik and Evgeny Shabalin and Vitaly Lavrukhin and Boris Ginsburg},
  year   = {2023},
  eprint = {2302.14036},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.14036v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.14036.svg)](https://codeofpaper.com/paper/2302.14036)

Discussion