Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning

Feng-Ting Liao, Yung-Chieh Chan, Yi-Chang Chen, Chan-Jan Hsu, Da-shan Shiu
7/18/2023
eess.AScs.CLcs.LG

Official code: mtkresearch/clairaudience (26 stars, Apache-2.0, last commit 2023-10-10).

Abstract

In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt contexts, with our model gaining a Word Error Rate (WER) reduction of up to 33% on unseen datasets from various domains, such as medical conversation, air traffic control communication, and financial meetings. Considering the limited availability of audio-transcript pair data, we further extend our method to text-only fine-tuning to achieve domain sensitivity as well as domain adaptation. We demonstrate that our text-only fine-tuned model can also attend to various prompt contexts, with the model reaching the most WER reduction of 29% on the medical conversation dataset.

View on arXivView PDF

Code Implementations(1)

mtkresearch/clairaudienceOfficial100%
Apache-2.0

Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning (ASRU2023)

263PythonJul 18, 20233 years agoApache-2.0

Similar Papers

Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning

86% match2025

Transliterated Zero-Shot Domain Adaptation for Automatic Speech Recognition

84% match2024

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers

83% match2025

ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis

83% match2026

Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models

82% match2023

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

82% match1 repo1512023

Cite this paper

@article{liao2023zeroshot,
  title  = {Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning},
  author = {Feng-Ting Liao and Yung-Chieh Chan and Yi-Chang Chen and Chan-Jan Hsu and Da-shan Shiu},
  year   = {2023},
  eprint = {2307.10274},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.10274v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.10274.svg)](https://codeofpaper.com/paper/2307.10274)

Discussion