Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models

Loren Lugosch, Brett Meyer, Derek Nowrouzezahrai, Mirco Ravanelli
10/21/2019
eess.AS

Official code: lorenlugosch/end-to-end-SLU (231 stars, Apache-2.0, last commit 2021-03-23).

Abstract

End-to-end models are an attractive new approach to spoken language understanding (SLU) in which the meaning of an utterance is inferred directly from the raw audio without employing the standard pipeline composed of a separately trained speech recognizer and natural language understanding module. The downside of end-to-end SLU is that in-domain speech data must be recorded to train the model. In this paper, we propose a strategy for overcoming this requirement in which speech synthesis is used to generate a large synthetic training dataset from several artificial speakers. Experiments on two open-source SLU datasets confirm the effectiveness of our approach, both as a sole source of training data and as a form of data augmentation.

View on arXivView PDF

Code Implementations(1)

lorenlugosch/end-to-end-SLUOfficial100%
Apache-2.0

PyTorch code for end-to-end spoken language understanding (SLU) with ASR-based transfer learning

23150PythonMar 5, 20195 years agoApache-2.0

Similar Papers

Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis

85% match2023

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

84% match1 repo22.2k2023

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

84% match2022

Instruction Data Generation and Unsupervised Adaptation for Speech Language Models

84% match2024

Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learning

83% match1 repo132023

Two-Pass Low Latency End-to-End Spoken Language Understanding

83% match2022

Cite this paper

@article{lugosch2019using,
  title  = {Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models},
  author = {Loren Lugosch and Brett Meyer and Derek Nowrouzezahrai and Mirco Ravanelli},
  year   = {2019},
  eprint = {1910.09463},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1910.09463}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1910.09463.svg)](https://codeofpaper.com/paper/1910.09463)

Discussion