Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding

Yi Xuan Tan, Navonil Majumder, Soujanya Poria
5/20/2023
cs.CLcs.AIcs.SD

No official code found. Best implementation: declare-lab/segue (9 stars, last commit 2023-05-29).

Abstract

The pre-trained speech encoder wav2vec 2.0 performs very well on various spoken language understanding (SLU) tasks. However, on many tasks, it trails behind text encoders with textual input. To improve the understanding capability of SLU encoders, various studies have used knowledge distillation to transfer knowledge from natural language understanding (NLU) encoders. We use a very simple method of distilling from a textual sentence embedder directly into wav2vec 2.0 as pre-training, utilizing paired audio-text datasets. We observed that this method is indeed capable of improving SLU task performance in fine-tuned settings, as well as full-data and few-shot transfer on a frozen encoder. However, the model performs worse on certain tasks highlighting the strengths and weaknesses of our approach.

View on arXivView PDF

Code Implementations(1)

declare-lab/segueCommunity100%

Codes and Checkpoints of the Interspeech paper "Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding"

91PythonAug 18, 20223 years ago
speech-synthesis

Similar Papers

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

86% match1 repo32.2k2020

Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23

85% match2023

wav2vec: Unsupervised Pre-training for Speech Recognition

85% match1 repo32.2k2019

Unsupervised Speech Recognition

84% match1 repo32.2k2021

Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Training

83% match2022

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

81% match2024

Cite this paper

@article{tan2023sentence,
  title  = {Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding},
  author = {Yi Xuan Tan and Navonil Majumder and Soujanya Poria},
  year   = {2023},
  eprint = {2305.12301},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.12301v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.12301.svg)](https://codeofpaper.com/paper/2305.12301)

Discussion