Improving End-to-End SLU performance with Prosodic Attention and Distillation

Shangeth Rajaa
5/14/2023
cs.CLcs.LGcs.SD

No official code found. Best implementation: skit-ai/slu-prosody (27 stars, last commit 2023-05-17).

Abstract

Most End-to-End SLU methods depend on the pretrained ASR or language model features for intent prediction. However, other essential information in speech, such as prosody, is often ignored. Recent research has shown improved results in classifying dialogue acts by incorporating prosodic information. The margins of improvement in these methods are minimal as the neural models ignore prosodic features. In this work, we propose prosody-attention, which uses the prosodic features differently to generate attention maps across time frames of the utterance. Then we propose prosody-distillation to explicitly learn the prosodic information in the acoustic encoder rather than concatenating the implicit prosodic features. Both the proposed methods improve the baseline results, and the prosody-distillation method gives an intent classification accuracy improvement of 8\% and 2\% on SLURP and STOP datasets over the prosody baseline.

View on arXivView PDF

Code Implementations(1)

skit-ai/slu-prosodyCommunity100%

Code repository for the paper "Improving End-to-End SLU performance with Prosodic Attention and Distillation" accepted at Interspeech 2023.

273Python, Jupyter NotebookMar 9, 20233 years ago

Similar Papers

ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models

84% match1 repo462025

PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding

80% match2024

ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs

80% match2025

The Role of Prosody in Spoken Question Answering

80% match2025

Non-verbal information in spontaneous speech -- towards a new framework of analysis

80% match2024

Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning

80% match2025

Cite this paper

@article{rajaa2023improving,
  title  = {Improving End-to-End SLU performance with Prosodic Attention and Distillation},
  author = {Shangeth Rajaa},
  year   = {2023},
  eprint = {2305.08067},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.08067v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.08067.svg)](https://codeofpaper.com/paper/2305.08067)

Discussion