A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion

Zeinab Sadat Taghavi, Ali Satvaty, Hossein Sameti
7/21/2023
cs.SDcs.CLcs.LG

Official code: ICLR2023AChangeOfHeart/meld-modality-conversion (3 stars, MIT, last commit 2023-04-19).

Abstract

Speech Emotion Recognition (SER) is a challenging task. In this paper, we introduce a modality conversion concept aimed at enhancing emotion recognition performance on the MELD dataset. We assess our approach through two experiments: first, a method named Modality-Conversion that employs automatic speech recognition (ASR) systems, followed by a text classifier; second, we assume perfect ASR output and investigate the impact of modality conversion on SER, this method is called Modality-Conversion++. Our findings indicate that the first method yields substantial results, while the second method outperforms state-of-the-art (SOTA) speech-based approaches in terms of SER weighted-F1 (WF1) score on the MELD dataset. This research highlights the potential of modality conversion for tasks that can be conducted in alternative modalities.

View on arXivView PDF

Code Implementations(1)

ICLR2023AChangeOfHeart/meld-modality-conversionOfficial100%
MIT
30Jupyter NotebookApr 17, 20233 years agoMIT

Similar Papers

MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction

84% match2024

On the Contribution of Lexical Features to Speech Emotion Recognition

84% match2025

Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques

82% match1 repo52024

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

82% match2024

Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speech

81% match2025

Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

81% match2024

Cite this paper

@article{taghavi2023change,
  title  = {A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion},
  author = {Zeinab Sadat Taghavi and Ali Satvaty and Hossein Sameti},
  year   = {2023},
  eprint = {2307.11584},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.11584v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.11584.svg)](https://codeofpaper.com/paper/2307.11584)

Discussion