Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

Ohad Cohen, Gershon Hazan, Sharon Gannot
9/14/2024
cs.SDcs.LGeess.AS

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

This paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance emotion recognition accuracy in challenging acoustic conditions. Our approach combines a modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-channel audio processing with an R(2+1)D Convolutional Neural Networks (CNN) model for video analysis. We evaluate our proposed method on a reverberated version of the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using synthetic and real-world Room Impulse Responsess (RIRs). Our results demonstrate that integrating audio and video modalities yields superior performance compared to uni-modal approaches, especially in challenging acoustic conditions. Moreover, we show that the multimodal (audiovisual) approach that utilizes multiple microphones outperforms its single-microphone counterpart.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Multi-Microphone Speech Emotion Recognition using the Hierarchical Token-semantic Audio Transformer Architecture

91% match2024

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

83% match1 repo92024

Explainable Transformer-CNN Fusion for Noise-Robust Speech Emotion Recognition

82% match2025

Versatile audio-visual learning for emotion recognition

82% match2023

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

82% match2024

Dynamic Graph Neural ODE Network for Multi-modal Emotion Recognition in Conversation

82% match2024

Cite this paper

@article{cohen2024multimicrophone,
  title  = {Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment},
  author = {Ohad Cohen and Gershon Hazan and Sharon Gannot},
  year   = {2024},
  eprint = {2409.09545},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.09545v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.09545.svg)](https://codeofpaper.com/paper/2409.09545)

Discussion