Cascaded Cross-Modal Transformer for Request and Complaint Detection

Nicolae-Catalin Ristea, Radu Tudor Ionescu
7/27/2023
cs.CLcs.LGcs.MM

No official code found. Best implementation: ristea/ccmt (3 stars, MIT, last commit 2023-08-11).

Abstract

We propose a novel cascaded cross-modal transformer (CCMT) that combines speech and text transcripts to detect customer requests and complaints in phone conversations. Our approach leverages a multimodal paradigm by transcribing the speech using automatic speech recognition (ASR) models and translating the transcripts into different languages. Subsequently, we combine language-specific BERT-based models with Wav2Vec2.0 audio features in a novel cascaded cross-attention transformer model. We apply our system to the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for the complaint and request classes, respectively.

View on arXivView PDF

Code Implementations(1)

ristea/ccmtCommunity65%
MIT

Cascaded Cross-Modal Transformer for Request and Complaint Detection

30PythonJul 29, 20233 years agoMIT

Similar Papers

Cascaded Cross-Modal Transformer for Audio-Textual Classification

86% match1 repo32024

Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers

82% match2023

HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition

82% match2023

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

82% match1 repo772023

Exploring Attention Mechanisms for Multimodal Emotion Recognition in an Emergency Call Center Corpus

81% match2023

Speech Recognition Transformers: Topological-lingualism Perspective

81% match2024

Cite this paper

@article{ristea2023cascaded,
  title  = {Cascaded Cross-Modal Transformer for Request and Complaint Detection},
  author = {Nicolae-Catalin Ristea and Radu Tudor Ionescu},
  year   = {2023},
  eprint = {2307.15097},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.15097v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.15097.svg)](https://codeofpaper.com/paper/2307.15097)

Discussion