Enhancing Multimodal Affective Analysis with Learned Live Comment Features

Zhaoyuan Deng, Amith Ananthram, Kathleen McKeown
10/21/2024
cs.CLcs.AIcs.MM

No official code found. Best implementation: dzy49/AffectiveLiveComm (1 star, last commit 2024-12-17).

Abstract

Live comments, also known as Danmaku, are user-generated messages that are synchronized with video content. These comments overlay directly onto streaming videos, capturing viewer emotions and reactions in real-time. While prior work has leveraged live comments in affective analysis, its use has been limited due to the relative rarity of live comments across different video platforms. To address this, we first construct the Live Comment for Affective Analysis (LCAffect) dataset which contains live comments for English and Chinese videos spanning diverse genres that elicit a wide spectrum of emotions. Then, using this dataset, we use contrastive learning to train a video encoder to produce synthetic live comment features for enhanced multimodal affective content analysis. Through comprehensive experimentation on a wide range of affective analysis tasks (sentiment, emotion recognition, and sarcasm detection) in both English and Chinese, we demonstrate that these synthetic live comment features significantly improve performance over state-of-the-art methods.

View on arXivView PDF

Code Implementations(1)

dzy49/AffectiveLiveCommCommunity64%

Code for "Enhancing Multimodal Affective Analysis with Learned Live Comment Features" -- Accepted at AAAI 2025

10Dec 17, 20241 years ago

Similar Papers

Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting

83% match1 repo32024

Semantic Frame Aggregation-based Transformer for Live Video Comment Generation

83% match1 repo32025

Infer Induced Sentiment of Comment Response to Video: A New Task, Dataset and Baseline

80% match1 repo72026

GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysis

79% match2024

VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models

79% match1 repo152025

MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations

78% match2025

Cite this paper

@article{deng2024enhancing,
  title  = {Enhancing Multimodal Affective Analysis with Learned Live Comment Features},
  author = {Zhaoyuan Deng and Amith Ananthram and Kathleen McKeown},
  year   = {2024},
  eprint = {2410.16407},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.16407v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.16407.svg)](https://codeofpaper.com/paper/2410.16407)

Discussion