Mapping Memes to Words for Multimodal Hateful Meme Classification

Giovanni Burbi, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, Alberto Del Bimbo
10/12/2023
cs.CV

Official code: miccunifi/ISSUES (27 stars, MIT, last commit 2025-04-17).

Abstract

Multimodal image-text memes are prevalent on the internet, serving as a unique form of communication that combines visual and textual elements to convey humor, ideas, or emotions. However, some memes take a malicious turn, promoting hateful content and perpetuating discrimination. Detecting hateful memes within this multimodal context is a challenging task that requires understanding the intertwined meaning of text and images. In this work, we address this issue by proposing a novel approach named ISSUES for multimodal hateful meme classification. ISSUES leverages a pre-trained CLIP vision-language model and the textual inversion technique to effectively capture the multimodal semantic content of the memes. The experiments show that our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets. The code and the pre-trained models are publicly available at https://github.com/miccunifi/ISSUES.

View on arXivView PDF

Code Implementations(1)

miccunifi/ISSUESOfficial100%
MIT

[ICCVW 2023] - Mapping Memes to Words for Multimodal Hateful Meme Classification

277Shell, PythonAug 19, 20231 years agoMIT

Similar Papers

MIMIC: Multimodal Islamophobic Meme Identification and Classification

90% match2024

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

89% match2025

MemeBLIP2: A novel lightweight multimodal system to detect harmful memes

88% match2025

Prompting for Multimodal Hateful Meme Classification

88% match2023

Decoding the Underlying Meaning of Multimodal Hateful Memes

88% match1 repo232023

Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions

88% match2025

Cite this paper

@article{burbi2023mapping,
  title  = {Mapping Memes to Words for Multimodal Hateful Meme Classification},
  author = {Giovanni Burbi and Alberto Baldrati and Lorenzo Agnolucci and Marco Bertini and Alberto Del Bimbo},
  year   = {2023},
  eprint = {2310.08368},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.08368v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.08368.svg)](https://codeofpaper.com/paper/2310.08368)

Discussion