Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?

Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, Joyce Chai
10/31/2023
cs.AIcs.CLcs.CV

No official code found. Best implementation: vl-illusion/GVIL (15 stars, last commit 2024-01-25).

Abstract

Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world. However, known as visual illusions, human's perception of reality isn't always faithful to the physical world. This raises a key question: do VLMs have the similar kind of illusions as humans do, or do they faithfully learn to represent reality? To investigate this question, we build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusions in state-of-the-art VLMs. Our findings have shown that although the overall alignment is low, larger models are closer to human perception and more susceptible to visual illusions. Our dataset and initial findings will promote a better understanding of visual illusions in humans and machines and provide a stepping stone for future computational models that can better align humans and machines in perceiving and communicating about the shared visual world. The code and data are available at https://github.com/vl-illusion/dataset.

View on arXivView PDF

Code Implementations(1)

vl-illusion/GVILCommunity68%

Code and data for EMNLP 2023 paper "Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?"

151PythonOct 23, 20232 years ago

Similar Papers

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

89% match1 repo242024

Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

89% match2026

Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions?

89% match1 repo2025

Evaluating Model Perception of Color Illusions in Photorealistic Scenes

88% match2 repos12026

The Illusion-Illusion: Vision Language Models See Illusions Where There are None

88% match2024

SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking

87% match2025

Cite this paper

@article{zhang2023grounding,
  title  = {Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?},
  author = {Yichi Zhang and Jiayi Pan and Yuchen Zhou and Rui Pan and Joyce Chai},
  year   = {2023},
  eprint = {2311.00047},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.00047v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.00047.svg)](https://codeofpaper.com/paper/2311.00047)

Discussion