Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint

Heekyung Lee, Jiaxin Ge, Tsung-Han Wu, Minwoo Kang, Trevor Darrell +1 more

5/29/2025

cs.CLcs.AIcs.CV

No official code found. Best implementation: Kyunnilee/visual_puzzles (15 stars, last commit 2025-09-22).

Abstract

Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs). Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns. In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels"). We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.

View on arXiv View PDF

Code Implementations(1)

Kyunnilee/visual_puzzlesCommunity100%

🧩 Official code repository for “Puzzled by Puzzles: When Vision-Language Models Can’t Take a Hint.”

152Shell, PythonMay 28, 20251 years ago

Similar Papers

$\left|\,\circlearrowright\,\boxed{\text{BUS}}\,\right|$: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles

89% match2025

Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles

88% match2026

Eye-Q: A Multilingual Benchmark for Visual Word Puzzle Solving and Image-to-Phrase Reasoning

88% match2026

Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

87% match2026

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

86% match1 repo132024

Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models

86% match2026

Cite this paper

@article{lee2025puzzled,
  title  = {Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint},
  author = {Heekyung Lee and Jiaxin Ge and Tsung-Han Wu and Minwoo Kang and Trevor Darrell and David M. Chan},
  year   = {2025},
  eprint = {2505.23759},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.23759v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

[![Code of Paper](https://codeofpaper.com/badge/2505.23759.svg)](https://codeofpaper.com/paper/2505.23759)