Vision LLMs Are Bad at Hierarchical Visual Understanding, and LLMs Are the Bottleneck

Yuwen Tan, Yuan Qing, Boqing Gong
5/30/2025
cs.CVcs.AIcs.CL

No official code found. Best implementation: yuanqing-ai/LLM-Hierarchical-Consistency (16 stars, MIT, last commit 2025-11-10).

Abstract

This paper reveals that many state-of-the-art large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual understanding (e.g., recognizing Anemone Fish but not Vertebrate). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect to some extent because the VQA tasks improve the LLM's hierarchical consistency more than the vision LLM's. We conjecture that one cannot make vision LLMs understand visual concepts fully hierarchical until LLMs possess corresponding taxonomy knowledge.

View on arXivView PDF

Code Implementations(1)

yuanqing-ai/LLM-Hierarchical-ConsistencyCommunity62%
MIT

Official implementation of "Vision LLMs Are Bad at Hierarchical Visual Understanding, and LLMs Are the Bottleneck"

161Shell, PythonMay 22, 202511 months agoMIT

Similar Papers

Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

87% match2026

Intriguing Properties of Large Language and Vision Models

85% match1 repo82024

Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMs

85% match2026

Where do Large Vision-Language Models Look at when Answering Questions?

85% match1 repo742025

Hidden in plain sight: VLMs overlook their visual representations

85% match2025

How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding

85% match2025

Cite this paper

@article{tan2025vision,
  title  = {Vision LLMs Are Bad at Hierarchical Visual Understanding, and LLMs Are the Bottleneck},
  author = {Yuwen Tan and Yuan Qing and Boqing Gong},
  year   = {2025},
  eprint = {2505.24840},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.24840v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.24840.svg)](https://codeofpaper.com/paper/2505.24840)

Discussion