Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?

Jiahe Jin, Yanheng He, Mingyan Yang
2/12/2025
cs.AI

Official code: LLM-class-group/Revisiting-3D-LLM-Benchmarks (8 stars, Apache-2.0, last commit 2025-05-20).

Abstract

In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. Code and data are available at https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks

View on arXivView PDF

Code Implementations(1)

LLM-class-group/Revisiting-3D-LLM-BenchmarksOfficial100%
Apache-2.0

[ACL 2025] Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?

80Shell, PythonFeb 3, 20251 years agoApache-2.0

Similar Papers

PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment

85% match2026

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

83% match1 repo1402026

MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

83% match2024

Empir3D : A Framework for Multi-Dimensional Point Cloud Assessment

83% match2023

HueManity: Probing Fine-Grained Visual Perception in MLLMs

82% match2025

Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

82% match1 repo2872023

Cite this paper

@article{jin2025revisiting,
  title  = {Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?},
  author = {Jiahe Jin and Yanheng He and Mingyan Yang},
  year   = {2025},
  eprint = {2502.08503},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.08503v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.08503.svg)](https://codeofpaper.com/paper/2502.08503)

Discussion