EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models

Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, Zhongyu Wei
6/9/2024
cs.AIcs.CLcs.CV

No official code found. Best implementation: mengfeidu/EmbSpatial-Bench (32 stars, last commit 2024-06-24).

Abstract

The recent rapid development of Large Vision-Language Models (LVLMs) has indicated their potential for embodied tasks.However, the critical skill of spatial understanding in embodied environments has not been thoroughly evaluated, leaving the gap between current LVLMs and qualified embodied intelligence unknown. Therefore, we construct EmbSpatial-Bench, a benchmark for evaluating embodied spatial understanding of LVLMs.The benchmark is automatically derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective.Experiments expose the insufficient capacity of current LVLMs (even GPT-4V). We further present EmbSpatial-SFT, an instruction-tuning dataset designed to improve LVLMs' embodied spatial understanding.

View on arXivView PDF

Code Implementations(1)

mengfeidu/EmbSpatial-BenchCommunity100%
323Apr 6, 20242 years ago

Similar Papers

ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models

88% match2 repos122026

EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models

87% match2026

Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions

87% match1 repo52026

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

86% match2025

BEAR: Benchmarking and Enhancing Multimodal Language Models for Atomic Embodied Capabilities

86% match2025

ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models

86% match1 repo832025

Cite this paper

@article{du2024embspatialbench,
  title  = {EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models},
  author = {Mengfei Du and Binhao Wu and Zejun Li and Xuanjing Huang and Zhongyu Wei},
  year   = {2024},
  eprint = {2406.05756},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.05756v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.05756.svg)](https://codeofpaper.com/paper/2406.05756)

Discussion