Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark

Evan M. Williams, Kathleen M. Carley
5/10/2024
cs.CVcs.AIcs.CL

No official code found. Best implementation: EvanUp/VNA_Benchmark (1 star, MIT, last commit 2024-05-31).

Abstract

We evaluate the zero-shot ability of GPT-4 and LLaVa to perform simple Visual Network Analysis (VNA) tasks on small-scale graphs. We evaluate the Vision Language Models (VLMs) on 5 tasks related to three foundational network science concepts: identifying nodes of maximal degree on a rendered graph, identifying whether signed triads are balanced or unbalanced, and counting components. The tasks are structured to be easy for a human who understands the underlying graph theoretic concepts, and can all be solved by counting the appropriate elements in graphs. We find that while GPT-4 consistently outperforms LLaVa, both models struggle with every visual network analysis task we propose. We publicly release the first benchmark for the evaluation of VLMs on foundational VNA tasks.

View on arXivView PDF

Code Implementations(1)

EvanUp/VNA_BenchmarkCommunity100%
MIT

Code and data for the paper "Multimodal LLMs Struggle with Basic Visual Network Analysis: a Visual Network Analysis Benchmark"

10May 8, 20242 years agoMIT

Similar Papers

Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

82% match1 repo132024

VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models

81% match1 repo12024

VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context

81% match1 repo172024

The Underappreciated Power of Vision Models for Graph Structural Understanding

80% match1 repo22025

Suitability of KANs for Computer Vision: A preliminary investigation

79% match2024

Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization

79% match2025

Cite this paper

@article{williams2024multimodal,
  title  = {Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark},
  author = {Evan M. Williams and Kathleen M. Carley},
  year   = {2024},
  eprint = {2405.06634},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.06634v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.06634.svg)](https://codeofpaper.com/paper/2405.06634)

Discussion