MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems

Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, Meng Jiang
10/18/2024
cs.CLcs.CV

Official code: Zivenzhu/Multi-chart-QA (13 stars, last commit 2025-06-02).

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive abilities across various tasks, including visual question answering and chart comprehension, yet existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. Current benchmarks primarily focus on single-chart tasks, neglecting the multi-hop reasoning required to extract and integrate information from multiple charts, which is essential in practical applications. To fill this gap, we introduce MultiChartQA, a benchmark that evaluates MLLMs' capabilities in four key areas: direct question answering, parallel question answering, comparative reasoning, and sequential reasoning. Our evaluation of a wide range of MLLMs reveals significant performance gaps compared to humans. These results highlight the challenges in multi-chart comprehension and the potential of MultiChartQA to drive advancements in this field. Our code and data are available at https://github.com/Zivenzhu/Multi-chart-QA

View on arXivView PDF

Code Implementations(1)

Zivenzhu/Multi-chart-QAOfficial100%

This is the repository for Multi-chart Benchmark.

130PythonSep 30, 20241 years agoNOASSERTION

Similar Papers

POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering

92% match2025

ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering

88% match1 repo472025

In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding

87% match1 repo32025

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

87% match1 repo222024

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models

87% match1 repo242025

Distill Visual Chart Reasoning Ability from LLMs to MLLMs

86% match1 repo612024

Cite this paper

@article{zhu2024multichartqa,
  title  = {MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems},
  author = {Zifeng Zhu and Mengzhao Jia and Zhihan Zhang and Lang Li and Meng Jiang},
  year   = {2024},
  eprint = {2410.14179},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.14179v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.14179.svg)](https://codeofpaper.com/paper/2410.14179)

Discussion