A Survey on Benchmarks of Multimodal Large Language Models

Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai +9 more
8/16/2024
cs.CLcs.AIcs.CV

Official code: swordlidev/Evaluation-Multimodal-LLMs-Survey (159 stars, last commit 2025-07-01).

Abstract

Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better. For more details, please visit our GitHub repository: https://github.com/swordlidev/Evaluation-Multimodal-LLMs-Survey.

View on arXivView PDF

Code Implementations(1)

swordlidev/Evaluation-Multimodal-LLMs-SurveyOfficial100%

A Survey on Benchmarks of Multimodal Large Language Models

15914Aug 16, 20241 years ago

Similar Papers

Surveying the MLLM Landscape: A Meta-Review of Current Surveys

92% match2024

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

92% match1 repo18.0k2024

A Survey on Evaluation of Multimodal Large Language Models

91% match2024

A Survey of Multimodal Large Language Model from A Data-centric Perspective

90% match1 repo942024

Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

89% match2025

Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

89% match2024

Cite this paper

@article{li2024survey,
  title  = {A Survey on Benchmarks of Multimodal Large Language Models},
  author = {Jian Li and Weiheng Lu and Hao Fei and Meng Luo and Ming Dai and Min Xia and Yizhang Jin and Zhenye Gan and Ding Qi and Chaoyou Fu and Ying Tai and Wankou Yang and Yabiao Wang and Chengjie Wang},
  year   = {2024},
  eprint = {2408.08632},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.08632v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.08632.svg)](https://codeofpaper.com/paper/2408.08632)

Discussion