MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

Corentin Royer, Bjoern Menze, Anjany Sekuboyina
2/14/2024
cs.CV

No official code found. Best implementation: corentin-ryr/MultiMedEval (89 stars, MIT, last commit 2025-08-05).

Abstract

We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.

View on arXivView PDF

Code Implementations(1)

corentin-ryr/MultiMedEvalCommunity100%
MIT

A Python tool to evaluate the performance of VLM on the medical domain.

899PythonNov 5, 20231 years agoMIT
benchmarkevaluationllavamedical-imagingvision-language-model

Similar Papers

MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine

88% match2025

MoME: Mixture of Visual Language Medical Experts for Medical Imaging Segmentation

88% match2025

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

88% match2026

A Survey of Medical Vision-and-Language Applications and Their Techniques

88% match1 repo352024

VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

88% match1 repo4.3k2024

Beyond the Hype: A dispassionate look at vision-language models in medical scenario

87% match1 repo12024

Cite this paper

@article{royer2024multimedeval,
  title  = {MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models},
  author = {Corentin Royer and Bjoern Menze and Anjany Sekuboyina},
  year   = {2024},
  eprint = {2402.09262},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.09262v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.09262.svg)](https://codeofpaper.com/paper/2402.09262)

Discussion