Holistic Evaluation of GPT-4V for Biomedical Imaging

Zhengliang Liu, Hanqi Jiang, Tianyang Zhong, Zihao Wu, Chong Ma +45 more
11/10/2023
eess.IVcs.AI

No official code found. Best implementation: Mayo-Clinic-RadOnc-Foundation-Models/Holistic-evaluation-of-gpt-4v-for-biomedical-imaging (2 stars, last commit 2024-10-26).

Abstract

In this paper, we present a large-scale evaluation probing GPT-4V's capabilities and limitations for biomedical image analysis. GPT-4V represents a breakthrough in artificial general intelligence (AGI) for computer vision, with applications in the biomedical domain. We assess GPT-4V's performance across 16 medical imaging categories, including radiology, oncology, ophthalmology, pathology, and more. Tasks include modality recognition, anatomy localization, disease diagnosis, report generation, and lesion detection. The extensive experiments provide insights into GPT-4V's strengths and weaknesses. Results show GPT-4V's proficiency in modality and anatomy recognition but difficulty with disease diagnosis and localization. GPT-4V excels at diagnostic report generation, indicating strong image captioning skills. While promising for biomedical imaging AI, GPT-4V requires further enhancement and validation before clinical deployment. We emphasize responsible development and testing for trustworthy integration of biomedical AGI. This rigorous evaluation of GPT-4V on diverse medical images advances understanding of multimodal large language models (LLMs) and guides future work toward impactful healthcare applications.

View on arXivView PDF

Code Implementations(1)

Mayo-Clinic-RadOnc-Foundation-Models/Holistic-evaluation-of-gpt-4v-for-biomedical-imagingCommunity63%

Holistic evaluation of gpt-4v for biomedical imaging

21Jan 31, 20241 years ago

Similar Papers

Towards Generalist Biomedical AI

90% match1 repo4312023

Practical Applications of Advanced Cloud Services and Generative AI Systems in Medical Image Analysis

90% match2024

BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

90% match1 repo7102023

A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis

89% match2023

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

89% match2024

Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation

89% match2025

Cite this paper

@article{liu2023holistic,
  title  = {Holistic Evaluation of GPT-4V for Biomedical Imaging},
  author = {Zhengliang Liu and Hanqi Jiang and Tianyang Zhong and Zihao Wu and Chong Ma and Yiwei Li and Xiaowei Yu and Yutong Zhang and Yi Pan and Peng Shu and Yanjun Lyu and Lu Zhang and Junjie Yao and Peixin Dong and Chao Cao and Zhenxiang Xiao and Jiaqi Wang and Huan Zhao and Shaochen Xu and Yaonai Wei and Jingyuan Chen and Haixing Dai and Peilong Wang and Hao He and Zewei Wang and Xinyu Wang and Xu Zhang and Lin Zhao and Yiheng Liu and Kai Zhang and Liheng Yan and Lichao Sun and Jun Liu and Ning Qiang and Bao Ge and Xiaoyan Cai and Shijie Zhao and Xintao Hu and Yixuan Yuan and Gang Li and Shu Zhang and Xin Zhang and Xi Jiang and Tuo Zhang and Dinggang Shen and Quanzheng Li and Wei Liu and Xiang Li and Dajiang Zhu and Tianming Liu},
  year   = {2023},
  eprint = {2312.05256},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.05256v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.05256.svg)](https://codeofpaper.com/paper/2312.05256)

Discussion