Exploring the Design Space of 3D MLLMs for CT Report Generation

Mohammed Baharoon, Jun Ma, Congyu Fang, Augustin Toma, Bo Wang
6/26/2025
eess.IVcs.CVcs.LG

Official code: bowang-lab/AMOS-MM-Solution (16 stars, MIT, last commit 2025-09-13).

Abstract

Multimodal Large Language Models (MLLMs) have emerged as a promising way to automate Radiology Report Generation (RRG). In this work, we systematically investigate the design space of 3D MLLMs, including visual input representation, projectors, Large Language Models (LLMs), and fine-tuning techniques for 3D CT report generation. We also introduce two knowledge-based report augmentation methods that improve performance on the GREEN score by up to 10%, achieving the 2nd place on the MICCAI 2024 AMOS-MM challenge. Our results on the 1,687 cases from the AMOS-MM dataset show that RRG is largely independent of the size of LLM under the same training protocol. We also show that larger volume size does not always improve performance if the original ViT was pre-trained on a smaller volume size. Lastly, we show that using a segmentation mask along with the CT volume improves performance. The code is publicly available at https://github.com/bowang-lab/AMOS-MM-Solution

View on arXivView PDF

Code Implementations(1)

bowang-lab/AMOS-MM-SolutionOfficial100%
MIT

Solution to the AMOS-MM challenge

160Shell, PythonSep 3, 20241 years agoMIT

Similar Papers

Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation

88% match1 repo512024

Resource-Efficient Medical Report Generation using Large Language Models

85% match2024

MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

85% match2024

U-VLM: Hierarchical Vision Language Modeling for Report Generation

84% match1 repo202026

Leveraging LLMs for Multimodal Retrieval-Augmented Radiology Report Generation via Key Phrase Extraction

84% match2025

SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models

84% match2024

Cite this paper

@article{baharoon2025exploring,
  title  = {Exploring the Design Space of 3D MLLMs for CT Report Generation},
  author = {Mohammed Baharoon and Jun Ma and Congyu Fang and Augustin Toma and Bo Wang},
  year   = {2025},
  eprint = {2506.21535},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.21535v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.21535.svg)](https://codeofpaper.com/paper/2506.21535)

Discussion