Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou +1 more
10/27/2023
cs.CVcs.AIcs.CL

No official code found. Best implementation: williamliujl/Qilin-Med-VL (67 stars, last commit 2023-12-01).

Abstract

Large Language Models (LLMs) have introduced a new era of proficiency in comprehending complex healthcare and biomedical topics. However, there is a noticeable lack of models in languages other than English and models that can interpret multi-modal input, which is crucial for global healthcare accessibility. In response, this study introduces Qilin-Med-VL, the first Chinese large vision-language model designed to integrate the analysis of textual and visual data. Qilin-Med-VL combines a pre-trained Vision Transformer (ViT) with a foundational LLM. It undergoes a thorough two-stage curriculum training process that includes feature alignment and instruction tuning. This method enhances the model's ability to generate medical captions and answer complex medical queries. We also release ChiMed-VL, a dataset consisting of more than 1M image-text pairs. This dataset has been carefully curated to enable detailed and comprehensive interpretation of medical data using various types of images.

View on arXivView PDF

Code Implementations(1)

williamliujl/Qilin-Med-VLCommunity100%

The first Chinese medical large vision-language model designed to integrate the analysis of textual and visual data

6710Shell, PythonOct 26, 20232 years ago

Similar Papers

MedM-VL: What Makes a Good Medical LVLM?

91% match1 repo572025

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI

91% match1 repo1022024

On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

90% match2024

MoME: Mixture of Visual Language Medical Experts for Medical Imaging Segmentation

90% match2025

Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models

90% match1 repo92024

A Survey of Medical Vision-and-Language Applications and Their Techniques

90% match1 repo352024

Cite this paper

@article{liu2023qilinmedvl,
  title  = {Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare},
  author = {Junling Liu and Ziming Wang and Qichen Ye and Dading Chong and Peilin Zhou and Yining Hua},
  year   = {2023},
  eprint = {2310.17956},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.17956v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.17956.svg)](https://codeofpaper.com/paper/2310.17956)

Discussion