PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents

Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang +2 more
3/13/2023
cs.CVcs.CLcs.LG

No official code found. Best implementation: WeixiongLin/PMC-CLIP (242 stars, MIT, last commit 2024-08-30).

Abstract

Foundation models trained on large-scale dataset gain a recent surge in CV and NLP. In contrast, development in biomedical domain lags far behind due to data scarcity. To address this issue, we build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset, which is 8 times larger than before. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level, i.e., subfigure and subcaption. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-of-the-art results on various downstream tasks, including image-text retrieval on ROCO, MedMNIST image classification, Medical VQA, i.e. +8.1% R@10 on image-text retrieval, +3.9% accuracy on image classification.

View on arXivView PDF

Code Implementations(1)

WeixiongLin/PMC-CLIPCommunity100%
MIT

The official codes for "PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents"

24218PythonMar 13, 20232 years agoMIT

Similar Papers

Training CLIP models on Data from Scientific Papers

85% match1 repo42023

Advancing Medical Representation Learning Through High-Quality Data

84% match1 repo182025

Unified Medical Image-Text-Label Contrastive Learning With Continuous Prompt

84% match2 repos1682023

UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

84% match1 repo1702024

CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning

83% match1 repo182024

BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature

82% match2 repos1072026

Cite this paper

@article{lin2023pmcclip,
  title  = {PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents},
  author = {Weixiong Lin and Ziheng Zhao and Xiaoman Zhang and Chaoyi Wu and Ya Zhang and Yanfeng Wang and Weidi Xie},
  year   = {2023},
  eprint = {2303.07240},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.07240v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.07240.svg)](https://codeofpaper.com/paper/2303.07240)

Discussion