Training CLIP models on Data from Scientific Papers

Calvin Metzger
11/8/2023
cs.CVcs.LG

Official code: nopperl/clip_arxiv_pmc (4 stars, Apache-2.0, last commit 2023-11-09).

Abstract

Contrastive Language-Image Pretraining (CLIP) models are able to capture the semantic relationship of images and texts and have enabled a wide range of applications, from image retrieval to classification. These models are trained with datasets extracted from web crawls, which are of large quantity but limited quality. This paper explores whether limited amounts higher quality data in a specific domain improve the general performance of CLIP models. To this purpose, we extract text-image data from scientific papers hosted in the arXiv and PubMed Central repositories. Experiments on small-scale CLIP models (ViT B/32) show that model performance increases on average, but only moderately. This result indicates that using the data sources considered in the paper to train large-scale CLIP models is a worthwile research direction.

View on arXivView PDF

Code Implementations(1)

nopperl/clip_arxiv_pmcOfficial100%
Apache-2.0

Training CLIP models on Data from Scientific Papers

41TeX, ShellSep 11, 20232 years agoApache-2.0
clipdatadatasetimg2datasetmultimodal

Similar Papers

Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantity

87% match1 repo102024

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

86% match2024

PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents

85% match1 repo2422023

What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights

84% match1 repo272024

CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement

84% match2023

Improving CLIP Training with Language Rewrites

84% match1 repo2912023

Cite this paper

@article{metzger2023training,
  title  = {Training CLIP models on Data from Scientific Papers},
  author = {Calvin Metzger},
  year   = {2023},
  eprint = {2311.04711},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.04711v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.04711.svg)](https://codeofpaper.com/paper/2311.04711)

Discussion