A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

Nagarajan Ganapathy, Podakanti Satyajith Chary, Teja Venkata Ramana Kumar Pithani, Pavan Kavati, Arun Kumar S
10/25/2024
cs.CV

No official code found. Best implementation: Satyajithchary/MedInfoLab_Capsule_Vision_2024_Challenge (0 stars, last commit 2024-12-24).

Abstract

This Paper presents an advanced approach for fine-tuning BiomedCLIP PubMedBERT, a multimodal model, to classify abnormalities in Video Capsule Endoscopy (VCE) frames, aiming to enhance diagnostic efficiency in gastrointestinal healthcare. By integrating the PubMedBERT language model with a Vision Transformer (ViT) to process endoscopic images, our method categorizes images into ten specific classes: angioectasia, bleeding, erosion, erythema, foreign body, lymphangiectasia, polyp, ulcer, worms, and normal. Our workflow incorporates image preprocessing and fine-tunes the BiomedCLIP model to generate high-quality embeddings for both visual and textual inputs, aligning them through similarity scoring for classification. Performance metrics, including classification, accuracy, recall, and F1 score, indicate the models strong ability to accurately identify abnormalities in endoscopic frames, showing promise for practical use in clinical diagnostics.

View on arXivView PDF

Code Implementations(1)

Satyajithchary/MedInfoLab_Capsule_Vision_2024_ChallengeCommunity100%

This repository implements a multimodal approach using BiomedCLIP-PubMedBERT for classifying abnormalities in Video Capsule Endoscopy (VCE) images of Capsule Vision Challenge 2024. It contributes to automating the diagnostic process, potentially improving efficiency and accuracy in gastrointestinal diagnostics.

01Jupyter NotebookOct 25, 20241 years ago

Similar Papers

Enhanced Multi-Class Classification of Gastrointestinal Endoscopic Images with Interpretable Deep Learning Model

87% match2025

Integrating Deep Feature Extraction and Hybrid ResNet-DenseNet Model for Multi-Class Abnormality Detection in Endoscopic Images

86% match2024

Classification of Endoscopy and Video Capsule Images using CNN-Transformer Model

86% match2024

Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification

85% match2025

Capsule Vision Challenge 2024: Multi-Class Abnormality Classification for Video Capsule Endoscopy

85% match2024

Optimizing Gastrointestinal Diagnostics: A CNN-Based Model for VCE Image Classification

85% match2024

Cite this paper

@article{ganapathy2024multimodal,
  title  = {A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT},
  author = {Nagarajan Ganapathy and Podakanti Satyajith Chary and Teja Venkata Ramana Kumar Pithani and Pavan Kavati and Arun Kumar S},
  year   = {2024},
  eprint = {2410.19944},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.19944v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.19944.svg)](https://codeofpaper.com/paper/2410.19944)

Discussion