Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

Poojitha Thota, Jai Prakash Veerla, Partha Sai Guttikonda, Mohammad S. Nasr, Shirin Nilizadeh +1 more
1/4/2024
eess.IVcs.CVq-bio.TO

Official code: jaiprakash1824/VLM_Adv_Attack (5 stars, MIT, PyTorch, last commit 2024-05-18).

Abstract

In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predictions, underscoring its susceptibility to adversarial perturbations. The qualitative analysis of adversarial examples delves into the interpretability challenges, shedding light on nuanced changes in predictions induced by adversarial manipulations. These findings contribute crucial insights into the interpretability, domain adaptation, and trustworthiness of Vision Language Models in medical imaging. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models. The source codes for this experiment can be found at https://github.com/jaiprakash1824/VLM_Adv_Attack.

View on arXivView PDF

Code Implementations(1)

jaiprakash1824/VLM_Adv_AttackOfficial100%
PyTorchMIT

In the dynamic landscape of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks like PGD adversarial attack.

51Jupyter NotebookMar 1, 20242 years agoMIT
adversarial-attacksattention-mechanismattention-visualizationclipcontrastive-learning+13 more

Similar Papers

CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Models

88% match2025

Securing the Diagnosis of Medical Imaging: An In-depth Analysis of AI-Resistant Attacks

88% match2024

Vision Transformers: the threat of realistic adversarial patches

87% match2025

DFT-Based Adversarial Attack Detection in MRI Brain Imaging: Enhancing Diagnostic Accuracy in Alzheimer's Case Studies

85% match2024

Hidden in Plain Sight: Undetectable Adversarial Bias Attacks on Vulnerable Patient Populations

85% match1 repo2024

Hierarchical Self-Supervised Adversarial Training for Robust Vision Models in Histopathology

85% match1 repo122025

Cite this paper

@article{thota2024demonstration,
  title  = {Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging},
  author = {Poojitha Thota and Jai Prakash Veerla and Partha Sai Guttikonda and Mohammad S. Nasr and Shirin Nilizadeh and Jacob M. Luber},
  year   = {2024},
  eprint = {2401.02565},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.02565v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.02565.svg)](https://codeofpaper.com/paper/2401.02565)

Discussion