Zero-Shot Robustness of Vision Language Models Via Confidence-Aware Weighting

Nikoo Naghavian, Mostafa Tavassolipour
10/3/2025
cs.CV

No confident code implementation found for this paper. 5 possible matches can be shown below.

Abstract

Vision-language models like CLIP demonstrate impressive zero-shot generalization but remain highly vulnerable to adversarial attacks. In this work, we propose Confidence-Aware Weighting (CAW) to enhance zero-shot robustness in vision-language models. CAW consists of two components: (1) a Confidence-Aware loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a feature alignment regularization that preserves semantic consistency by minimizing the distance between frozen and fine-tuned image encoder features on adversarial inputs. These components work jointly to improve both clean and robust accuracy without sacrificing generalization. Extensive experiments on TinyImageNet and 14 additional datasets show that CAW outperforms recent methods such as PMG-AFT and TGA-ZSR under strong attacks like AutoAttack, while using less memory.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 5 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

93% match1 repo32026

Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models

92% match2025

Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models

92% match2025

Pre-Trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness

92% match1 repo252026

Revisiting the Robust Generalization of Adversarial Prompt Tuning

91% match2024

Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

91% match1 repo202026

Cite this paper

@article{naghavian2025zeroshot,
  title  = {Zero-Shot Robustness of Vision Language Models Via Confidence-Aware Weighting},
  author = {Nikoo Naghavian and Mostafa Tavassolipour},
  year   = {2025},
  eprint = {2510.02913},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.02913v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.02913.svg)](https://codeofpaper.com/paper/2510.02913)

Discussion