Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?

Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado +2 more
7/22/2023
cs.CVcs.AIcs.LG

Official code: CEWu/PTNL (27 stars, MIT, last commit 2023-09-20).

Abstract

Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data. A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We find that such a prompt tuning process is highly robust to label noises. This intrigues us to study the key reasons contributing to the robustness of the prompt tuning paradigm. We conducted extensive experiments to explore this property and find the key factors are: 1) the fixed classname tokens provide a strong regularization to the optimization of the model, reducing gradients induced by the noisy samples; 2) the powerful pre-trained image-text embedding that is learned from diverse and generic web data provides strong prior knowledge for image classification. Further, we demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt, significantly enhancing prediction accuracy in the unsupervised setting. The code is available at https://github.com/CEWu/PTNL.

View on arXivView PDF

Code Implementations(1)

CEWu/PTNLOfficial100%
MIT

[ICCV 2023] Official repository of paper titled "Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?"

273Shell, PythonSep 2, 20223 years agoMIT

Similar Papers

Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

90% match1 repo502023

Vision-Language Models are Strong Noisy Label Detectors

89% match2 repos182024

Towards Calibrating Prompt Tuning of Vision-Language Models

88% match2026

Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models

88% match2023

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification

87% match1 repo192024

CLIPArTT: Adaptation of CLIP to New Domains at Test Time

87% match1 repo142024

Cite this paper

@article{wu2023prompt,
  title  = {Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?},
  author = {Cheng-En Wu and Yu Tian and Haichao Yu and Heng Wang and Pedro Morgado and Yu Hen Hu and Linjie Yang},
  year   = {2023},
  eprint = {2307.11978},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.11978v1},
  journal = {ICCV 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.11978.svg)](https://codeofpaper.com/paper/2307.11978)

Discussion