SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

Xixu Hu, Runkai Zheng, Jindong Wang, Cheuk Hang Leung, Qi Wu +1 more
1/2/2024
cs.CVcs.LG

Official code: microsoft/robustlearn (507 stars, MIT, last commit 2024-07-12).

Abstract

Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs' attention layers, we enhance the model's robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn.

View on arXivView PDF

Code Implementations(1)

microsoft/robustlearnOfficial100%
MIT

Robust machine learning for responsible AI

50762Shell, PythonOct 20, 20222 years agoMIT

Similar Papers

Improving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentation

85% match2023

ADAPT to Robustify Prompt Tuning Vision Transformers

85% match1 repo2024

S-E Pipeline: A Vision Transformer (ViT) based Resilient Classification Pipeline for Medical Imaging Against Adversarial Attacks

84% match2024

Protego: Detecting Adversarial Examples for Vision Transformers via Intrinsic Capabilities

84% match2025

LipShiFT: A Certifiably Robust Shift-based Vision Transformer

83% match2025

Exploring Adversarial Watermarking in Transformer-Based Models: Transferability and Robustness Against Defense Mechanism for Medical Images

82% match2025

Cite this paper

@article{hu2024specformer,
  title  = {SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization},
  author = {Xixu Hu and Runkai Zheng and Jindong Wang and Cheuk Hang Leung and Qi Wu and Xing Xie},
  year   = {2024},
  eprint = {2402.03317},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.03317v2},
  journal = {ECCV 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.03317.svg)](https://codeofpaper.com/paper/2402.03317)

Discussion