How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao +4 more
11/27/2023
cs.CVcs.CLcs.LG

Official code: UCSC-VLAA/vllm-safety-benchmark (90 stars, last commit 2023-11-28).

Abstract

This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness. For the OOD evaluation, we present two novel VQA datasets, each with one variant, designed to test model performance under challenging conditions. In exploring adversarial robustness, we propose a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses. Moreover, we assess the efficacy of two jailbreaking strategies, targeting either the vision or language component of VLLMs. Our evaluation of 21 diverse models, ranging from open-source VLLMs to GPT-4V, yields interesting observations: 1) Current VLLMs struggle with OOD texts but not images, unless the visual information is limited; and 2) These VLLMs can be easily misled by deceiving vision encoders only, and their vision-language training often compromise safety protocols. We release this safety evaluation suite at https://github.com/UCSC-VLAA/vllm-safety-benchmark.

View on arXivView PDF

Code Implementations(1)

UCSC-VLAA/vllm-safety-benchmarkOfficial100%

[ECCV 2024] Official PyTorch Implementation of "How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs"

905PythonNov 23, 20232 years ago
adversarial-attacksbenchmarkdatasetsllmmultimodal-llm+3 more

Similar Papers

REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLM

87% match1 repo22025

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

86% match1 repo912024

Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

85% match1 repo32024

Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models

85% match1 repo142025

The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?

85% match1 repo432026

On Evaluating Adversarial Robustness of Large Vision-Language Models

85% match1 repo2312023

Cite this paper

@article{tu2023many,
  title  = {How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs},
  author = {Haoqin Tu and Chenhang Cui and Zijun Wang and Yiyang Zhou and Bingchen Zhao and Junlin Han and Wangchunshu Zhou and Huaxiu Yao and Cihang Xie},
  year   = {2023},
  eprint = {2311.16101},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.16101v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.16101.svg)](https://codeofpaper.com/paper/2311.16101)

Discussion