Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

Kejia Zhang, Keda Tao, Jiasheng Tang, Huan Wang
1/31/2025
cs.CV

No official code found. Best implementation: KejiaZhang-Robust/VAP (36 stars, MIT, last commit 2025-09-21).

Abstract

Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.

View on arXivView PDF

Code Implementations(1)

KejiaZhang-Robust/VAPCommunity65%
MIT

Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

360CSS, HTMLFeb 1, 20251 years agoMIT
hallucinationvision-language-model

Similar Papers

Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding

91% match1 repo152025

On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models

90% match1 repo112025

Reducing Hallucinations in Vision-Language Models via Latent Space Steering

90% match1 repo1202024

Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs

89% match1 repo22025

AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models

89% match1 repo222024

SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

89% match1 repo162025

Cite this paper

@article{zhang2025poison,
  title  = {Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs},
  author = {Kejia Zhang and Keda Tao and Jiasheng Tang and Huan Wang},
  year   = {2025},
  eprint = {2501.19164},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2501.19164v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.19164.svg)](https://codeofpaper.com/paper/2501.19164)

Discussion