ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

Yeji Park, Deokyeong Lee, Junsuk Choe, Buru Chang
8/25/2024
cs.CVcs.AIcs.LG

No official code found. Best implementation: yejipark-m/ConVis (25 stars, MIT, last commit 2024-09-26).

Abstract

Hallucinations in Multimodal Large Language Models (MLLMs) where generated responses fail to accurately reflect the given image pose a significant challenge to their reliability. To address this, we introduce ConVis, a novel training-free contrastive decoding method. ConVis leverages a text-to-image (T2I) generation model to semantically reconstruct the given image from hallucinated captions. By comparing the contrasting probability distributions produced by the original and reconstructed images, ConVis enables MLLMs to capture visual contrastive signals that penalize hallucination generation. Notably, this method operates purely within the decoding process, eliminating the need for additional data or model updates. Our extensive experiments on five popular benchmarks demonstrate that ConVis effectively reduces hallucinations across various MLLMs, highlighting its potential to enhance model reliability.

View on arXivView PDF

Code Implementations(1)

yejipark-m/ConVisCommunity100%
MIT

[AAAI 2025] ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

251C, C++Aug 25, 20242 years agoMIT

Similar Papers

Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -

92% match1 repo92025

Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models

92% match1 repo92024

Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

92% match2024

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

91% match2 repos4202024

Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding

91% match2025

CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models

90% match1 repo42026

Cite this paper

@article{park2024convis,
  title  = {ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models},
  author = {Yeji Park and Deokyeong Lee and Junsuk Choe and Buru Chang},
  year   = {2024},
  eprint = {2408.13906},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.13906v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.13906.svg)](https://codeofpaper.com/paper/2408.13906)

Discussion