Pathological Truth Bias in Vision-Language Models

Yash Thube
9/14/2025
cs.CV

No official code found. Best implementation: thubZ09/mats-vlm (4 stars, MIT, last commit 2025-10-09).

Abstract

Vision Language Models (VLMs) are improving quickly, but standard benchmarks can hide systematic failures that reduce real world trust. We introduce MATS (Multimodal Audit for Truthful Spatialization), a compact behavioral audit that measures whether models reject visually contradicted statements, and two metrics Spatial Consistency Score (SCS) and Incorrect Agreement Rate (IAR). Instruction tuned generative VLMs (LLaVA 1.5, QwenVLchat) exhibit very low SCS and high IAR, while contrastive encoders (CLIP, SigLIP) are far more robust. Activation patching causally localizes failure loci (mid to late cross attention for generative models, pooled projection components for contrastive models) and suggests concrete repair paths.

View on arXivView PDF

Code Implementations(1)

thubZ09/mats-vlmCommunity69%
MIT

Official implementation of the paper "Pathological Truth Bias in Vision-Language Models"

40PythonJun 12, 20251 years agoMIT

Similar Papers

Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification

84% match2025

Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations

84% match2025

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

83% match1 repo232026

DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation

83% match2 repos22026

Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset

82% match2025

TRACE: A Framework for Analyzing and Enhancing Stepwise Reasoning in Vision-Language Models

82% match2025

Cite this paper

@article{thube2025pathological,
  title  = {Pathological Truth Bias in Vision-Language Models},
  author = {Yash Thube},
  year   = {2025},
  eprint = {2509.22674},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.22674v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.22674.svg)](https://codeofpaper.com/paper/2509.22674)

Discussion