Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations

Sihao Ding, Santosh Vasa, Aditi Ramadwar
9/27/2025
cs.CVcs.AI

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

Vision-Language Models (VLMs) often produce fluent Natural Language Explanations (NLEs) that sound convincing but may not reflect the causal factors driving predictions. This mismatch of plausibility and faithfulness poses technical and governance risks. We introduce Explanation-Driven Counterfactual Testing (EDCT), a fully automated verification procedure for a target VLM that treats the model's own explanation as a falsifiable hypothesis. Given an image-question pair, EDCT: (1) obtains the model's answer and NLE, (2) parses the NLE into testable visual concepts, (3) generates targeted counterfactual edits via generative inpainting, and (4) computes a Counterfactual Consistency Score (CCS) using LLM-assisted analysis of changes in both answers and explanations. Across 120 curated OK-VQA examples and multiple VLMs, EDCT uncovers substantial faithfulness gaps and provides regulator-aligned audit artifacts indicating when cited concepts fail causal tests.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection

85% match2025

LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop

84% match2024

Pathological Truth Bias in Vision-Language Models

84% match1 repo42025

Explainable Knowledge Graph Retrieval-Augmented Generation (KG-RAG) with KG-SMILE

84% match2025

ICE: Intervention-Consistent Explanation Evaluation with Statistical Grounding for LLMs

84% match2026

Causal Consistency Regularization: Training Verifiably Sensitive Reasoning in Large Language Models

83% match2025

Cite this paper

@article{ding2025explanationdriven,
  title  = {Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations},
  author = {Sihao Ding and Santosh Vasa and Aditi Ramadwar},
  year   = {2025},
  eprint = {2510.00047},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.00047v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.00047.svg)](https://codeofpaper.com/paper/2510.00047)

Discussion