Evaluating the Reliability of Self-Explanations in Large Language Models

Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony Lindgren
7/19/2024
cs.CL

No official code found. Best implementation: k-randl/self-explaining_llms (1 star, GPL-3.0, last commit 2025-07-15).

Abstract

This paper investigates the reliability of explanations generated by large language models (LLMs) when prompted to explain their previous output. We evaluate two kinds of such self-explanations - extractive and counterfactual - using three state-of-the-art LLMs (2B to 8B parameters) on two different classification tasks (objective and subjective). Our findings reveal, that, while these self-explanations can correlate with human judgement, they do not fully and accurately follow the model's decision process, indicating a gap between perceived and actual model reasoning. We show that this gap can be bridged because prompting LLMs for counterfactual explanations can produce faithful, informative, and easy-to-verify results. These counterfactuals offer a promising alternative to traditional explainability methods (e.g. SHAP, LIME), provided that prompts are tailored to specific tasks and checked for validity.

View on arXivView PDF

Code Implementations(1)

k-randl/self-explaining_llmsCommunity100%
GPL-3.0

Official implementation of the papers "Evaluating the Reliability of Self-Explanations in Large Language Models" and "Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models"

10Shell, PythonJun 17, 20241 years agoGPL-3.0
counterfactualslarge-language-modelsself-explanations

Similar Papers

Are self-explanations from Large Language Models faithful?

91% match1 repo142024

Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations

91% match2026

Can LLMs Explain Themselves Counterfactually?

91% match1 repo2025

Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models

90% match2024

Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMs

89% match2024

LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations

89% match1 repo32025

Cite this paper

@article{randl2024evaluating,
  title  = {Evaluating the Reliability of Self-Explanations in Large Language Models},
  author = {Korbinian Randl and John Pavlopoulos and Aron Henriksson and Tony Lindgren},
  year   = {2024},
  eprint = {2407.14487},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.14487v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.14487.svg)](https://codeofpaper.com/paper/2407.14487)

Discussion