Explanations can be manipulated and geometry is to blame

Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller +1 more
6/19/2019
stat.MLcs.CRcs.LG

Official code: pankessel/adv_explanation_ref (39 stars, Apache-2.0, last commit 2022-07-24).

Abstract

Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

View on arXivView PDF

Code Implementations(1)

pankessel/adv_explanation_refOfficial100%
Apache-2.0

reference implementation for "explanations can be manipulated and geometry is to blame"

3913Shell, PythonOct 25, 20194 years agoApache-2.0

Similar Papers

Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance

86% match1 repo102023

A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i

85% match2025

When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances

84% match2024

Robustness of Visual Explanations to Common Data Augmentation

84% match1 repo42023

TextCAM: Explaining Class Activation Map with Text

83% match2025

Quantifying the Intrinsic Usefulness of Attributional Explanations for Graph Neural Networks with Artificial Simulatability Studies

83% match2023

Cite this paper

@article{dombrowski2019explanations,
  title  = {Explanations can be manipulated and geometry is to blame},
  author = {Ann-Kathrin Dombrowski and Maximilian Alber and Christopher J. Anders and Marcel Ackermann and Klaus-Robert Müller and Pan Kessel},
  year   = {2019},
  eprint = {1906.07983},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1906.07983}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1906.07983.svg)](https://codeofpaper.com/paper/1906.07983)

Discussion