Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity

Miquel Miró-Nicolau, Antoni Jaume-i-Capó, Gabriel Moyà-Alcover
12/14/2024
cs.CV

No official code found. Best implementation: explainingAI/stability (0 stars, last commit 2025-02-10).

Abstract

The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multiple authors have proposed evaluating this feature using objective evaluation measures. Nonetheless, many questions remain. With this work, we propose a novel approach to meta-evaluate these metrics, i.e. analyze the correctness of the evaluators. We propose two new tests that allowed us to evaluate two different stability measures: AVG-Sensitiviy and MAX-Senstivity. We tested their reliability in the presence of perfect and robust explanations, generated with a Decision Tree; as well as completely random explanations and prediction. The metrics results showed their incapacity of identify as erroneous the random explanations, highlighting their overall unreliability.

View on arXivView PDF

Code Implementations(1)

explainingAI/stabilityCommunity100%
00PythonFeb 27, 20241 years ago

Similar Papers

EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

79% match2026

A comprehensive study on fidelity metrics for XAI

79% match1 repo22024

TCD-Arena: Assessing Robustness of Time Series Causal Discovery Methods Against Assumption Violations

79% match1 repo62026

When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking

79% match2026

What Paired Evaluations Reveal under Visual Perturbations

79% match2026

A Call for Standardization and Validation of Text Style Transfer Evaluation

78% match2023

Cite this paper

@article{mirnicolau2024metaevaluating,
  title  = {Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity},
  author = {Miquel Miró-Nicolau and Antoni Jaume-i-Capó and Gabriel Moyà-Alcover},
  year   = {2024},
  eprint = {2412.10942},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.10942v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.10942.svg)](https://codeofpaper.com/paper/2412.10942)

Discussion