Mean Opinion Score as a New Metric for User-Evaluation of XAI Methods

Hyeon Yu, Jenny Benois-Pineau, Romain Bourqui, Romain Giot, Alexey Zhukov
7/29/2024
cs.CVeess.IV

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

This paper investigates the use of Mean Opinion Score (MOS), a common image quality metric, as a user-centric evaluation metric for XAI post-hoc explainers. To measure the MOS, a user experiment is proposed, which has been conducted with explanation maps of intentionally distorted images. Three methods from the family of feature attribution methods - Gradient-weighted Class Activation Mapping (Grad-CAM), Multi-Layered Feature Explanation Method (MLFEM), and Feature Explanation Method (FEM) - are compared with this metric. Additionally, the correlation of this new user-centric metric with automatic metrics is studied via Spearman's rank correlation coefficient. MOS of MLFEM shows the highest correlation with automatic metrics of Insertion Area Under Curve (IAUC) and Deletion Area Under Curve (DAUC). However, the overall correlations are limited, which highlights the lack of consensus between automatic and user-centric metrics.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Five Pitfalls When Assessing Synthetic Medical Images with Reference Metrics

86% match2024

Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images

84% match2024

Do Metrics for Counterfactual Explanations Align with User Perception?

83% match2026

Image Intrinsic Scale Assessment: Bridging the Gap Between Quality and Resolution

82% match1 repo172025

On the Evaluation Consistency of Attribution-based Explanations

82% match2026

Benchmarking Image Similarity Metrics for Novel View Synthesis Applications

82% match2025

Cite this paper

@article{yu2024mean,
  title  = {Mean Opinion Score as a New Metric for User-Evaluation of XAI Methods},
  author = {Hyeon Yu and Jenny Benois-Pineau and Romain Bourqui and Romain Giot and Alexey Zhukov},
  year   = {2024},
  eprint = {2407.20427},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.20427v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.20427.svg)](https://codeofpaper.com/paper/2407.20427)

Discussion