An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set

Ali Ismail-Fawaz, Angus Dempster, Chang Wei Tan, Matthieu Herrmann, Lynn Miller +6 more
5/19/2023
stat.MEcs.AIcs.LG

No official code found. Best implementation: MSD-IRIMAS/Multi_Comparison_Matrix (23 stars, GPL-3.0, last commit 2025-05-22).

Abstract

The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demšar (2006), have important shortcomings and, we show, are open to both inadvertent and intentional manipulation. To address these issues, we propose a new approach to presenting the results of benchmark comparisons, the Multiple Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes the means of manipulating experimental results in existing approaches. MCM can be used to show the results of an all-pairs comparison, or to show the results of a comparison between one or more selected algorithms and the state of the art. MCM is implemented in Python and is publicly available.

View on arXivView PDF

Code Implementations(1)

MSD-IRIMAS/Multi_Comparison_MatrixCommunity100%
GPL-3.0

Multi Comparison Matrix: A long term approach to benchmark evaluations

232TeX, PythonMar 6, 20231 years agoGPL-3.0
benchmarkingcritical-difference-diagramevaluationtimeseriesclassification

Similar Papers

A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets

81% match1 repo52022

Comparison of classifiers in challenge scheme

78% match2023

Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

78% match2024

Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks

78% match1 repo242024

WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging

77% match1 repo82025

Interpretable Meta-Measure for Model Performance

77% match1 repo32020

Cite this paper

@article{ismailfawaz2023approach,
  title  = {An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set},
  author = {Ali Ismail-Fawaz and Angus Dempster and Chang Wei Tan and Matthieu Herrmann and Lynn Miller and Daniel F. Schmidt and Stefano Berretti and Jonathan Weber and Maxime Devanne and Germain Forestier and Geoffrey I. Webb},
  year   = {2023},
  eprint = {2305.11921},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.11921v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.11921.svg)](https://codeofpaper.com/paper/2305.11921)

Discussion