Reproducibility Requires Consolidated Artifacts

Iordanis Fostiropoulos, Bowman Brown, Laurent Itti
5/21/2023
cs.LGcs.AIcs.SE

No confident code implementation found for this paper.

Abstract

Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

88% match2024

Reproducibility in Machine Learning-Driven Research

86% match2023

The Unreasonable Effectiveness of Open Science in AI: A Replication Study

83% match2024

Investigating Reproducibility in Deep Learning-Based Software Fault Prediction

82% match2024

Position: Why We Must Rethink Empirical Research in Machine Learning

80% match2024

A K-Means, Ward and DBSCAN repeatability study

80% match2025

Cite this paper

@article{fostiropoulos2023reproducibility,
  title  = {Reproducibility Requires Consolidated Artifacts},
  author = {Iordanis Fostiropoulos and Bowman Brown and Laurent Itti},
  year   = {2023},
  eprint = {2305.12571},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.12571v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.12571.svg)](https://codeofpaper.com/paper/2305.12571)

Discussion