OpenML Benchmarking Suites

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter +4 more
8/11/2017
stat.MLcs.LG

Official code: openml/OpenML (757 stars, BSD-3-Clause, last commit 2026-10-19). 1 more implementation below.

Abstract

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular.

View on arXivView PDF

Code Implementations(2)

openml/OpenMLOfficial100%
BSD-3-Clause

Open Machine Learning

757129CSS, PHPDec 11, 20123 days agoBSD-3-Clause
citizen-scientistscollaborationdatasetshacktoberfestmachine-learning+3 more
openml/benchmark-suitesOfficial100%
94R, ShellNov 10, 20175 years ago

Similar Papers

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

86% match2026

MLPerf Training Benchmark

84% match1 repo1.8k2019

Torchmeta: A Meta-Learning library for PyTorch

83% match1 repo2.1k2019

Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification

82% match1 repo472023

EvalAI: Towards Better Evaluation Systems for AI Agents

82% match1 repo2.0k2019

The Game of Hidden Rules: A New Kind of Benchmark Challenge for Machine Learning

82% match2022

Cite this paper

@article{bischl2017openml,
  title  = {OpenML Benchmarking Suites},
  author = {Bernd Bischl and Giuseppe Casalicchio and Matthias Feurer and Pieter Gijsbers and Frank Hutter and Michel Lang and Rafael G. Mantovani and Jan N. van Rijn and Joaquin Vanschoren},
  year   = {2017},
  eprint = {1708.03731},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1708.03731}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1708.03731.svg)](https://codeofpaper.com/paper/1708.03731)

Discussion