A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data

Ethan Harvey, Wansu Chen, David M. Kent, Michael C. Hughes
11/29/2023
cs.LG

Official code: tufts-ml/extrapolating-classifier-accuracy-to-larger-datasets (1 star, MIT, last commit 2024-08-28).

Abstract

Practitioners building classifiers often start with a smaller pilot dataset and plan to grow to larger data in the near future. Such projects need a toolkit for extrapolating how much classifier accuracy may improve from a 2x, 10x, or 50x increase in data size. While existing work has focused on finding a single "best-fit" curve using various functional forms like power laws, we argue that modeling and assessing the uncertainty of predictions is critical yet has seen less attention. In this paper, we propose a Gaussian process model to obtain probabilistic extrapolations of accuracy or similar performance metrics as dataset size increases. We evaluate our approach in terms of error, likelihood, and coverage across six datasets. Though we focus on medical tasks and image modalities, our open source approach generalizes to any kind of classifier.

View on arXivView PDF

Code Implementations(1)

tufts-ml/extrapolating-classifier-accuracy-to-larger-datasetsOfficial
MIT
10TeX, ShellJun 9, 20232 years agoMIT

Similar Papers

A conformalized learning of a prediction set with applications to medical imaging classification

82% match2024

What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?

82% match2024

A comprehensive review of classifier probability calibration metrics

81% match2025

Prior Distribution and Model Confidence

81% match2025

Learning Sample Difficulty from Pre-trained Models for Reliable Prediction

81% match2023

Probabilistic Scores of Classifiers, Calibration is not Enough

81% match1 repo2024

Cite this paper

@article{harvey2023probabilistic,
  title  = {A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data},
  author = {Ethan Harvey and Wansu Chen and David M. Kent and Michael C. Hughes},
  year   = {2023},
  eprint = {2311.18025},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.18025v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.18025.svg)](https://codeofpaper.com/paper/2311.18025)

Discussion