PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski +3 more
6/21/2023
cs.LG

Official code: automl/mf-prior-exp (3 stars, MIT, last commit 2023-10-24).

Abstract

Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs

View on arXivView PDF

Code Implementations(1)

automl/mf-prior-expOfficial100%
MIT
32Just, ShellAug 18, 20222 years agoMIT

Similar Papers

Multi-objective Hyperparameter Optimization in the Age of Deep Learning

88% match2025

Iterative Deepening Hyperband

87% match2023

Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization

85% match2025

Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization

84% match2025

Scaling Laws for Hyperparameter Optimization

84% match2023

Hyperparameters in Reinforcement Learning and How To Tune Them

84% match1 repo872026

Cite this paper

@article{mallik2023priorband,
  title  = {PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning},
  author = {Neeratyoy Mallik and Edward Bergman and Carl Hvarfner and Danny Stoll and Maciej Janowski and Marius Lindauer and Luigi Nardi and Frank Hutter},
  year   = {2023},
  eprint = {2306.12370},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.12370v2},
  journal = {NEURIPS 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.12370.svg)](https://codeofpaper.com/paper/2306.12370)

Discussion