Practical Transfer Learning for Bayesian Optimization

Matthias Feurer, Benjamin Letham, Frank Hutter, Eytan Bakshy
2/6/2018
stat.MLcs.AI

Official code: automl/transfer-hpo-framework (20 stars, MIT, last commit 2022-10-05).

Abstract

When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extensions to Bayesian optimization and establish a worst-case bound compared to vanilla Bayesian optimization. Using a large collection of hyperparameter optimization benchmark problems, we demonstrate that our contributions substantially reduce optimization time compared to standard Gaussian process-based Bayesian optimization and improve over the current state-of-the-art for transfer hyperparameter optimization.

View on arXivView PDF

Code Implementations(1)

automl/transfer-hpo-frameworkOfficial100%
MIT

Code accompanying https://arxiv.org/abs/1802.02219

202Shell, PythonOct 4, 20214 years agoMIT

Similar Papers

A System for Massively Parallel Hyperparameter Tuning

88% match1 repo222018

Pre-training helps Bayesian optimization too

86% match1 repo1082022

Parameter Optimization with Conscious Allocation (POCA)

86% match2023

Bayesian Optimization Meets Self-Distillation

86% match1 repo102023

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

86% match2026

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

85% match1 repo3392026

Cite this paper

@article{feurer2018practical,
  title  = {Practical Transfer Learning for Bayesian Optimization},
  author = {Matthias Feurer and Benjamin Letham and Frank Hutter and Eytan Bakshy},
  year   = {2018},
  eprint = {1802.02219},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1802.02219}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1802.02219.svg)](https://codeofpaper.com/paper/1802.02219)

Discussion