Using Large Language Models for Hyperparameter Optimization

Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, Jimmy Ba
12/7/2023
cs.LGcs.AI

No official code found. Best implementation: michaelrzhang/LLM-HyperOpt (32 stars, last commit 2024-05-13).

Abstract

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

View on arXivView PDF

Code Implementations(1)

michaelrzhang/LLM-HyperOptCommunity100%

Using Large Language Models for Hyperparameter Optimization

327Python, Jupyter NotebookFeb 15, 20242 years ago

Similar Papers

Crafting Efficient Fine-Tuning Strategies for Large Language Models

91% match2024

Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

91% match2026

Large Language Model Agent for Hyper-Parameter Optimization

90% match1 repo322024

Sequential Large Language Model-Based Hyper-parameter Optimization

89% match2 repos322024

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

88% match1 repo52023

LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

88% match2025

Cite this paper

@article{zhang2023using,
  title  = {Using Large Language Models for Hyperparameter Optimization},
  author = {Michael R. Zhang and Nishkrit Desai and Juhan Bae and Jonathan Lorraine and Jimmy Ba},
  year   = {2023},
  eprint = {2312.04528},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.04528v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.04528.svg)](https://codeofpaper.com/paper/2312.04528)

Discussion