BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique

Abstract

The performance of Bayesian optimization (BO), a highly sample-efficient method for expensive black-box problems, is critically governed by the selection of its hyperparameters, including the kernel and acquisition functions. This presents a significant practical challenge: an inappropriate combination can lead to poor performance and wasted evaluations. While individual improvements to kernel functions (e.g., tree-based kernels, deep kernel learning) and acquisition functions (e.g., multi-step lookahead, tree-based planning) have been actively explored, the joint and autonomous selection of the best pair has been largely overlooked, forcing practitioners to rely on heuristics or costly manual tuning. We propose BOOST (Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique), a novel framework that automates this selection. BOOST utilizes a lightweight, offline evaluation stage to predict the performance of various kernel-acquisition pairs and identify the most promising pair before committing to expensive evaluations. Using K-means clustering, BOOST first selects initial subsets from previously observed data-in-hand and prepares all possible kernel-acquisition pairs from user-chosen candidates. For each pair, BOOST conducts internal BO runs starting with the initial subset, evaluating how many iterations are required to find the target value within the remaining data, thereby identifying the pair with the best retrospective performance for future optimization. Experiments on synthetic benchmarks and real-world hyperparameter optimization tasks demonstrate that BOOST consistently outperforms standard BO with fixed hyperparameters and state-of-the-art adaptive methods, highlighting its effectiveness and robustness in diverse problem landscapes.