COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier

Gaoxiang Luo, Aryan Deshwal
10/1/2025
cs.LGcs.AI

No official code found. Best implementation: GaoxiangLuo/COM-BOM (3 stars, MIT, last commit 2025-11-04).

Abstract

Selecting an optimal set of exemplars is critical for good performance of in-context learning. However, prior exemplar search methods narrowly optimize for predictive accuracy, critically neglecting model calibration--a key determinant of trustworthiness and safe deployment. In this paper, we formulate exemplar selection as a multi-objective optimization problem, explicitly targeting both the maximization of predictive accuracy and the minimization of expected calibration error. We solve this problem with a sample-efficient Combinatorial Bayesian Optimization algorithm (COM-BOM) to find the Pareto front that optimally trades off the two objectives of accuracy and calibration. We evaluate COM-BOM on multiple tasks from unsaturated MMLU-Pro benchmark and find that COM-BOM beats or matches the baselines at jointly optimizing the two objectives, while requiring a minimal number of LLM API calls.

View on arXivView PDF

Code Implementations(1)

GaoxiangLuo/COM-BOMCommunity67%
MIT

[EMNLP 2025] COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier

30Jinja, PythonMay 21, 202511 months agoMIT
bayesian-optimizationfew-shot-promptingin-context-learningmulti-objective-optimization

Similar Papers

Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space

81% match2025

ExCalibR: Expected Calibration of Recommendations

81% match2023

Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives

80% match2023

LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization

79% match1 repo132025

Optimizing Data Augmentation through Bayesian Model Selection

79% match2025

ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models

79% match2024

Cite this paper

@article{luo2025combom,
  title  = {COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier},
  author = {Gaoxiang Luo and Aryan Deshwal},
  year   = {2025},
  eprint = {2510.01178},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.01178v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.01178.svg)](https://codeofpaper.com/paper/2510.01178)

Discussion