Multicalibration for LLM-based Code Generation

Viola Campos, Robin Kuschnereit, Adrian Ulges
12/9/2025
cs.SEcs.AIcs.LG

No official code found. Best implementation: violacampos/multicalibration (1 star, MIT, last commit 2026-01-20).

Abstract

As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.

View on arXivView PDF

Code Implementations(1)

violacampos/multicalibrationCommunity63%
MIT

Code for the paper 'Multicalibration for LLM-based Code Generation'

10Python, Jupyter NotebookOct 24, 20259 months agoMIT

Similar Papers

Showing LLM-Generated Code Selectively Based on Confidence of LLMs

86% match2024

Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models

83% match1 repo142024

Calibration and Correctness of Language Models for Code

83% match2024

A Survey of Calibration Process for Black-Box LLMs

83% match1 repo212024

Rethinking Verification for LLM Code Generation: From Generation to Testing

83% match1 repo132025

CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement

83% match2024

Cite this paper

@article{campos2025multicalibration,
  title  = {Multicalibration for LLM-based Code Generation},
  author = {Viola Campos and Robin Kuschnereit and Adrian Ulges},
  year   = {2025},
  eprint = {2512.08810},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2512.08810v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2512.08810.svg)](https://codeofpaper.com/paper/2512.08810)

Discussion