RepairBench: Leaderboard of Frontier Models for Program Repair

André Silva, Martin Monperrus
9/27/2024
cs.SEcs.LG

No official code found. Best implementation: ASSERT-KTH/repairbench-framework (22 stars, last commit 2026-04-27). 1 more below.

Abstract

AI-driven program repair uses AI models to repair buggy software by producing patches. Rapid advancements in AI surely impact state-of-the-art performance of program repair. Yet, grasping this progress requires frequent and standardized evaluations. We propose RepairBench, a novel leaderboard for AI-driven program repair. The key characteristics of RepairBench are: 1) it is execution-based: all patches are compiled and executed against a test suite, 2) it assesses frontier models in a frequent and standardized way. RepairBench leverages two high-quality benchmarks, Defects4J and GitBug-Java, to evaluate frontier models against real-world program repair tasks. We publicly release the evaluation framework of RepairBench. We will update the leaderboard as new frontier models are released.

View on arXivView PDF

Code Implementations(2)

ASSERT-KTH/repairbench-frameworkCommunity100%

Automatic Repair Framework with LLMs ❤️ https://arxiv.org/pdf/2409.18952

2210Shell, PythonNov 22, 20225 months ago
ASSERT-KTH/repairbenchCommunity100%

Leaderboard of Frontier Models for Program Repair https://repairbench.github.io/

112CSS, HTMLAug 29, 202411 months ago

Similar Papers

RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

83% match1 repo1052024

RunBugRun -- An Executable Dataset for Automated Program Repair

83% match2023

RePair: Automated Program Repair with Process-based Feedback

83% match1 repo72024

Repair-R1: Better Test Before Repair

82% match1 repo102025

The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models

82% match2025

Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs

81% match1 repo62024

Cite this paper

@article{silva2024repairbench,
  title  = {RepairBench: Leaderboard of Frontier Models for Program Repair},
  author = {André Silva and Martin Monperrus},
  year   = {2024},
  eprint = {2409.18952},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.18952v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.18952.svg)](https://codeofpaper.com/paper/2409.18952)

Discussion