An Empirical Study on LLM-based Agents for Automated Bug Fixing

Xiangxin Meng, Zexiong Ma, Pengfei Gao, Chao Peng
11/15/2024
cs.SEcs.AI

No confident code implementation found for this paper. 5 possible matches can be shown below.

Abstract

Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWE-bench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 5 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs

90% match1 repo202025

RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

89% match1 repo1052024

Evaluating Agent-based Program Repair at Google

89% match2025

Do AI models help produce verified bug fixes?

88% match2025

Meta-RAG on Large Codebases Using Code Summarization

87% match2025

SpecRover: Code Intent Extraction via LLMs

87% match2024

Cite this paper

@article{meng2024empirical,
  title  = {An Empirical Study on LLM-based Agents for Automated Bug Fixing},
  author = {Xiangxin Meng and Zexiong Ma and Pengfei Gao and Chao Peng},
  year   = {2024},
  eprint = {2411.10213},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.10213v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.10213.svg)](https://codeofpaper.com/paper/2411.10213)

Discussion