Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests

Jingzhi Gong, Giovanni Pinna, Yixin Bian, Jie M. Zhang
1/8/2026
cs.SEcs.AI

No official code found. Best implementation: gjz78910/PR-MCI (2 stars, last commit 2026-01-25).

Abstract

Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that "descriptions claim unimplemented changes" was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5 times longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.

View on arXivView PDF

Code Implementations(1)

gjz78910/PR-MCICommunity62%

Repo for the paper "Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests".

20PythonJan 25, 20268 months ago

Similar Papers

Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub

89% match1 repo2026

How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests

88% match2026

Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study

86% match2026

Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions

82% match2026

AIDev: Studying AI Coding Agents on GitHub

82% match2026

On the Adoption of AI Coding Agents in Open-source Android and iOS Development

81% match2026

Cite this paper

@article{gong2026analyzing,
  title  = {Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests},
  author = {Jingzhi Gong and Giovanni Pinna and Yixin Bian and Jie M. Zhang},
  year   = {2026},
  eprint = {2601.04886},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.04886v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.04886.svg)](https://codeofpaper.com/paper/2601.04886)

Discussion