RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian

Kla Tantithamthavorn, Yaotian Zou, Andy Wong, Michael Gupta, Zhe Wang +6 more
1/3/2026
cs.SEcs.AIcs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Large Language Models (LLMs)-powered code review automation has the potential to transform code review workflows. Despite the advances of LLM-powered code review comment generation approaches, several practical challenges remain for designing enterprise-grade code review automation tools. In particular, this paper aims at answering the practical question: how can we design a review-guided, context-aware, quality-checked code review comment generation without fine-tuning? In this paper, we present RovoDev Code Reviewer, an enterprise-grade LLM-based code review automation tool designed and deployed at scale within Atlassian's development ecosystem with seamless integration into Atlassian's Bitbucket. Through the offline, online, user feedback evaluations over a one-year period, we conclude that RovoDev Code Reviewer is effective in generating code review comments that could lead to code resolution for 38.70% (i.e., comments that triggered code changes in the subsequent commits); and offers the promise of accelerating feedback cycles (i.e., decreasing the PR cycle time by 30.8%), alleviating reviewer workload (i.e., reducing the number of human-written comments by 35.6%), and improving overall software quality (i.e., finding errors with actionable suggestions).

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{tantithamthavorn2026rovodev,
  title  = {RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian},
  author = {Kla Tantithamthavorn and Yaotian Zou and Andy Wong and Michael Gupta and Zhe Wang and Mike Buller and Ryan Jiang and Matthew Watson and Minwoo Jeong and Kun Chen and Ming Wu},
  year   = {2026},
  eprint = {2601.01129},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.01129v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.01129.svg)](https://codeofpaper.com/paper/2601.01129)

Discussion