1 Introduction↩︎

Automated code review using large language models (LLMs) has become a core component of modern software development toolchains. Systems such as CodeRabbit, GitHub Copilot code review, and VibeOps [1] use LLMs to detect bugs, security vulnerabilities, and architectural issues in pull request (PR) diffs. A critical engineering decision in these systems is model selection: which LLM produces the highest-quality reviews at acceptable cost and latency?

Conventional wisdom holds that larger, more expensive models should outperform smaller ones on complex reasoning tasks. Code review, which requires understanding control flow, data dependencies, and domain conventions, would seem to favor the most capable models available.

This paper tests that assumption empirically. It does not hold. But the data also reveals a deeper problem with evaluation methodology: synthetic benchmarks, the standard tool for measuring code review quality, are misleading. When real pull requests are introduced, model performance drops by an order of magnitude.

Our contributions are:

1. A reusable two-pass evaluation framework for code review quality combining deterministic matching with LLM adjudication.

2. A 150-sample benchmark spanning synthetic mutations and real pull requests across TypeScript, Python, and Go.

3. Evidence that Claude Haiku 4.5 consistently outperforms Claude Sonnet 4.6 on code review, contradicting the larger-is-better assumption, across three independent conditions and independently confirmed on the Martian Code Review Benchmark [2].

4. Quantification of the synthetic-to-real gap: a 92% F1 degradation when models are evaluated on real PRs alone, with diff size as the dominant confound.

5. Analysis of model ensemble strategies, per-language performance, and output behavior differences that explain the Haiku–Sonnet gap.

2 Related Work↩︎

2.0.0.1 LLM-based code review.

[3] introduced CodeReviewer, a pre-trained model for code review automation evaluated on comment generation and code refinement. [4] demonstrated that pre-trained models can boost code review automation by generating review comments. [5] explored parameter-efficient fine-tuning for code review with LLaMA-Reviewer. More recently, [6] presented early results on using commercial LLMs for code review, and [7] benchmarked LLM-based code review across multiple dimensions. [8] introduced CR-Bench for evaluating real-world code review agents. Our work differs in directly comparing models across providers and price tiers on both synthetic and real-world data, and in isolating the synthetic-to-real generalization gap.

2.0.0.2 Code review benchmarks.

[9] proposed CRQBench for code reasoning evaluation. Bug benchmarks such as Defects4J [10] and BugsInPy [11] provide ground truth for bug detection but were designed for testing, not code review. The Martian Code Review Benchmark [2] evaluates code review tools on 50 real PRs with human-curated golden comments; we use it for external validation (Section 4.8). Our benchmark complements these by combining synthetic and real data with structured annotations for fine-grained analysis.

2.0.0.3 LLM-as-judge evaluation.

[12] established the LLM-as-judge paradigm with MT-Bench, showing that strong LLMs can approximate human judgment. We adapt this approach with a cost-reducing two-pass system: deterministic matching handles clear-cut cases (\({\sim}\)​70%), with LLM adjudication reserved for ambiguous findings.

2.0.0.4 Smaller models outperforming larger ones.

[13] demonstrated that smaller models can outperform larger ones through distillation. Our finding is distinct: Haiku 4.5 outperforms Sonnet 4.6 without task-specific fine-tuning, suggesting that code review as a task may not benefit from additional model scale beyond a capability threshold.

3 Method↩︎

This section defines the evaluation task, describes the two-pass judge framework, and details the experimental setup.

3.1 Problem Definition↩︎

We formalize automated code review as a finding detection task. Given a pull request diff \(D\) and optional context \(C\) (repository conventions, surrounding files), a model \(M\) produces a set of findings \(F_M = \{f_1, \ldots, f_k\}\). Each finding \(f_i\) specifies: a file path, line range, comment type (security, bug, architecture, performance, best practice), severity (critical, high, medium, low), description, and optional code suggestion.

The ground truth for each sample is a set of annotations \(G = \{g_1, \ldots, g_m\}\) with the same structure. We evaluate \(F_M\) against \(G\) using precision (fraction of model findings that match real issues), recall (fraction of real issues found by the model), and F1 (harmonic mean). We additionally compute severity-weighted F1, where critical findings count \(4\times\), high \(2\times\), medium \(1\times\), and low \(0.5\times\).

3.2 Two-Pass Evaluation Framework↩︎

The central challenge in code review evaluation is matching: determining whether a model’s finding corresponds to a ground truth annotation. Unlike code generation, where test-pass/fail provides a binary signal, code review findings are free-form text with approximate line references, making exact matching infeasible.

3.2.0.1 Pass 1: Deterministic matching.

Each model finding is matched to ground truth annotations using three conjunctive criteria: (a) normalized file path match, (b) line range overlap with \(\pm\)​5-line tolerance, (c) comment type compatibility (e.g., “bug” matches “logic”). Clear matches become true positives (TP); findings with no file match become false positives (FP); unmatched annotations become false negatives (FN). Ambiguous cases (same file but line or type mismatch) are deferred to Pass 2.

3.2.0.2 Pass 2: LLM judge.

Claude Opus 4.6 (temperature 0.0) adjudicates deferred cases. Following [12], we use a structured rubric to reduce judge variance. The judge classifies uncertain findings as true_positive, false_positive, or partial_match (scored as 0.5 TP). It also scores four qualitative dimensions on a 1–5 scale:

• Depth: surface pattern matching (1) vs.semantic runtime reasoning

  (5)

• Context awareness: ignores codebase (1) vs.leverages conventions

  (5)

• Specificity: vague (1) vs.precise with concrete failing inputs (5)

• Suggestion correctness: wrong/breaks code (1) vs.compilable fix

  (5)

This two-pass design reduces cost: the deterministic pass handles \({\sim}\)​70% of cases at zero LLM cost, with Opus invoked only for genuinely ambiguous matches.

3.3 Models Under Test↩︎

We evaluate five models spanning three providers and a 15\(\times\) cost range (Table 1).

All models received the identical production prompt (the VibeOps review system prompt) instructing structured JSON output with comment type, severity, file path, line numbers, description, concrete failing input, and citation. Temperature was set to 0.1 for all models. No provider-specific prompt tuning was applied.

3.4 Dataset↩︎

Our benchmark comprises 150 samples from two sources.

3.4.0.1 Synthetic mutations (\(n{=}100\)).

We implemented 13 mutation operators that inject known bugs into 8 realistic code templates across TypeScript, Python, and Go. Operators include: off-by-one errors, SQL injection, XSS (sanitization removal), missing null checks, hardcoded secrets, missing await, resource leaks, wrong comparison operators, N+1 queries, unbounded queries, missing rate limiting, and missing input validation. Each mutation produces a diff and automatically-generated ground truth annotations. Synthetic diffs are small (median 5 lines, max 40). All mutations use deterministic seed 42 for reproducibility.

3.4.0.2 Real bug-fix PRs (\(n{=}50\)).

We mined merged pull requests with bug-related labels from eight major repositories using the GitHub API (Table 2). Selection criteria: merged, 1–10 files changed, 20–500 lines of diff. Ground truth annotations were extracted from the diff structure; removed or changed hunks in the fix represent bug locations. Real PR diffs are substantially larger (median 117 lines, max 562) and contain multi-file changes, formatting noise, and interleaved refactoring.

3.4.0.3 Distribution.

Logic 81 (54%), Security 34 (23%), Performance 16 (11%), Architecture 14 (9%), Best Practice 5 (3%). Languages: TypeScript 70 (47%), Python 43 (29%), Go 37 (25%).

3.5 Experimental Conditions↩︎

We conducted three experiments with increasing dataset size and difficulty:

4 Results↩︎

4.1 Overall Performance↩︎

Table 4 shows F1 scores across all three conditions. Haiku 4.5 ranks #1 in two of three conditions (and #2 in the third, behind Minimax by 0.02) and has the lowest mean rank (1.3). Crucially, Haiku outperforms Sonnet in every condition.

Figure 1 visualizes the F1 degradation as dataset difficulty increases. All models decline sharply from Experiment 2 to 3, but relative rankings remain stable; Haiku leads in every condition.

4.2 Haiku vs.Sonnet: Head-to-Head↩︎

Table 5 presents the direct comparison. Haiku outperforms Sonnet on 8 of 10 metrics. Sonnet wins on precision and severity-weighted F1, reflecting a more conservative review style; it produces fewer total findings (148 vs.Haiku’s 199) but with a higher proportion correct. However, this conservatism costs 17 additional missed bugs.

4.3 Category Analysis↩︎

Table 6 shows recall by bug category.

Security detection is commoditized (\({\sim}\)​70% recall across all models). Logic bugs are where models diverge: Haiku reaches 24.5% recall versus Sonnet’s 19.6%. Performance bugs (N+1 queries, unbounded queries) are undetectable at 0% recall for 4 of 5 models.

4.4 The Synthetic-to-Real Gap↩︎

The magnitude of degradation on real PRs is severe. Table 7 shows F1 computed separately on synthetic and real subsets.

On real PRs alone, the best model (Haiku) achieves F1 \(=\) 0.066, barely above random. Minimax and GLM are effectively at zero (Figure 2). This is not a gradual degradation; it is a near-total collapse driven by 20\(\times\) larger diffs, formatting noise, and multi-file interactions absent from synthetic samples.

4.5 Diff Size as the Dominant Predictor↩︎

To isolate the effect of diff size, we stratify the combined dataset by line count (Figure 3).

F1 drops from 0.657–0.800 on small diffs to 0.043–0.070 on diffs exceeding 50 lines. The 10–50 line bucket performs best, suggesting a sweet spot where enough context exists to identify the bug without overwhelming the model. This points to diff preprocessing (splitting large PRs into per-function or per-hunk chunks) as the highest-leverage intervention for real-world review quality.

4.6 Per-Language Analysis↩︎

Go reviews are markedly better (F1 0.485 vs. and 0.209) with near-perfect precision (0.941, only 1 FP). Go’s simpler syntax, explicit error handling, and stronger typing likely make bugs more structurally identifiable from diffs.

4.7 Model Ensemble Analysis↩︎

We test whether combining models (running two models independently and taking the union of their findings) improves coverage (Table 9). If the models detect different bugs, the union should have higher recall.

Ensembles hurt F1. The models largely detect the same bugs; adding a second model introduces its false positives without meaningfully increasing true positives. Inter-model agreement confirms this: for 55 of 150 samples, all five models found something (overlap); for 19 samples, no model found anything (shared blind spot). Model diversity does not address the fundamental capability gap.

4.8 External Validation: Martian Code Review Benchmark↩︎

To verify our findings are not artifacts of our benchmark or judge, we evaluated against the Martian Code Review Benchmark [2], an independent, open-source benchmark maintained by Martian AI. The offline track evaluates code review tools against 50 real PRs from 5 repositories (Sentry, Grafana, Cal.com, Discourse, Keycloak) with 136 human-curated “golden comments” and an independent LLM judge. The Martian leaderboard includes commercial tools such as Cubic Dev, Qodo, Augment, GitHub Copilot, CodeRabbit, and Greptile.

Haiku outperforms Sonnet (F1 36.4% vs.%), placing #9 on the leaderboard between GitHub Copilot and CodeRabbit. Sonnet places below CodeRabbit.

4.8.0.1 Cross-benchmark consistency.

Table 11 shows the finding replicates across two independent evaluations with different datasets, repos, judges, and languages.

The gap is larger on Martian (34% F1 difference vs.%), likely because Martian uses exclusively real PRs with larger diffs, the regime where Sonnet’s conservatism is most costly.

4.9 Output Behavior Analysis↩︎

To understand why Haiku outperforms Sonnet, we examine their output behavior (Table 12).

Haiku generates 38% more findings than Sonnet (186 vs.), with the gap widening on real PRs (2.0 vs. per sample). This higher attempt rate translates directly to recall. Sonnet’s lower output token count (346 vs.) points to more aggressive self-censoring. Minimax and GLM are an interesting contrast: high token counts (919, 841) but few structured findings, suggesting verbose reasoning that does not produce actionable output.

4.10 Cost-Efficiency Frontier↩︎

Figure 4 plots F1 against cost per review. Two models define the Pareto frontier: GPT-5.4 mini (lowest cost) and Haiku 4.5 (highest quality). Sonnet 4.6 is dominated: Haiku achieves higher F1 at 3.2\(\times\) lower cost.

4.11 Qualitative Assessment↩︎

Haiku scores highest on all four dimensions. The widest margin is on specificity (3.39 vs.Sonnet’s 2.99, +13.4%), indicating more concrete and actionable findings. Sonnet places third, behind both Haiku and GPT-5.4 mini.

5 Discussion↩︎

5.1 Why Does Haiku Outperform Sonnet?↩︎

The output behavior data (Table 12) suggests several contributing factors:

5.1.0.1 1. Calibration threshold.

Code review requires balancing comprehensiveness with restraint. Sonnet’s deeper reasoning may set an overly conservative threshold; it produces 27% fewer findings than Haiku (135 vs.) and 30% fewer output tokens, despite greater capacity.

5.1.0.2 2. Pattern recognition sufficiency.

The bugs in our benchmark (SQL injection, null dereferences, missing error handling) are well-represented in training data. For these patterns, additional reasoning depth yields diminishing returns.

5.1.0.3 3. Output format compliance.

The prompt requires structured JSON with 8+ fields per finding. Smaller instruction-tuned models may comply more consistently, producing more complete and parseable output.

5.1.0.4 4. Real-PR generation gap.

On real PRs, Haiku averages 2.0 findings per sample versus Sonnet’s 1.1. The higher attempt rate translates directly to recall gains. This is consistent with [13], who showed model size advantages are task-dependent.

5.2 The Synthetic Evaluation Trap↩︎

The 92% F1 degradation on real-only samples (Table 7) deserves scrutiny. Synthetic mutations are localized (median 5 lines), familiar (well-studied patterns), and unambiguous (single injected bug). Real bugs share none of these properties. Evaluations on synthetic benchmarks alone [3], [4], [14] may overstate capability by up to 12\(\times\) (\(0.847 / 0.066\)). Future evaluations should include real PRs, or at minimum report synthetic and real results separately.

5.3 The Performance Blind Spot↩︎

All five models fail on performance bugs (0% recall for four, 4.5% for one). Detecting N+1 queries and unbounded queries requires reasoning about execution context, data scale, and architectural role, information that PR diffs simply do not contain. Static analysis (AST pattern matching, data flow analysis) remains necessary for this category.

5.4 Implications for Practitioners↩︎

1. Use smaller models. Haiku-class models match or exceed Sonnet-class at 3\(\times\) lower cost.

2. Preprocess large diffs. The 15\(\times\) F1 degradation on large diffs (Figure 3) is the most actionable finding.

3. Supplement with deterministic rules. Performance bugs and architectural issues need AST-based detection.

5.5 Limitations↩︎

1. Judge bias. Claude Opus judged all models, including Anthropic’s own. Qualitative scoring may favor Anthropic’s output style.

2. Ground truth quality. Real PR annotations are auto-extracted, not human-labeled. Some may capture refactoring alongside bugs.

3. Single prompt. Provider-specific optimization may change rankings.

4. No fine-tuning. All models evaluated in general-purpose configuration.

5. Temporal validity. Results reflect model versions as of March

6. Sample size. 150 samples (50 real) identifies large effects but not small differences with statistical significance.

5.6 Data and Code Availability↩︎

The evaluation framework, synthetic mutation engine, and raw results are available in the vibeops-mcp/evals/ directory of the VibeOps repository. Synthetic samples are reproducible from seed 42. Real PR samples are derived from public repositories via provided mining scripts. We plan to release the dataset on HuggingFace Datasets following human validation of the real PR annotations.

6 Conclusion↩︎

Five LLMs were evaluated on automated code review across 150 samples. Four conclusions stand out:

1. Bigger isn’t always better. Haiku 4.5 outperforms Sonnet 4.6 on F1, recall, and all qualitative dimensions at 3.2\(\times\) lower cost, across three conditions and confirmed on the Martian benchmark.

2. Synthetic benchmarks are dangerously optimistic. F1 \(=\) 0.066 on real PRs vs. on synthetic, a 92% drop.

3. Diff size is the key bottleneck. F1 drops 15\(\times\) between small and large diffs.

4. Performance detection is beyond current LLMs. Near-zero recall on N+1 queries and unbounded queries.

For production systems, Haiku 4.5 is the recommended primary model. Deterministic rules and diff preprocessing address the categories where LLMs fail. Ensembling multiple models does not improve results.

Acknowledgments↩︎

This work was conducted as part of the VibeOps project.

References↩︎