¹

1 Introduction↩︎

Most coding agent benchmarks ask one question: did the agent finish the task [1], [2]? For a research leaderboard, a single pass/fail bit is often enough. But for a coding assistant that developers use all day, most of what matters is not captured by that bit. A run can pass a narrow check and still be unreliable, overconfident, unpleasant to work with, or subtly misaligned with what the user actually asked for. Conversely, a run that fails the final check may have done plenty of useful things on the way there: localized the bug correctly, used its tools safely, recovered gracefully from an error and so on.

The problem is most important for tasks that have no natural binary outcome, such as, e.g., writing project documentation. A single success flag cannot say much about it: the useful question is not whether a file appeared on disk, but whether the document is accurate, appropriately scoped, readable, grounded in the actual codebase, and helpful to the readers.

Our proposed benchmark, AgentLens, is built to keep this structure visible, evaluating complete trajectories rather than final states alone. A trajectory is the full record of a session: messages from the (simulated) user, the agent’s replies, its tool calls, file edits, command executions, verification attempts, the final response, and the final state of the repository. AgentLens treats this record as the primary object of evaluation and uses it to measure quality closer to the way a user would perceive it. The benchmark uses textual judge reviews as the main artifacts, while formal verification supplies complementary objective checks wherever such checks exist.

We built AgentLens because the benchmarks we had did not match what we needed to know: some lacked realistic user interaction, some collapsed everything to a binary or final-state outcome, and some used tasks that were either too simple or too far from the day-to-day work of a coding assistant. We wanted something that evaluates production-like interactive work and then explains how agents differ.

Our main contributions in this work are:

• a production-assessed benchmark for interactive code agents;

• a trajectory-level evaluation protocol that combines formal verification with LLM judge reviews;

• side-by-side reviews for model comparison, feature evaluation, and regression analysis;

• an open-source release of the benchmark code at https://github.com/agent-lens/agent-lens-bench;

• a nightly evaluation pipeline that detects regressions in an actively developed coding agent.

The rest of the paper is organized as follows. Section 2 places AgentLens in the context of prior code-agent, tool-use, and LLM-as-judge benchmarks. Section 3 is the core of the paper: it describes the task set and simulated users, the trajectory review protocol and its metrics, the quality index, side-by-side reviews, the reliability and bias checks, and the production pipeline that runs all of this nightly. Section 4 puts the benchmark to work, with a leaderboard of current models (Section 4.1), a qualitative analysis of how they differ and what the reviews catch (Section 4.3), and a correlation study against public benchmarks (Section 4.4). Sections 5 and 6 conclude the paper with limitations and future work.

2 Related Work↩︎

AgentLens builds on three lines of work: code-agent benchmarks, interactive tool-use environments, and LLM-as-judge evaluation. The main novelty of AgentLens is in the object it evaluates, which is a complete, production-like trajectory including the user request, the agent’s messages, its tool calls, file edits, verification attempts, recovery behavior, final answer, and final repository state.

2.0.0.1 Code and software-engineering benchmarks.

Early code generation benchmarks evaluate isolated programming tasks, usually with unit tests as the success criterion. Repository-level benchmarks move closer to real software engineering. SWE-bench evaluates whether language models can resolve real GitHub issues by modifying repositories and passing tests [1], and SWE-agent shows that the agent–computer interface is itself an important factor in automated software engineering [3]. These benchmarks are close to AgentLens in their focus on coding agents and realistic repositories, but they primarily measure final-state task resolution. AgentLens keeps formal verification where it is available and complements it with trajectory-level reviews that capture instruction following, tool discipline, validation behavior, recovery from errors, and user-facing communication.

2.0.0.2 Interactive and tool-using agents.

A parallel line of work evaluates LLMs as agents acting over multiple steps in external environments. AgentBench studies agent performance across diverse environments [4]; ToolLLM evaluates the ability to use large collections of real-world APIs [5]; WebArena and OSWorld introduce realistic web and computer-control environments [6], [7]. Most relevant to our interaction model, \(\tau\)-bench evaluates tool-agent-user interaction in real-world domains using a simulated user with a role and a hidden goal [8]. AgentLens borrows this idea for coding assistant evaluation: user behavior is generated by another LLM conditioned on a persona and a task objective. This lets us evaluate production-like interactive coding workflows not only by whether the agent completes the repository task, but also by how it behaves with the user throughout the session. Each recorded trajectory can be reused for flexible comparisons across agents and quality dimensions.

2.0.0.3 Terminal and production-agent benchmarks.

Terminal-Bench evaluates AI agents in terminal environments on tasks spanning software engineering, system administration, security, and data science [2]. AgentLens also benchmarks CLI agents on realistic tasks drawn from everyday developer workflows, where agents must inspect projects, edit files, run tools, and recover from failures, but AgentLens is built mostly for product assessment; the goal is, again, not only to measure task resolution but to diagnose user-visible quality in a deployed coding assistant.

2.0.0.4 LLM-as-judge evaluation.

Finally, AgentLens relates to LLM-as-judge and pairwise preference evaluation. MT-Bench and Chatbot Arena show how LLM judges and side-by-side comparisons can support scalable evaluation of open-ended assistant behavior [9]. AgentLens uses LLM judges to score trajectories along metric-specific dimensions and, moreover, supplements the scores with written reviews that lay out the evidence behind them. These reviews make it possible to tell whether a low score came from poor tool use, incomplete verification, misleading final claims, weak instruction following, or simply an unpleasant interaction style. AgentLens then summarizes reviews across all trajectories in a run, surfacing the most frequent and important failure modes so developers can prioritize fixes.

3 AgentLens↩︎

3.1 Task Set and Runs↩︎

AgentLens is designed to be easy to extend. Adding a task requires two things: a task description for the user simulator and a verification setup for the formal metrics. In common cases, verification can reuse existing checks—running tests, inspecting repository state, or searching for an expected code change—and for specialized cases one can drop in a task-specific verifier (the available verifiers are listed in Table 1). New judge metrics can be added when a benchmark fold targets a specific behavior or when developers want to inspect an additional quality dimension; a test-generation fold, for instance, can score mocking strategy with a test-specific metric instead of general coding-assistant behavior.

We release an initial fold of 16 coding agent scenarios. Each scenario is run with two user personas, a neutral default user and a toxic user (personas are defined below), which yields 32 trajectories per evaluated agent. The folds are deliberately compact so that each one can run often in production. The design follows a simple observation that any developer who has lived with a coding assistant for a week will recognize: after four or five real sessions you can usually tell which agent is nicer to work with, and what its recurring strengths and failure modes are. AgentLens approximates that judgment by having LLM judges review and score complete interaction trajectories rather than isolated final outputs. The low run-to-run variance reported in Section 4.1 suggests that this compact setup is, in fact, stable.

3.1.0.1 Where the tasks come from.

The released fold is organized around workflows: each scenario specifies a task objective together with a sequence of instructions and interaction steps. We obtained these workflows from interviews in which programmers walked us through work they had done earlier that day or the day before. We asked follow-up questions to recover the specific actions involved—inspecting code, editing files, searching documentation, running tests, debugging failures, reviewing changes—and we asked when each activity had last occurred, giving more weight to workflows developers had performed recently and treating rarely performed activities as less representative.

In addition to these interview-derived workflows, the fold includes production-derived scenarios constructed from anonymized usage summaries, which complement the interviews with workflows observed in real product use. These scenarios come from chats of users who consented to anonymous data collection. We first converted each conversation into an anonymized task summary and processed only the summaries from then on, extracting coarse tags such as programming language, technology, task type, and application domain. This privacy-preserving usage-analysis step is similar in spirit to Clio, which summarizes and clusters real assistant conversations to surface aggregate usage patterns without exposing raw conversations [10]. We clustered the tag vectors with \(k\)-means and inspected large clusters that were not yet represented in the benchmark (cluster examples appear in Appendix 10.0.0.1). For each such cluster, a programmer picked a matching open-source project, generated a candidate scenario with the coding assistant, and then revised it for realism, anonymity, and verifiability.

3.1.0.2 User simulator personas.

Because the user is itself an LLM, we can vary its temperament and watch how robust an agent is to different interaction styles. We instantiate the simulator with three personas:

• the default persona is a relaxed user who does not proactively help the agent and intervenes only when there is clear evidence of failure;

• the helpful persona is cooperative: when the agent makes a visible mistake, it briefly points out the issue, suggests a possible correction, or offers concise guidance in a neutral tone;

• the toxic persona is mildly frustrated and occasionally impolite—it may challenge or tease the agent—but it still preserves the underlying task objective and never gives intentionally misleading instructions.

Together these let us test whether an agent can finish a task not only with a cooperative user, but also under sparse feedback and mildly adversarial conditions. The released fold pairs each scenario with the default and toxic personas; the helpful persona is available for studies of agents that benefit from in-loop guidance.

3.2 Trajectory Review Protocol↩︎

The unit of evaluation is a full trajectory, evaluated by LLMs. This raises several concerns.

3.2.0.1 Can we trust an LLM to grade other LLMs?

This is the first question to ask, and it is a fair one. In our experience, human annotation is not the obvious gold standard it has been often assumed to be for long coding trajectories: each item requires reading a lengthy interaction trace and applying a multi-page rubric, which is exhausting and error-prone. Early in the project we ran some informal internal pairwise-annotation experiments, and our consistent impression there was that agreement between an LLM judge and our human annotators was at least as high as agreement among the annotators themselves; this is especially plausible given how taxing these trajectories are to grade by hand. We note that those experiments were preliminary and small in size, so we do not claim a conclusive result here and mark a formal agreement study for future work. However, it is directionally consistent with prior LLM-as-judge work, which finds that strong judges can reach human-level agreement with human preferences, comparable to the agreement between two human annotators [9].

3.2.0.2 Formal verification.

Each trajectory also receives formal checks wherever they apply: tests, regular-expression checks, repository-state checks, and task-specific validators. These checks are valuable because they give objective evidence about specific requirements. But they are only one part of the picture: on their own they usually cannot see interaction quality, instruction compliance, misleading validation, unsafe tool use, or partial coverage of a requirement. Reviewing the full trajectory also blunts reward hacking against a narrow success predicate. An agent can satisfy a binary check while using brittle shortcuts, skipping required validation, or misreporting the final state; all of that stays visible in the trajectory review even when the check is green. Table 1 lists the verifier types currently available.

3.2.0.3 LLM-judge metrics.

Each trajectory is reviewed by LLM judges along five dimensions:

• EndResult looks at the final outcome only, measuring completeness and fitness for purpose with respect to the user’s request;

• InstructionCompliance checks whether the agent followed the user’s explicit rules, requested steps, ordering constraints, and meta-instructions, independently of how good the final outcome was;

• Pitfalls catches the fixable behavioral failures that hinder the user: tool misuse, non-productive loops, premature completion, missing validation, workflow instability;

• Pleasantness captures user-facing interaction quality: clarity, accuracy, conciseness, helpfulness, and whether the session feels productive and non-disruptive;

• ToolCalls assesses tool-use quality: tool choice, argument correctness, success or failure, recovery from tool errors, and efficient versus futile calls.

This set is meant to be comprehensive for general coding agent evaluation (test generation uses a different metric set; see the repository). Read together, the five dimensions separate the ways an agent can be good or bad: EndResult asks whether the assistant ultimately solved the task; InstructionCompliance asks whether it solved the requested task under the user’s constraints; Pitfalls exposes process-level instability; ToolCalls diagnoses failures in agent–environment interaction; and Pleasantness captures what it actually feels like to work with the assistant.

For every dimension the judge produces both a score and a written review (see the example prompt in Appendix 9.1). The review cites the trajectory evidence behind the score, so an aggregated report can preserve not only metric values but the reasons for them. Scores live on a small ordinal scale (for Pitfalls, \(0\) for miserable, \(0.5\) for tolerable, \(1\) for actually good), and each review ends with a handful of structured, reusable evidence lines of the form Aspect | Severity | Evidence. When scores are aggregated across a run, the reviews are summarized into a single report; hyperlinks such as [R1] point back to the exact trajectory locations where a problem occurred, so a developer can jump straight to the evidence (see the example report in Appendix 7, and the condensed excerpt in Figure 1).

None

3.2.0.4 Deterministic telemetry.

For every run we also collect hard statistics: interaction cost, cache hit rate, latency, termination reason, and generation-token throughput. We compute tool-use telemetry too—the mean number of parallel tool calls, mean tool calls per chat, total tool-call count, and per-tool success rates. These quantify efficiency and operational reliability independently of any subjective judgment.

Taken together, all of these metrics separate outcome quality from process quality, which matters because coding agents fail in different ways. One may reach a good final answer inefficiently; another may follow instructions perfectly yet fail verification; a third may use tools incorrectly behind a plausible-looking response; a fourth may produce something useful through a thoroughly unpleasant interaction. Scoring these dimensions separately gives diagnostic evidence rather than a single opaque number, which is exactly what you want when comparing general-purpose coding agents on realistic, multi-step work.

3.3 Quality Index↩︎

AgentLens reports the individual metric scores and a single aggregate quality index (QI). The current index is the unweighted mean of the component scores, i.e., the five judge metrics together with formal verification: \[Q(a) = \frac{1}{|M|}\sum_{m \in M} s_m(a),\] where \(a\) is an agent, \(s_m(a) \in [0,100]\) is its aggregate score on metric \(m\) (five listed above plus \(\mathrm{\small Formal}\)).

We treat QI as a deliberately simple proxy for production quality. The metrics already cover the main dimensions of user-visible coding-agent behavior—final usefulness, instruction following, tool use, recovery from failures, and interaction quality—so by Occam’s razor the natural first index is just their unweighted average. As a sanity check, we compared this index against the qualitative “vibe check” developers form from the reports, and the ranking is mostly intuitive: agents that feel reliable score high, while agents that need babysitting lose points for exactly the visible reasons one would expect—unfinished work, misleading validation, broken final states, or poor workflow control.

We also inspect correlations between metric aggregates across systems. The raw correlations in Table ¿tbl:tab:absolute-metric-correlations? are uniformly positive, which is unsurprising: stronger systems tend to score higher on several dimensions at once. Raw correlations therefore mostly confirm a broad “quality factor” rather than telling us which metrics are redundant.

To factor out this common-quality effect, we also look at each metric divided by the quality index. Because the index is a scaled sum of the components, the normalized metrics for a system sum to a constant, so Table [tbl:tab:relative-metric-correlations] describes metric profiles: a positive correlation means two dimensions tend to take a large share of the total together (compensated elsewhere), while a negative correlation means they act as direct counterweights.

This profile view suggests the metrics carry largely non-redundant information. Strong positive off-diagonal correlations are rare after normalization, so no two dimensions consistently occupy the same part of the quality profile. The clearest counterweights are End Result versus Tool Calls (\(r=-0.66\)) and Formal Verification versus Pleasantness (\(r=-0.74\)): some systems earn a larger share of their score from final-state success or formal passes, while others earn it from tool-use discipline or user-facing interaction quality. Instruction Compliance has only weak pairwise associations, which suggests it is not merely a proxy for final outcome or tool use. These results support reporting the component metrics, not just the aggregate index.

3.4 Side-by-Side Reviews↩︎

AgentLens also supports side-by-side comparison of two trajectories from the same scenario. We reach for this when absolute scores are not enough to explain a practical difference between agents, and we use it for three things. First, it exposes model-specific behavior—verbosity, literalness, recovery from errors, verification discipline. Second, it helps compare strong and weak models beyond their leaderboard position. Third, it detects regressions introduced by a change to the agent, the tool layer, or the runtime environment.

3.4.0.1 Pairwise LLM judge.

For each task instance, the pairwise judge compares two agents that solved the same user request. It receives both interaction trajectories together with the independent single-trajectory reviews for the dimension under comparison. Those prior reviews are the primary evidence; the raw trajectories are consulted only to verify concrete details such as a specific tool call, an error message, or a final output.

3.4.0.2 Prompt and input structure.

The prompt asks the judge to compare Agent 1 and Agent 2 with respect to a single performance dimension—instruction compliance, tool use, end-result quality, test quality, and so on. It explicitly tells the judge to stay within that dimension, avoid introducing unrelated criteria, and avoid re-deriving the individual reviews from scratch. This keeps the comparison anchored to the same evidence for both agents and reduces positional bias (Section 3.5.2). Each pairwise prompt contains four conceptual blocks: (1) the trajectory of Agent 1; (2) the trajectory of Agent 2; (3) a prior single-agent review of Agent 1 for the target dimension; (4) the corresponding review of Agent 2. Both trajectories correspond to the same task instance, and the dimension-specific prior reviews are the main basis for the decision (see the example prompt in Appendix 9.2).

3.4.0.3 Output format.

The judge produces a short comparative explanation followed by structured evidence lines:

Aspect: <description> | Winner: A1/A2/Tie |
Severity: low/medium/high | Evidence: <short evidence>

It then assigns an integer pairwise score. Note that severity here is relative: it measures the gap between the two agents on an aspect, not how bad either is in absolute terms, so two agents that fail identically score a tie.

3.5 Reliability and Bias Checks↩︎

3.5.1 Score Variance↩︎

To estimate run-to-run variance, we repeated the same GLM-5.1 (self-hosted) configuration five times. We picked GLM-5.1 on purpose as a mid-range model: strong enough to complete many scenarios, but shaky enough to actually expose variance in the benchmark. Across the five runs, the quality index had mean \(67.28\) and standard deviation \(0.94\) on a \(0\)–\(100\) scale.

Most of that variance came from formal verification rather than variance in judge scores. A component-level decomposition attributed \(60.5\%\) of index variance to formal verification, followed by Pitfalls at \(18.1\%\) and End Result at \(16.2\%\). In other words, repeated runs mainly expose unstable task completion: whether the agent reaches a formally verified final state on borderline scenarios.

Figure 2 shows the same effect at the scenario level. Of the 32 scenario–persona points, 15 passed formal verification in all five runs, one failed in all five, and the remaining 16 were flaky. The benchmark’s variance is thus concentrated in a small set of unstable points rather than distributed uniformly across all tasks, which makes repeated evaluation useful to pinpoint the scenarios where an agent’s behavior is nondeterministic or fragile.

3.5.2 Side-by-Side Positional Bias↩︎

To check whether side-by-side reviews are sensitive to the order in which the agents are presented, we ran an order-swap test. For a pair of agents \(A\) and \(B\), we compare two prompts: \(A\) as Agent 1 with \(B\) as Agent 2 and vice versa. Scores are oriented as the advantage of Agent 2 over Agent 1, so perfect antisymmetry would give \(s(A,B) \approx -s(B,A)\). For each metric we report the stable model difference, \(\mathrm{effect}=(s(A,B)-s(B,A))/2\), and the order residual, \(\mathrm{residual}=(s(A,B)+s(B,A))/2\).

Table ¿tbl:tab:sbs-order-swap? reports two such checks. The GLM-5.1 pair is a close same-model comparison; the Opus–Haiku pair is a high-contrast one (Haiku was run in no-thinking mode to widen the quality gap), and all five metrics preserve their direction after swapping sides. In both settings the residuals are small in absolute value. This does not prove that positional bias is absent, but it does suggest that order effects are secondary in these aggregate checks.

3.5.3 Judge Self-Preference↩︎

LLM judges can favor outputs from their own model family, especially when the systems being compared are close in quality. To gauge how large this effect is in our setting, we compared the same GPT-5.5-vs-Sonnet-4.6 trajectories twice, once with GPT-5.5 as the judge and once with Sonnet 4.6. These are two frontier models with close quality indices, so this is precisely the regime where self-preference should be the most visible.

The two judges picked different winners in \(23\%\) of task–metric comparisons, and the disagreements were not symmetric: in \(18\%\) of all comparisons each judge preferred the agent from its own family, while in \(5\%\) each preferred the other agent. Half of this self-favoring bias comes from Pleasantness alone, which contributes 11 self-favoring flips and zero opposite-favoring ones; it makes sense that this happened with the most subjective metric (Table [tbl:tab:self-preference]). On the score scale, the average shift between the two judges is about \(4\%\) of the full pairwise range.

In this comparison both judges still prefer the GPT-5.5 agent on average; the judge choice changes preference strength and borderline decisions, not the average winner; we report this as a caveat for close frontier-model comparisons. In the leaderboard experiments we keep GPT-5.4 fixed as the judge.

3.6 Production Pipeline↩︎

AgentLens is wired directly into our agent-development workflow. A candidate version is run on the benchmark set, producing trajectories, formal-verification results, metric reviews, aggregate scores, and side-by-side comparisons against a baseline.

3.6.0.1 CI benchmark workflow.

A continuous integration pipeline runs the automated benchmark evaluation. It is triggered either on a schedule or manually by a researcher. A preparation stage decides the evaluation mode (manual, nightly, or weekly). The benchmark stage then runs the selected tasks in parallel across multiple language- and IDE-specific environments. Each execution produces raw traces, logs, and agent-interaction dumps, stored as intermediate artifacts; after the run, these are aggregated and merged into a unified evaluation dataset. From the merged dataset the pipeline computes the quantitative metrics and prepares a report containing both metric values and judge reviews (an example report is in Appendix 7).

Beyond absolute metrics, the pipeline detects regressions through side-by-side comparison. The current run is compared against an anchor run, which is supplied explicitly for manual experiments and chosen automatically for scheduled ones. This comparison flags statistically significant performance changes between the current agent version and the reference (see the example report in Appendix 8). Detected degradations are treated as potential regressions and trigger a notification to maintainers. Both reports are saved in the experiment-tracking system. Figure 3 sketches the workflow.

Consider a real example: a race condition in parallel tool calling. It shows up as trajectory degradation, an abnormal termination reason together with worse ToolCalls and Pitfalls reviews. Since AgentLens exposes both the full interaction trajectory and the chat termination reason, the judges can attribute the regression to the underlying harness failure rather than merely noting a poor final answer. In this case the Pitfalls judge wrote, in the side-by-side review: “Tool and harness stability was mixed, but the most debug-relevant failures include Agent 2’s explicit ConcurrentModificationException crash in [C5]…“ (the full side-by-side review is boxed in Section 4.3, Example 2).

3.6.0.2 Extensibility.

The benchmark is not tied to a single agent or IDE. Different CLI agents can be plugged in, and IDE-based assistants can be evaluated by implementing the required integration interfaces. For IntelliJ IDEA plugins, the integration is done directly through a specialized AgentEngine interface; for CLI-based agents, the same contract is exposed through a lightweight Python adapter that translates benchmark requests into command-line invocations and returns the resulting trajectories. Although this work focuses on Java, the benchmark adapts readily to other languages. We open-source support for IntelliJ IDEA and PyCharm and plan to extend the integration to more IDEs and languages.

4 Evaluation↩︎

4.1 Leaderboard↩︎

We evaluate AgentLens on Java coding-assistant tasks using two harnesses: our own Explyt AI Agent²(EAA) and Claude Code³. Table 2 reports full-system performance, so a score reflects the model together with its provider, agent loop, tool interface, and default execution policy—not the model in isolation. All values are percentages, and rows are sorted by quality index. All single-run reviews use GPT-5.4 as the judge. We use high reasoning effort where supported, except where the table notes otherwise; OpenAI and Anthropic models use their official providers, GLM-5.1 (self-hosted) runs locally, and the remaining models are served through OpenRouter.

8pt

4.2 Model Behavior Case Study↩︎

The leaderboard is the least interesting thing the benchmark produces. The reviews are where the story is, so in this section we walk through the most revealing anomalies and close calls, using single-run and side-by-side reviews to understand why models differ in reliability, tool use, instruction following, and raw API behavior.

4.2.0.1 Gemini 3.1 Pro Preview.

Gemini 3.1 Pro Preview underperforms as an interactive coding agent relative to its frontier-model billing, and the report makes clear that the problem is agent-loop reliability rather than weak code generation. It names workflow-control failures (“skipped or merged steps,” “premature edits/tests before approval”), validation failures (agents that “claimed all checks passed despite actual failures”), and editing failures (“stale-patch or destructive-edit loops,” “brittle bulk replacements or sed that corrupted code”). The point is not that Gemini cannot generate code—it is that high model capability did not translate into a dependable agent loop in this setup (full report in Appendix 7).

4.2.0.2 DeepSeek V4 Pro vs.DeepSeek V4 Flash.

The near-tie between DeepSeek V4 Pro and Flash hides a more interesting trade-off than the QI suggests. Pro is the more controlled agent: the side-by-side report shows a statistically significant instruction-compliance advantage, especially on step gating, required pauses, formatting, and file-boundary control (full report in Appendix 8). Flash trades compliance for progress—by relaxing step gating, formatting, and boundary constraints more often, it completes more work and more often lands in a usable, passing state within the fixed budget. Higher throughput (more than \(2\times\) in our run) reinforces the effect under a fixed task budget. So the similar indices reflect genuinely different strategies: Pro earns quality through control and safer interaction, Flash through completion and follow-through. A simple sum exposes the trade-off rather than hiding it.

4.2.0.3 EAA Haiku vs.Claude Code Haiku.

The gap between Explyt AI Agent Haiku and Claude Code Haiku is likely explained in part by Claude Code Haiku’s familiarity with the Claude Code harness, which may reflect prior training or evaluation exposure. The difference shows up in the unusually low success rate of the edit_file tool in EAA relative to Claude Code (\(0.66\) vs.\(0.93\)). It is also visible in the pitfall analysis, where Claude Code is significantly favored (metrics_value \(=0.1812\), \(p=0.038\)); the review puts the core distinction plainly, noting that Explyt AI Agent”more often makes poorly grounded or brittle changes and presents weaker verification.” For the stronger Opus 4.7 model, though, EAA already comes out ahead.

4.2.0.4 EAA Opus vs.Claude Code Opus.

This comparison splits along operational reliability versus explanatory and discovery strengths. Explyt AI Agent separates only narrowly on final outcomes—“no meaningful separation” overall, with identical formal verification at \(26\) vs.\(26\)—but the reviews still favor it on user-visible execution because it was “more consistent at finishing changes cleanly and verifying them.” Claude Code’s high-contrast weakness is operational risk: judges cite repeated “file/path and tool-contract errors,” “serious misses of explicit user constraints,” a “routing blast-radius regression,” a “leftover workspace mutation,” and the only timeout. Interestingly, the edit_file success rate reverses the Haiku pattern here: Claude Code Opus is lower than EAA(\(0.92\) vs.\(1.00\)).

4.2.0.5 GPT-5.5 vs.Opus 4.7.

This pair shows a clean split between procedural compliance and semantic reliability. GPT-5.5 often handles the explicit task protocol better—required message format, step boundaries, waiting for user confirmation, runner targeting, and compile/test closure. But the reviews credit Opus with higher “requirement coverage and final fitness,” fewer “likely route regressions” or cases of moving a “DB lookup ahead of the feature gating,” and stronger validation discipline: GPT-5.5 was more often penalized for “verification without covering changed branches,” “unsupported root-cause claims,” “missed affected tests,” or marking a task done “despite not reproducing the failure.” In short, GPT-5.5 is better at following the task form; Opus is better at preserving the task intent.

4.3 Qualitative Analysis↩︎

The case study above used reviews to compare models. But the reviews are most clearly useful when they catch something a score alone would get wrong. Here are two such cases; in both, the bare leaderboard number is, by itself, misleading, and only the written review recovers the truth.

4.3.0.1 A bottom-of-the-table score that is really a provider bug.

Kimi K2.6 sits dead last in Table 2, which invites the obvious reading: it is a weak coding agent. But the ToolCalls review says otherwise: the dominant failure is not reasoning but a malformed-argument parsing problem on the OpenRouter side (the same JSON-wrapper mistake repeated across several tools), and once the arguments are flattened the agent does in fact use its tools successfully.

4.3.0.2 A regression hiding behind a normal-looking answer.

The second example comes from the nightly pipeline (Section 3.6). During a weekly regression run, a race condition in parallel tool calling left little obvious trace in the final answer, but the side-by-side Pitfalls review pinned it to a concrete crash. Because AgentLens exposes the termination reason alongside the full trajectory, the judge could attribute the degradation to the harness—an explicit ConcurrentModificationException—rather than vaguely noting that the output had gotten worse.

For complete artifacts see Appendix 7 and Appendix 8.

4.4 Correlation with Public Benchmarks↩︎

To place QI against the wider benchmark landscape, we computed Spearman rank correlations between our QI and every Artificial Analysis (AA)⁴ evaluation that reports scores on the current generation of models. Eleven evaluations qualified, including the agentic benchmarks APEX-Agents-AA, GDPval-AA, Terminal-Bench Hard, and \(\tau^2\)-Bench Telecom; ten cover all eleven IDE-agent models in Figure 4, and APEX-Agents-AA covers seven. AA also aggregates these into a single Intelligence Index (hereafter the AA composite)⁵, a weighted average designed to track overall model capability. All ranks below are within these eleven models.

The QI column shows that Mimo V2.5 Pro and Gemini 3.1 Pro Preview sit much higher on the AA composite than on our QI (rank gaps of \(+4.9\) and \(+3.8\); \(90\%\) bootstrap CIs \([+3.4,+6.4]\) and \([+2.4,+5.1]\)), consistent with optimization pressure toward AA-style academic and reasoning evaluations. Claude Opus 4.7, Claude Sonnet 4.6, and DeepSeek V4 Flash show the opposite, with rank gaps from \(-3.4\) to \(-3.9\) and CIs strictly negative: they handle long-horizon IDE work substantially better than the AA composite credits them for, with the gap concentrated on IFBench and \(\tau^2\)-Bench Telecom, where the Anthropic models in particular score well below their overall agentic quality. GPT-5.5 and DeepSeek V4 Pro sit at zero with CIs spanning the origin, the only two models for which the two rankings agree.

Two things follow. First, QI tracks an axis of model quality that does not collapse to any single AA evaluation: the strongest individual correlate (APEX-Agents-AA, \(\rho=0.82\)) is itself an agentic benchmark, while reasoning- and instruction-following-oriented evaluations diverge sharply, including the negative correlations of IFBench (\(\rho=-0.41\)) and \(\tau^2\)-Bench Telecom (\(\rho=-0.25\)). Second, the rank-gap residuals look systematic rather than noisy: they split the cohort along a recognizable line between models trained for evaluation-style competence and models built for sustained tool use under realistic constraints. The full inter-benchmark matrix in Figure 5 makes the same point from another angle: the public evaluations correlate strongly with one another, while our QI is the row that stands apart—it agrees with the agentic benchmarks and openly dissents from the reasoning- and instruction-following-oriented ones.

5 Limitations↩︎

We note several limitations of our study. First, AgentLens evaluates a specific class of coding-agent tasks, not general model intelligence, and the present release is Java-only.

Second, Several evaluated models are accessed through third-party API providers (e.g.OpenRouter), so latency, routing, model versioning, and availability are outside our control. The DeepSeek Pro-vs-Flash case in Section 4.2 is a specific illustration for this effect: a \(2\times\) throughput difference under a fixed task budget significantly affects how much work each agent can finish, which means some leaderboard differences reflect serving conditions as much as model quality. Third, benchmark cost is also non-trivial: a single run with Opus 4.7 can exceed $100.

Finally, AgentLens was originally built around our own coding assistant, which may make some tasks or integration assumptions more natural for our system. We mitigate this by releasing the benchmark and reporting full trajectory reviews, but broader external use is needed to assess neutrality. The validity argument for our LLM judges (Section 3.2) currently rests on internal pairwise-annotation experience rather than a published agreement study; quantifying judge–human and judge–judge agreement on these long coding trajectories is an important direction for future work.

6 Conclusion↩︎

In this work, we have presented AgentLens, a benchmark that evaluates code agents through production-assessed trajectory reviews. It captures the user-visible quality dimensions that binary success metrics miss, and because every score comes with a written, evidence-linked review, it supports far more than ranking: diagnosis, feature evaluation, and nightly product-regression detection for a deployed coding assistant.

In future work, we plan to add integrations for more coding agents, including Codex and popular open-source agents such as OpenCode. Since AgentLens is pluggable, new CLI agents can be connected by implementing the required execution interface, and IDE-based assistants through the corresponding IDE integration layer. AgentLens already supports the same benchmark style across other JetBrains IDEs, including PyCharm, Rider, and WebStorm, and we plan to open-source these variants as the repository matures. The benchmark is also organized into task folds—test-writing tasks, for example, have their own fold with test-specific verifiers and judge metrics—which we likewise plan to release. This lets the benchmark grow by adding focused task families rather than by treating all coding-agent work as one homogeneous distribution.

Overall, we feel that the field of agentic coding benchmarks is undersaturated (albeit rapidly growing), and we hope that AgentLens fills a niche that is currently vacant.

7 Example Single-Run Report↩︎

8 Example Side-by-Side Report↩︎

Workflows↩︎

TL;DR↩︎

The main operational split is that Agent 1 was more trustworthy about protocol, diagnostic grounding, and not overstating validation, while Agent 2 more often got code and tests to a buildable, fully executed state but also had several serious cases of claiming success on weak or misleading verification evidence. On the outcome itself, the reviews are genuinely mixed rather than decisive: Agent 2 had more wins on completeness, fallback work, and executed verification, but Agent 1 had several of the more consequential wins where Agent 2 missed explicit requested behavior, introduced contract or fixture errors, or overfit the task; the aggregate outcome edge toward Agent 2 is small and not statistically reliable (p=0.117). Instruction-following shows a similar pattern, with Agent 2 somewhat stronger on sequencing, message format, and actually finishing required test steps, while Agent 1 was better on scope control in several important cases and more consistently followed special procedural requirements; the formal verification count was tied at 28-28, which is worth noting but only as a rough heuristic rather than strong proof of correctness. Operationally, Agent 2 was much cheaper and faster (\(9.13 vs\)​21.20, 10,973s vs 18,987s, both statistically significant), and its much higher command/run success rates align with the reviews showing steadier completion, whereas Agent 1’s extra cost and slower pace came with more timeouts/exceptions and poorer run-configuration success, even though its final reporting was often the more dependable of the two.

Total number of points (chats): 32↩︎

formal_verification_result↩︎

• alert: no

• warning: no

• p-value: 1.0

• metrics:

  • agent 1: 28

  • agent 2: 28

termination_reason↩︎

• alert: no

• warning: YES

• common: User decided to finish the dialog (26), Waiting for the agent response timed out after 900s (1)

• only_run1: Agent iteration failed with an unexpected exception: Agent loop failed with an unexpected error: Cannot reach provider OpenAI-compatible (1), Scenario timed out after 1200s (3), Waiting for the agent response timed out after 900s (1)

• only_run2: User decided to finish the dialog (5)

EndResult_Judge↩︎

• alert: no

• warning: YES

Review:

Across requirement fit and scope/completeness, the results are mixed: A2 leads in 20 Ck while A1 leads in 11, with the most important A2 wins coming from delivered or more complete fallback/test work in [C32], [C29], [C22], and [C26], but A1 has notable high-severity wins where A2 missed explicit requested behavior or overfit the task, especially [C25], [C21], and [C14].

On correctness and behavior preservation, neither side dominates overall (A2 wins 10 Ck, A1 wins 9), but the most consequential failures are split: A1 has high-severity wins in [C25] and [C21] because A2 introduced a wrong ‘/v2’ route contract and left an invalid username fixture, while A2 has high-severity wins in [C11], [C29], and [C4] because A1 changed control-flow semantics, left a broken buildable state, or omitted required ‘501’ handling.

Verification/build evidence favors A2 in 9 Ck versus 2 for A1, with the strongest gaps in [C29], [C32], and [C22] where A2 delivered passing builds or suites while A1 was broken, unrun, or incomplete; A1’s main counterexample is [C19], where A2’s final verification depended on extra skip flags rather than the agreed full command.

For documentation/report accuracy, the pattern is again mixed (A2 wins 8 Ck, A1 wins 7), with A1’s most significant advantages in [C9], [C14], [C21], and [C24] where A2 had factual errors, overclaimed a fix, or used overly absolute wording, while A2’s strongest wins in [C10], [C12], [C16], and [C22] come from catching inconsistent prerequisites, invalid diagrams, inaccurate completion claims, or unsupported rationale.

Safety/logging hygiene shows only a small signal: A1 has the main substantive win in [C1] by removing a risky full-request log, while [C2] is a tie on safe logging choices.

• judge alert: YES

Scoring

• p-value: 0.117

• metrics_value (from [-1..1]): 0.1375

InstructionCompliance_Judge↩︎

• alert: no

• warning: YES

Review:

Across the step-gating and sequencing evidence, Agent 2 has the stronger record: A2 wins 14 comparisons in this group, while A1 wins 10 and 9 are ties; the clearest cases are [C1], [C2], [C29], [C30], and [C32] for A2, versus [C13], [C20], and [C21] for A1.

On message-format compliance, Agent 2 also has the edge, winning 9 items versus 8 for Agent 1, with many of A2’s wins being more severe, including high-severity header/STOP-structure cases in [C26] and [C27]; Agent 1’s main counterexamples are [C20], [C21], [C23], and [C28].

Agent 1 is stronger on scope control and special-procedure requirements: in scope/allowed-files behavior A1 wins 7 items versus 6 for A2, with major A1 wins in [C5], [C8], and [C15] but major A2 wins in [C7] and [C18]; in tools/diagnostics/language/protocol specifics A1 wins 8 items versus 3 for A2, driven by call-tree and mandated-phrase cases such as [C14] and [C19].

Test-execution/verification evidence favors Agent 2 overall, with 6 wins for A2 versus 2 for A1 and 2 ties, especially in [C21], [C22], [C29], and [C30] where A2 is credited for tighter test scope or actually completing the required verification/reporting stage.

• judge alert: YES

Scoring

• p-value: 0.855

• metrics_value (from [-1..1]): -0.0187

Pitfalls_Judge↩︎

• alert: no

• warning: YES

Review:

Across workflow/protocol and verification/build-state reliability, Agent 1 had the stronger record: in workflow-related proofs A1 was better in 14 entries and worse in 4, with major separations in [C10] and [C24], and in verification/reporting A1 was better in 10 entries and worse in 9, including high-severity cases where A2 treated failing or stale runs as success in [C4], [C14], [C19], and [C20]. Agent 1 also led on logic/contract correctness, with 7 wins versus 5 losses in that group; the most consequential A2 misses were fixes that did not address the reported path in [C14], a filtering/pagination regression in [C18], and fallback handling errors in [C26], while A2’s main counterexamples were A1’s routing miss in [C3] and semantics break in [C11]. Agent 2’s clearest advantages were narrower and more operational: it was better on edit stability in 8 entries versus 7 losses, especially where A1 destabilized files in [C3], [C25], and [C29], and it was also better on some scope-control items such as [C1], [C7], [C17], [C21], and [C26], though A2 had severe scope violations of its own in [C8] and [C18]. Tool/harness hygiene was mixed rather than decisive, but several A1 losses there were tied to repeated command/schema mistakes or noisy reruns (for example [C13], [C25], [C29], and [C31]), while A2 had its own notable tool-side regressions in [C20] and [C32]. Overall, the extracted proofs point to Agent 1 being more reliable on protocol, diagnostic grounding, and behavior correctness, while Agent 2 was somewhat steadier on avoiding file corruption and some scope-follow-through but with less favorable high-severity verification failures.

• judge alert: YES

Scoring

• p-value: 0.109

• metrics_value (from [-1..1]): -0.125

Pleasantness_Judge↩︎

• alert: no

• warning: YES

Review:

Across the strongest evidence, Agent 1 leads on interaction hygiene: it wins verification/reporting trust in 11 Ck and loses it in 5, including high-severity cases where Agent 2 gave contradictory or misleading validation narratives in [C4], [C12], and [C14]; Agent 1 also leads on structure/protocol in 6 Ck versus 3 losses, with the biggest gaps in [C21], [C20], and [C23] where Agent 2 skipped or broke the required step format.

Agent 2’s clearest advantage is responsiveness and end-to-end completion: it wins responsiveness/alignment in 8 Ck and loses 1, with major cases in [C26] and [C32], and it wins completion/final usefulness in 4 Ck versus 4 losses, including strong delivery wins in [C29] and [C32].

On operational smoothness, the split is close but important: Agent 1 wins 11 Ck and loses 11, yet Agent 2 owns several high-severity wins in [C25] and [C29], while Agent 1’s biggest smoothness wins come from avoiding disruptive behavior in [C12], [C13], and [C16].

Boundary-respecting behavior is mixed rather than one-sided: Agent 1 wins scope-discipline in 3 Ck and loses 3, with both agents drawing criticism for touching forbidden files or constraints (for example ([C5]/[C6]/[C8] versus [C7]/[C28])).

• judge alert: YES

Scoring

• p-value: 0.317

• metrics_value (from [-1..1]): -0.0875

ToolCalls_Judge↩︎

• alert: no

• warning: YES

Review:

Across the proof groups, the largest repeated differences are in edit stability and verification reliability: Agent 2 is favored in edit/repository stability in 7 comparisons and loses 8, but Agent 1’s worse cases include several high-severity corruptions or malformed edits such as [C18], [C29], and [C5], while Agent 2’s high-severity losses are concentrated in [C8] and [C16].

Verification and runner use lean toward Agent 2 in 11 comparisons versus 6 for Agent 1, with Agent 2’s strongest wins coming from correctly scoped or completed test/build execution in [C21], [C25], [C26], and [C32]; Agent 1’s strongest wins in this area are mostly about Agent 2 masking Maven failures with shell pipelines in [C19] and [C20].

Exploration efficiency is mixed but slightly favors Agent 1 overall, with 13 wins for Agent 1 versus 10 for Agent 2; Agent 1’s wins are mostly about avoiding broad or empty searches (for example [C1], [C2], [C6], [C14]), while Agent 2’s wins come from more targeted narrowing in search-heavy tasks such as [C23] and [C24] and less repeated configuration probing in [C21]–[C22].

Scope/context/completion shows the sharpest directional signal toward Agent 2: Agent 2 wins 5 comparisons and loses 2, including high-severity cases where Agent 1 edited forbidden files or failed to complete the final requested tool work ([C7], [C26], [C32]), while Agent 1’s counterexamples are narrower scope violations by Agent 2 in [C5] and [C8].

Miscellaneous argument/path/output handling is broadly noisy on both sides, with many ties and small wins split both ways; the most notable non-tie cases are Agent 1 being better when Agent 2 makes unknown-tool or bad-command mistakes ([C20], [C30]), and Agent 2 being better when Agent 1 makes invalid read ranges or unhelpful file-type reads ([C9], [C22]).

• judge alert: YES

Scoring

• p-value: 0.92

• metrics_value (from [-1..1]): -0.0125

setups↩︎

• alert: no

• warning: no

• difference: unchecked

Inference metrics↩︎

Tool call success rates↩︎

Metadata↩︎

run1

• model_info: deepseek-v4-pro

• plugin_hash: ***

• run_name: DeepSeek_v4_pro-*

run2

• model_info: deepseek-v4-flash

• plugin_hash: ***

• run_name: DeepSeek_v4_flash-*

judge

• api_calls: ***

• cached_tokens: ***

• completion_tokens: ***

• model: OpenAI/gpt-5.4

• prompt_tokens: ***

• total_price_usd: ***

9 Example Judge Prompts↩︎

9.1 Pitfalls single-run judge prompt↩︎

9.2 Pitfalls side-by-side judge prompt↩︎

10 Benchmark Tasks↩︎

10.0.0.1 Task categories.

Table 3 shows examples of anonymized production-usage clusters covered by benchmark scenarios. The descriptions are generated from anonymized task summaries and are included to illustrate the type of real coding work represented in the benchmark.

4pt

References↩︎

---

1. \(^{\ast}\)Equal contribution. \(^{\dagger}\)Corresponding author: vadim.lomshakov@explyt.com↩︎

2. https://explyt.ai/en↩︎

3. https://claude.com/product/claude-code↩︎

4. https://artificialanalysis.ai/↩︎

5. https://artificialanalysis.ai/methodology/intelligence-benchmarking↩︎