1 Introduction↩︎

LLM-powered coding agents are now widely used in developer environments [1]–[3]. They inspect repositories, edit files, invoke tools, run tests, and iterate on feedback. As these systems move into production workflows, practitioners need to compare not only whether an agent solved a task, but also how it behaved while attempting the task. As the key metric, resolve rate records whether the final patch passes target tests. This outcome is useful for ranking systems and guiding release decisions, but it hides the process differences that matter to agent developers.

This limitation appears even when two runs reach the same outcome. Figure 1 shows an example where Claude Code (Opus 4.6) and OpenCode (GLM-5) both resolve the same SWE-Bench task. The pass/fail label treats both runs as successes, but their processes differ. Claude Code reaches a targeted fix with no failed actions, while OpenCode takes a longer path with repeated failures and recovery work. This example illustrates why process evidence is required in addition to pass/fail outcomes. Prior work has approached this need from two directions. One direction questions whether pass/fail outcomes are reliable, showing that generated patches can exploit weak tests, recall benchmark artifacts, or be affected by infrastructure and run-to-run noise [4]–[8]. The other direction evaluates or characterizes trajectories directly. AgentBoard scores progress against predefined sub-goals [9], TrajEval grounds failure analysis against reference patches [10], process-centric analysis profiles behavior [11], and Majgaonkar et al. [12] summarize success and failure trajectories with aggregate statistics. These methods show that agent trajectories contain useful evidence. However, they still leave a gap for process diagnosis. Practitioners need deterministic diagnostics that name concrete trajectory patterns, compare runs under controlled references, and state the oracle assumptions behind each signal.

Closing this gap is not trivial because raw trajectories are not directly comparable. A single run interleaves multiple heterogeneous events with no shared schema, including reasoning, tool calls, edits, shell output, and test logs. The relevant signals are also sparse, such as an early read of the wrong file, a skipped validation step, or repeated searches for the same evidence. Aggregate counts lose this local context, while manual inspection does not scale across large trajectory corpora. Process diagnosis therefore requires a representation that preserves local evidence and still supports corpus-level comparison. On top of this representation, the analysis should support comparison across agent settings, not only case-by-case debugging. It should give practitioners concrete failure clues, point to avoidable effort in resolved runs, and compare agent behavior under controlled references. To be auditable, these signals need deterministic rules, explicit oracle assumptions, and named structural patterns rather than LLM judgments or aggregate statistics alone.

This work asks three diagnostic questions: what observable clues are associated with failure, where successful runs spend avoidable effort, and what comparisons among different agents’ runs reveal about behavioral divergences. We present a framework named TraceProbe to answer these questions through a shared representation for within-trajectory diagnosis and between-trajectory comparison. TraceProbe normalizes each raw trajectory into canonical actions from a nine-type taxonomy: file read, file write, search, command, sub-agent spawn, plan, navigate, fetch, and reason. It then assigns each action a deterministic effect label from observable state transitions. Two rule-based modules build on this substrate. The Insight module profiles one trajectory for recurring failure modes such as search loops and verification skips, while the Converge module aligns two trajectories and classifies divergences such as off-anchor exploration and rapid rewrite. The structural detectors and most effect labels are oracle-free, while off-anchor labels, milestones, and benchmark-informed divergence classifications use gold-patch anchors when available.

We scope TraceProbe as a diagnostic framework, not a replacement for resolve rate, a per-run root-cause oracle, or a learned policy for fixing agents. It provides auditable signals for comparing process distributions across agent settings, especially when the study design holds the task, scaffold, or model fixed. Within this scope, we apply TraceProbe to 2,500 trajectories from SWE-Bench Verified tasks across five production settings spanning three scaffolds (Claude Code, Codex, OpenCode) and three model backbones (Opus 4.6, GPT-5.4, GLM-5). This design supports both same scaffold model contrasts and same model scaffold contrasts. Our study surfaces three results. First, comparing failed and successful runs on the same task shows where failure evidence should be localized. File choice alone is too coarse, because two successful runs can touch the same files, but clearer signals appear in function selection and completion behavior. Second, the Insight module mainly provides corpus-level difficulty clues. Search loops are the most stable clue, while other signals are split-sensitive. A smaller completion evidence signal shows only a modest failed-run gap. Third, the Converge module milestones and classifications show that successful runs can differ in how quickly they reach relevant code, how much failed work they perform, and how their process changes with the scaffold or model.

The main contributions of this paper are threefold:

1. We introduce a deterministic canonical action representation for coding agent trajectories that separates oracle-free structural diagnostics from anchor-grounded milestones and divergence evidence.

2. We define a detector and alignment methodology that turns established SE and debugging anti-pattern concepts into exact trajectory predicates and produces reference-scoped divergence evidence with explicit oracle assumptions.

3. We conduct a multi-setting study showing that trajectories reveal differences between agent settings that resolve rate hides, including where runs diverge, when they reach relevant code, and how much failed work they perform. All data and code are released at https://doi.org/10.5281/zenodo.20789918 for future study.

2 Related Work↩︎

2.1 Code Agent Evaluation↩︎

SWE-Bench [13] uses resolve rate as the default metric for coding agents, but a final pass/fail outcome is only a coarse signal. Prior work shows two reasons for caution. First, a generated patch can satisfy permissive tests while missing developer intent [4], [5]. Second, reported scores can be distorted by benchmark memory, infrastructure noise, benchmark saturation, or run-to-run variance [6]–[8], [14]. Recent benchmarks respond by making outcome evaluation broader and more robust. Some benchmarks collect live or contamination-resistant tasks [15], [16], while others target enterprise repairs, multilingual repositories, unified coding agent skills, or security sensitive fixes [17]–[21]. These efforts make final outcomes more informative. Our work asks a complementary question: what process produced that outcome? Unlike AgentBoard [9], which scores progress against predefined sub-goal sequences, TraceProbe normalizes observed agent events and compares trajectories directly without requiring a hidden ideal path. Agent scaffolding systems such as SWE-agent, OpenHands, and Agentless [22]–[24] improve coding-agent performance, but their evaluations still focus mainly on final outcomes. TraceProbe adds a complementary process view (i.e., a common representation) for asking how agents searched, edited, validated, recovered, and diverged.

2.2 Agent Trajectory Analysis↩︎

Beyond final-outcome benchmarks, a growing body of work evaluates the trajectories that agents produce. TrajEval [10], TRAJECT-Bench [25], and process-level configuration evaluation [26] support fine-grained trajectory assessment using reference patches, synthetic reference trajectories, or environment configuration tasks. Process-centric analysis [11] and empirical trajectory characterization [12], [27]–[29] also show that trajectory structure matters. These studies show that trajectories are useful, but they usually summarize them as scores, graphs, profiles, or correlations. Our work instead pairs oracle-free detectors with run-to-run alignment. It maps each raw trace to a shared action format, then applies named detector rules with fixed thresholds. As a result, each measurement is tied to an explicit rule and can be compared across agent settings. TraceProbe targets a different use case than trajectory scoring or LLM judge localization [30], i.e., corpus level comparison of how agent settings behave. For that purpose, Converge adapts event sequence matching and visual comparison ideas [31] to coding agent traces: it aligns two runs, localizes unmatched or reordered action spans, and reports reference-scoped diagnostics with fixed thresholds.

Trajectory analysis is also used for failure attribution, inspection, monitoring, and agent improvement. Failure attribution systems [32]–[37] localize why failed trajectories went wrong, while MAST and silent failure detectors focus on multi-agent failures [38], [39]. Our scope is single-agent SWE coding tasks. TraceProbe compares successful and failed runs and links each diagnostic signal to trace evidence. Adaptive monitors such as TRACE [40] trade repeatability for model-based flexibility, while our work uses fixed rules for repeatable measurement. Because these systems answer different questions, direct accuracy comparisons would not measure the same construct. We therefore use baselines defined on the same trajectories: success/failure outcomes alone and raw telemetry such as steps, tokens, and tool counts. Inspection and visualization systems help users examine traces directly [41]–[44]. Other work uses trajectories to improve agents during or after execution, for example through reward models, course correction, trajectory reuse, harness evolution, or richer trace representations [45]–[52]. TraceProbe’s core contribution is a reusable detector catalog for real SWE trajectories, which helps practitioners apply the same diagnostic rules across agent settings and compare process differences with trace level evidence.

2.3 Anti-Pattern Detection in Software Engineering↩︎

Software engineering research has long used named patterns to describe code smells [53], design anti-patterns [54], organizational debt [55], socio-technical anti-patterns in ML-enabled software [56], and CI/CD configuration issues [57]. Many of those settings scan static artifacts, but agent trajectories are dynamic event sequences whose shape depends on scaffold, prompt, and tool interface. Our work brings this anti-pattern tradition to trajectory analysis by using named signals for recurring agent behavior. It maps raw events to canonical actions with deterministic effect labels, then defines structural detectors over that representation. For example, analysis paralysis and thrashing [54] motivate search-loop and re-read-churn detectors. Prior trajectory characterization and failure taxonomy studies identify important behavior patterns [58]. TraceProbe turns such patterns into a detector catalog that can be tested by outcome group, reused across agent settings, and used to explain what changed in an agent’s process.

3 TraceProbe: Offline Trajectory Analysis↩︎

3.1 Architecture Overview↩︎

At a high level, Figure 2 shows how TraceProbe separates representation construction from diagnostic analysis. Before describing the modules, we define six terms used throughout the methodology. A reference is another trajectory chosen for comparison. An anchor set is the set of task-relevant files supplied by a benchmark patch when available. A canonical action is a typed event record, and an effect label records its observable outcome. A divergence span is a contiguous block of actions that remains unmatched or reordered after Converge aligns two runs. A milestone is the first step that reaches a task-relevant artifact, and an Insight detector flags a named pattern in one trajectory. TraceProbe first converts raw trajectories into normalized steps, typed canonical actions, and deterministic effect labels. Insight then analyzes one run without a reference or anchor set. Converge compares a run with a reference trajectory and labels its divergence spans. Milestones use the anchor set to record when a run first reaches task-relevant artifacts.

3.2 Shared Representation↩︎

To compare trajectories across scaffolds, TraceProbe first builds a shared deterministic representation. The representation starts with normalized steps that preserve event context in a common schema. Each step is then converted into a canonical action, which records the action type and comparable target. On top of this action sequence, TraceProbe derives workflow phases to summarize coarse progress and effect labels to record observable action outcomes.

This process starts with scaffold adapters. They translate scaffold-specific traces into a common step schema that preserves role, typed tool-call records, resource use, timing, and error status. This layer absorbs format differences across agents. For example, Claude Code may split one API response across multiple trajectory entries, while OpenCode stores comparable information inline. TraceProbe then maps each normalized step to a canonical action with an action type and target. The action type uses the nine-label set in Table 1 to distinguish the kind of work performed by the step. The target stores the comparable object for the action, such as a path, command, query, scope, or plan summary. File targets use repo-relative paths, allowing Converge to align the same artifact across different local work directories.

TraceProbe also derives a deterministic workflow phase from observed step fields. The classifier assigns the phase from the step’s dominant event, separating planning, implementation, validation, debugging, repository inspection, and reporting activity. These phases support phase-level profiles and provide the phase set used by semantic tags.

To decide whether a read, search, or milestone touched task-relevant code, TraceProbe needs a task-relevance set. When a benchmark supplies a ground-truth patch, as in SWE-Bench Verified, the modified files define the trajectory’s anchor set, or gold-patch file set. TraceProbe uses these anchors for off-anchor labels, milestone extraction, and trajectory comparisons that refer to task-relevant artifacts. An off-anchor label marks a successful read or search outside the anchor set. For non-benchmark traces, TraceProbe can instead derive task relevance from surviving writes and test/import references. The detector catalog uses only trajectory events, not the SWE-Bench issue description. As a result, same-task comparisons are driven by what agents did rather than by prompt text that is identical across agents.

After action typing, TraceProbe assigns each action a deterministic effect label using only observed trajectory data, not an LLM judge. A write is survived when it persists to the final state, and reverted when a later write to the same file overwrites it. An action is justified when it reads or searches a task-relevant file, or runs a validation command. A non-workspace meta-action, such as a plan update, navigation, fetch, or sub-agent spawn, is recorded when it succeeds. When reporting effect percentages, TraceProbe excludes reasoning-only steps because they do not change files or return command status. These labels describe what happened during the run, not whether the action was correct.

File-level anchors can still be too coarse when two runs touch the same file but edit different code regions. For function-level analysis, TraceProbe deterministically maps each patch hunk to the most-specific enclosing def or class, using the hunk header and an indentation-aware scan without an LLM. When the resolver cannot identify an enclosing function, the hunk remains a valid file-level anchor.

The semantic layer is separate from this deterministic core. For detectors marked semantic, the step labeler prompts an LLM to choose one phase/action pair from the closed taxonomy in Table 1. The parser validates each response and derives the phase from a valid action when needed. Because these tags are produced by an LLM, they may change if the labeling model, prompt, or provider behavior changes. We therefore use them only for detectors marked semantic and for exploratory analyses, not for the main deterministic claims.

3.3 Anti-Patterns: Single-Trajectory Diagnosis↩︎

In Insight, an anti-pattern detector is a deterministic predicate over one trajectory. If the predicate is satisfied, Insight reports a named pattern together with the local steps that support it. To keep this evidence auditable, the reported catalog is intentionally selective rather than exhaustive. A detector is included only when its triggering condition can be stated exactly, its inputs are observable tool-event features, and its numeric threshold is frozen before evaluation.

Table 2 gives the exact predicates, thresholds, and inputs that support claims in this work. The first block contains the single-trajectory detectors used by Insight. For traceability, the same table also lists the frozen cross-trajectory patterns and milestone definitions introduced in the next subsections. Structural detectors support the main single-trajectory claims, while semantic detectors remain exploratory. Ambiguous patterns such as rewrites move to cross-trajectory comparison, where a reference can disambiguate them.

3pt 0pt

3.4 Divergence Patterns: Cross-Trajectory Diagnosis↩︎

Some process patterns are ambiguous without another run for comparison. Converge handles these cases by asking a reference-scoped question: how does one agent deviate from a reference process? The study design selects the reference trajectory, for example a resolved run for the same task or a contrast that holds either the scaffold or the model fixed. The compared trajectory is the run being diagnosed. Converge aligns the two trajectories and classifies the unmatched or reordered regions as named forms of avoidable effort, instability, or drift.

Converge begins with a monotonic longest-common-subsequence (LCS) alignment between a reference \(R\) and compared trajectory \(C\). Two actions can match only when their action types, targets, and effect labels are compatible. Compatibility is action-specific. File actions compare repo-relative paths, while non-file actions use normalized search scope, command intent, or task intent. Effect labels are matched conservatively: successful workspace effects can match each other, failure-like and non-workspace effects require the same label, and off-anchor actions are treated permissively so uncertain exploration does not inflate divergence. These rules preserve process differences such as wrong-file work, reversed writes, and failed actions where the reference succeeded, while avoiding over-counting equivalent local choices.

From the matched pairs and unmatched actions, Converge derives coverage, cost, and ordering metrics that summarize omitted reference behavior, added work, and out-of-order matches. This alignment is deterministic and runs in \(O(nm)\). To check whether classifications depend on the unit-cost LCS choice, we also compare against a weighted Needleman–Wunsch variant. After alignment, Converge groups unmatched or reordered actions into contiguous divergence spans. Each span is scored by a few observable metrics, such as off-anchor ratio, reversed or rapid rewrites, and dead-end or out-of-order counts. Each metric has frozen good and warn thresholds. A value past the warn threshold is high-tier evidence, and a value in the good-to-warn band is medium-tier.

Converge summarizes evidence from divergence spans in three diagnostic layers: file selection, edit stability, and completion behavior. The file-selection layer asks whether the compared run explored or changed a different part of the repository from the reference. The edit-stability layer captures unstable rewrites and repeated failed actions, including error-spiral counts from the structural detector catalog. The completion-behavior layer captures unresolved dead ends and actions that reach the same targets in an inefficient order. Within each layer, Converge combines pattern counts with evidence tiers and reports a strong, moderate, or weak classification. These labels describe agreement with the reference in that layer, not absolute run quality. A strong classification means there is no medium- or high-tier divergence evidence. A moderate classification means medium-tier evidence, and a weak classification means high-tier evidence or multiple medium-tier patterns. All classification thresholds are fixed before the reported analyses, and every classification remains relative to the chosen reference. To assess reference dependence, we repeat the analysis with a per-task most-efficient reference, defined as the resolved candidate reference with the fewest trajectory steps.

3.5 Task-Relevant Progress Milestones↩︎

Anti-patterns and divergence patterns describe what a trajectory contains and how it differs from another run. Milestones add a temporal view by recording when the agent first makes task-relevant progress. A milestone is the earliest step at which a specified event occurs with respect to the benchmark’s anchor set. TraceProbe extracts five milestones from each trajectory: M1 first read of an anchor file, M2 first write to an anchor file, M3 all anchor files written, M4 first passing validation command, M5 first justified action. Each milestone yields a (reached, step-to-first-reach) pair, and trajectories that never reach the event are right-censored and reported separately. Because milestones require the anchor set, they are oracle-grounded. They still operate on one trajectory at a time, so the empirical study uses them as a temporal process summary alongside anti-patterns and divergence patterns.

4 Empirical Setup↩︎

4.1 Corpora and Collection Protocol↩︎

We collect the trajectory corpus from all 500 SWE-Bench Verified [13], [59] tasks. Each task is run under five production settings. We define a setting as a scaffold plus a model: Claude Code + Opus 4.6, Codex + GPT-5.4, and OpenCode with Opus 4.6, GPT-5.4, and GLM-5. This yields 2,500 trajectories in total. This design supports two contrast families. The three OpenCode variants form same-scaffold contrasts where the model changes. Claude Code vs.OpenCode on Opus and Codex vs.OpenCode on GPT-5.4 form same-model contrasts where the scaffold changes. To measure run-to-run variability, we repeat a 50-task stratified random subsample three independent times under each setting, producing 750 additional trajectories. The sample is grouped by how many of the five settings resolve each task: 20 easy, 20 medium, and 10 hard tasks. This repeated-run check estimates within-task variance for each process-profile metric and keeps the main results from depending on single-trace attribution. For cross-benchmark transfer, we also use all 266 SWE-Bench Pro [17] Python tasks, collecting all 1,330 task-setting runs. We use SWE-Bench Pro to check whether the main associations and scaffold patterns carry to a harder same-family benchmark. All settings receive the same issue-only prompt and use CLI-default inference parameters. Each task runs once in an isolated working directory at the task’s base commit. Failed, timed-out, and zero-patch runs are retained rather than filtered. We use GPT-5.4 as the semantic step labeler over the closed taxonomy in Table 1.

4.2 Baselines and Controls↩︎

Because our focus is process diagnosis rather than task solving, we compare against baselines that use the same runs and the same outcome labels. The outcome-only baseline is resolve rate alone. The raw-telemetry baseline uses steps, tokens, durations, failed tool calls, and edit counts. We also use same-outcome and same-task controls, including resolved-vs-resolved pairs and frozen-threshold reuse. Together, these baselines separate trajectory-specific signal from outcome differences, generic run cost, and task difficulty.

5 Results↩︎

The empirical study asks what trajectory structure adds beyond success/failure outcomes. We organize the evaluation from single-run signals to controlled cross-run comparisons. RQ1 asks which anti-patterns act as failure clues, how same-task comparisons localize divergence, and whether the strongest detector signal transfers after thresholds are frozen. RQ2 asks whether deterministic effect labels explain process differences beyond raw telemetry such as steps and failed tool calls. RQ3 asks whether agent settings with similar resolve rates differ in process profiles and task-relevant progress timing. RQ4 asks whether reference-scoped cross-run classifications are stable under reference changes and whether same-model controls reveal scaffold-driven process differences.

5.1 RQ1: Controlled Failure Evidence↩︎

2pt

We use detector prevalence as a first pass over the catalog. Because each task appears under multiple settings, the aggregate comparisons in Table ¿tbl:tab:rq1-prevalence? are descriptive rather than causal. This screen identifies search loop as the clearest failure clue because it is common and associated with failed runs. Re-read churn provides a secondary but less stable clue, while tool oscillation is weaker. The difficulty columns qualify this interpretation. Many detectors become more common as tasks get harder, so the resolved–failed gap can reflect residual task difficulty rather than a failure-specific process. Other rows reinforce the same caution. Shell-over-tool mainly reflects scaffold convention, and no formal tail validation occurs more often in resolved runs. The catalog therefore provides deterministic pattern definitions and step-level evidence for inspection, rather than a list of failure predictors.

The same limitation appears in a simpler raw-telemetry baseline. Figure 3 shows that failed runs usually shift toward more steps within each setting, but the resolved and failed distributions still overlap. Harder SWE-Bench Verified tasks also have longer runs and a higher failure share. Step count can therefore spot expensive runs at the population level, but it does not explain why a particular run failed.

5pt

Same-task divergence. Detector prevalence and step counts can show that runs differ, but they do not show where the difference occurs. We therefore align each failed run against a resolved run of the same task. For each diagnostic layer, we report the percentage of alignments where Converge rates agreement with the reference as moderate or weak. File selection measures whether the compared run searched, read, or wrote in a different file scope from the reference. Edit stability measures unstable rewrites and repeated failed actions. Completion behavior measures unresolved dead ends and inefficient action ordering near the end of the run, while function selection applies the same comparison at the enclosing-function level when anchors are available. Across the failed/resolved contrasts in Table 3, file selection is usually the largest or tied-largest of the three layers, and completion behavior is also substantial. The useful signal is not simply that a failed run did more work, but where its work departed from a successful run on the same task. This points inspection first toward file scope, then toward completion behavior and edit stability.

Function-level localization. The same-task result motivates a finer target view. File-level divergence is broad because agents can inspect different files before reaching a valid solution. We therefore resolve each benchmark patch hunk to its enclosing function and ask whether the failed run reached the same code region as a resolved run for the same task. Figure 4 shows when the first function-level divergence appears in the failed run. Across contrasts, the median lead time is around 20 steps before the failed run ends. This means function-level divergence is available early enough to guide inspection, but it does not prove why the run failed. The distinction is useful because failures do not all have the same shape. Some runs work in the wrong file scope, some reach the right file but miss the relevant function, and some reach the right function but still make the wrong edit. A file-level view collapses these cases, while function-level analysis separates scope recovery from patch reasoning or validation failure.

2pt

Cross-benchmark transfer. Finally, we test whether the frozen detector catalog carries beyond SWE-Bench Verified. We apply it unchanged to all Python tasks in SWE-Bench Pro. In Table 4, the prevalence column shows how often a detector is observed. The outcome link column is directional: values below 1 mean the detector appears more in failed runs, and values above 1 mean it appears more in resolved runs. Search loop is the most stable transfer signal. It appears at different rates across the two benchmarks, but its outcome-link value stays below 1 in both, meaning it remains more common in failed runs. Re-read churn keeps the same direction but moves closer to 1 on Pro, so its association with failure weakens. Several other detectors move toward 1 or change direction, which means their prevalence is more benchmark-specific. The useful insight is that detector definitions can transfer, but detector thresholds should be audited on the target benchmark. Search loop remains a portable warning sign, while the shifted prevalence rates show why practitioners should not reuse thresholds blindly across benchmarks.

5.2 RQ2: Effect Labels as Diagnostics↩︎

This RQ asks whether deterministic effect labels reveal process differences that raw telemetry cannot show. Raw counts report how many steps, tool calls, or failures occurred. Effect labels add a different view: they record whether actions produced surviving changes, failed, were reverted, were justified by task-relevant evidence, or moved outside the anchor set. We therefore use Figure 5 to compare how agent settings distribute their work across these outcomes.

Effect labels give each setting a compact process profile. From the figure (per-setting values in Table ¿tbl:tab:rq3-agents?), Codex has the largest survived-action share, OpenCode/GPT-5.4 has the largest failed-action share, and OpenCode/Opus 4.6 has the lowest off-anchor share. These differences show that settings can spend their actions in different ways even when raw counts or final outcomes look similar. The profile yields three practical insights. The survived-action share shows how much work becomes durable code change, the failed and reverted shares show how much effort is lost, and the off-anchor share shows how much exploration falls outside the benchmark patch scope. These signals make effect labels useful for comparing settings and selecting runs for inspection. They should not be read as standalone explanations for why a specific run passed or failed.

5.3 RQ3: Setting-Level Process Differences↩︎

3pt

RQ3 asks whether agent settings with similar resolve rates still behave differently. Table ¿tbl:tab:rq3-agents? summarizes each setting with process metrics and oracle-grounded milestone reach. Most process rows are oracle-free action or effect metrics. The off-anchor row and milestone rows use the gold-patch anchor set, so they are interpreted only for benchmark tasks with anchors. Pairwise comparisons are paired by task, and attribution claims are limited to same-scaffold OpenCode or same-model contrasts.

The process rows show why resolve rate alone is incomplete. Codex has the highest resolve rate and the largest survived-action share, but it also uses many tool calls. OpenCode/GPT-5.4 has the fewest steps, yet it has the highest failed-action share and harmful ratio, which is the share of a run’s non-reasoning actions that failed or were reverted. OpenCode/Opus 4.6 has the lowest off-anchor share and the highest justified-action share. These differences show that settings with similar resolve rates can still differ in durable edits, failed work, and task-scope focus. For practitioners, this profile helps identify what changed after a model or scaffold update, even when the final resolve rate changes little.

Milestones add timing information that the aggregate process rows cannot show (Figure 6). In raw steps, OpenCode/Opus reaches the first anchor write earliest, while Claude Code and Codex reach it later. After normalizing by run length, OpenCode/GLM moves earlier because it reaches task-relevant files early within longer runs. This contrast separates quick progress from short runs. An agent can take more steps overall while still reaching relevant code early in its own trajectory. Milestones therefore measure progress timing, not patch correctness.

Taken together, the process profiles and milestones give practitioners two complementary checks. The profile rows show what kind of work a setting performs, while milestones show when the setting reaches task-relevant code. This makes RQ3 useful for comparing model or scaffold updates even when resolve rate changes little.

5.4 RQ4: Reference-Scoped Comparisons↩︎

2pt

RQ4 checks how Converge behaves when the comparison reference changes and when the model is held fixed while the scaffold changes. The GPT-5.4 contrast compares Codex-GPT with OpenCode-GPT, and the Opus 4.6 contrast compares Claude Code-Opus with OpenCode-Opus. These same-model contrasts let us ask whether scaffold-driven process differences are visible beyond resolve rate.

The middle column of Table 5 is the reference-change check. Converge aligns a compared run against a resolved reference run for the same task and assigns strong, moderate, or weak agreement for each diagnostic dimension. Because a task may have multiple resolved runs, we recompute the comparison with the fixed Codex reference and with the per-task most-efficient resolved reference. Reference change is the percentage of comparisons where that agreement category changes. File selection changes in only 0.69% of comparisons, so file-scope evidence is almost insensitive to reference choice. Edit stability changes in 22.1% because recovery and rewrite behavior can vary even among resolved runs. Completion behavior falls between them at 8.7%. Thus, reference-scoped comparison is robust for file selection, while edit stability should be read with more reference sensitivity.

The outer columns show that scaffold effects depend on the model being held fixed. Under GPT-5.4, the two scaffolds diverge strongly, especially in file selection. Under Opus 4.6, the same kind of scaffold contrast produces much less divergence in every diagnostic dimension. This means that a scaffold comparison is not a constant property of the scaffold pair. Holding the model fixed exposes process differences that endpoint resolve rates can hide. At the same time, the result is not an absolute quality ranking of the scaffolds; it describes how a compared run departs from a specified reference.

Cross-benchmark scaffold control. SWE-Bench Pro provides a further same-model scaffold check. Claude Code and OpenCode share Opus 4.6 and have similar resolve rates on this corpus, yet their detector patterns differ. Across the 266 shared Pro tasks, structured plan absence is higher for Claude Code (49.6% vs 0.4%) and re-read churn is higher for OpenCode (43.6% vs 16.9%). The difference is therefore not simply that one scaffold records more of every detector pattern. This result reinforces the RQ1 caution that a prevalence shift can be a scaffold signature unless it is checked against outcome, same-task, or same-model evidence. The Pro scaffold control therefore complements the detector-transfer result. The same frozen detector signals can reveal both outcome association and process shifts, but the interpretation depends on the contrast being analyzed.

6 Discussion and Threats to Validity↩︎

Discussion. This study shows why process diagnosis should be reported alongside outcome evaluation. Two runs can share the same success/failure outcome while following different processes, so a TraceProbe report should be read as inspection evidence rather than automatic remediation advice. To expose these differences, Insight identifies recurring anti-patterns that help prioritize runs and task families for review. For cross-run comparison, Converge localizes divergence against a specified reference, while effect labels and milestones describe what work was performed and when task-relevant code was reached. Taken together, these views provide the strongest evidence through same-task localization at file and function granularity, the stable search-loop clue, and the deterministic representation that makes comparisons repeatable. At the same time, the controls narrow the claim. Several aggregate signals weaken under task-fixed analysis, and the transfer check preserves only part of the detector pattern. For practical use, process profiles should accompany resolve rate. Practitioners should prefer controlled contrasts over aggregate prevalence shifts, and detector thresholds should be audited on the target benchmark before conclusions are reused. Under this scope, TraceProbe supports inspection and comparison, but it is not a replacement for resolve-rate evaluation, a per-run causal oracle, or a remediation method.

Threats to validity. The main calibration threat is benchmark-specific detector tuning. Several detector thresholds were chosen on SWE-Bench Verified. The Verified split and raw-threshold sensitivity check assess stability within that corpus after thresholds are fixed, while the SWE-Bench Pro check asks whether the strongest direction carries to a disjoint benchmark. These checks support stability after thresholds are fixed and bounded transfer, not independent threshold discovery. However, they do not show that the same thresholds will produce the same prevalence rates or conclusions on every benchmark. TraceProbe is general because new trajectories can be normalized, labeled, and compared through the same pipeline, but benchmark-specific claims still require benchmark-specific threshold audits. Beyond calibration, sampling and construct limits also shape the claims. Each main-corpus task/setting pair is run once, outcome labels inherit SWE-Bench test-suite limits, and Converge agreement assessments measure reference-relative divergence rather than absolute quality or human-validated labels. Both benchmarks are Python issue to PR repair, so results may not transfer to other languages or task types, and we do not yet evaluate whether the diagnostics change practitioner outcomes. Execution measurements add another limitation because they can depend on the provider and scaffold. Duration, token counts, and monetary cost can change with CLI versions, provider-side batching, tokenizer accounting, rate limits, tool permissions, and budget-stop reporting. For that reason, we interpret them as observed process-cost measurements in this collection, not as intrinsic model speed or efficiency.

7 Conclusion↩︎

Success/failure outcomes are necessary for evaluating coding agents, but they do not explain the process that produced an accepted or failed run. This work addresses that gap by treating coding agent trajectories as structured process evidence. TraceProbe normalizes raw traces into canonical actions and deterministic effect labels, uses Insight to identify recurring single-trajectory anti-patterns, and uses Converge to compare runs against explicit references. In our SWE-Bench Verified study and SWE-Bench Pro transfer check, TraceProbe identifies population-level failure clues, localizes where failed and resolved runs diverge, and shows that resolved runs can still differ in progress timing, failed work, and observed process cost. These findings support inspection and comparison, while the controls keep the claims short of explaining exactly why a particular run failed or automatically fixing it. Future directions include pairing these deterministic diagnostics with human-validated root-cause analysis and intervention studies. Those studies can test whether changes to prompts, retrieval, memory, tool policy, or validation strategy improve agent behavior while preserving controlled comparisons and keeping deterministic evidence separate from human judgment.

References↩︎

---

1. Preprint. Under review.↩︎