ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
January 01, 1970
Abstract
Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy’s own task-solving or a fixed multi-agent system’s emergent behavior, but none isolate the management ability of the single LLM acting as leader. We introduce ClawArena-Team, a benchmark of 41 multi-turn, multimodal, multi-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace. It commands a fixed, locally served subagent pool, so score differences reflect management skill, not raw capability. All scoring is execution-based with no LLM judge: an overall score—the Subagent-Management Score (Sms)—multiplies task correctness by a least-privilege and modality-routing factor. Across twelve proprietary, community-hosted, and self-hosted models, experiments show that the management bottleneck is privilege granting rather than perception (no model exceeds \(50\%\) workspace-permission precision); that cost and management quality are decoupled (API cost spans over \(100\times\) while the overall score spans under \(4\times\), with the cheapest open models on the Pareto frontier); and that most leaderboard scores cluster within a \(9.9\)-point band while orchestration behaviors diverge by more than an order of magnitude. Code and data will be released.
1 Introduction↩︎
Production LM agents are increasingly deployed not as lone problem-solvers but as managers: a single main model creates specialized subagents, delegates work to them, and orchestrates their parallel and asynchronous returns through dynamic, programmable workflows [1], [2]. Yet whether one model can actually run such a team is largely unmeasured. Management is a structured skill with three coupled requirements. The manager must route each task piece to the right specialist by modality and capability, so an image-grounded question reaches a vision worker rather than being attempted on the text it cannot see (modality routing). The manager must grant each subagent only the tools and workspace paths it needs, so over-granting does not expand the blast radius of a misbehaving worker or inflate context (least-privilege empowerment). The manager must schedule subagents concurrently, in the background, or as continued sessions, and integrate their returns into a correct deliverable rather than merely relay them (dynamic orchestration). A manager that routes correctly but over-grants, or grants tightly but cannot schedule concurrently, still produces unsafe, expensive, or wrong outcomes.
Existing benchmarks test fragments of this setting but not the full management capability. Single-agent benchmarks score a policy’s own reasoning, tool use, and policy compliance [3]–[8]. Multi-agent frameworks supply orchestration mechanisms but are validated as systems with predefined roles or peer dialogue rather than as benchmarks of a single manager [9]–[11]. Permission and least-privilege tooling formalizes over-granting as an external enforcement layer that the (often single) tool-calling agent is not trusted to perform itself, rather than as a measured ability of the agent [12], [13]. Taken together, existing benchmarks either fix the team and score its emergent behavior [14] or score a manager over a pre-given workflow set [15], leaving open the question of whether a single LM can create, empower, and orchestrate its own team from scratch.
We introduce ClawArena-Team(Workflow Orchestration variant in ClawArena series), a benchmark that isolates the subagent-management ability of a single LM acting as the main agent (Figure 1). ClawArena-Team frames the setting as a principal–agent problem (§3.2): the main agent (principal) must accomplish multi-turn tasks it cannot complete alone by creating, empowering, scheduling, and integrating subagents (agents). Three design choices make the score reflect management rather than raw capability: the main agent natively perceives only text and reaches part of the workspace only through subagents, so delegation is mandatory; the subagent pool is held fixed and locally served, so every manager commands the same workers and only the manager varies across runs; and all scoring is execution-based with no LLM judge. The benchmark comprises 41 multimodal, multi-directory scenarios spanning 258 evaluation rounds with 72 staged updates that change subsequent answers, with a composite Subagent-Management Score (Sms) that multiplies task correctness by a least-privilege and modality-routing management factor.
In summary, our primary contribution is ClawArena-Team, a controlled-comparison benchmark framework that isolates the subagent-management ability of a single LM main agent through a fixed locally-served subagent pool, fine-grained per-subagent management metrics, and execution-based scoring with no LLM judge. On the 41-scenario benchmark, evaluations across twelve proprietary, community-hosted, and self-hosted main-agent models show that the management bottleneck is privilege granting rather than perception (no model exceeds \(50\%\) workspace-permission precision), that cost and management quality are decoupled (API cost spans over \(100\times\) while Sms spans under \(4\times\), with the cheapest open models on the Pareto frontier), and that leaderboard scores cluster within a \(9.9\)-point band while orchestration behaviors diverge by more than an order of magnitude (subagent forbidden-access rates differ by roughly \(12\times\) among capable models).
2 Related Work↩︎
2.0.0.1 Agent benchmarks.
Single-agent benchmarks such as SWE-bench [3], GAIA [4], AgentBench [5], \(\tau\)-bench [6], OSWorld [7], AgentBoard [16], ToolLLM [17], and WebArena [18] score a single policy’s reasoning, tool use, and multi-turn interaction; ST-WebAgentBench [8] adds safety and policy compliance, but the unit of evaluation remains a single agent. Multi-agent frameworks such as AutoGen [9], CAMEL [10], MetaGPT [11], OpenAI Swarm [19], AgentOrchestra [20], and DynTaskMAS [21] supply orchestration mechanisms but are validated as systems with predefined roles or peer dialogue rather than runtime single-manager control. Neither group introduces a main-agent/subagent relationship that lets a single LM be scored on its management of a team.
2.0.0.2 Agent-as-manager and the closest precedents.
The Manager Agent challenge [15] is the only work that genuinely evaluates a manager orchestrating a team at runtime (the MA-Gym simulator, a GPT-5 manager, graph-editing actions). It differs from ClawArena-Team in three ways: its workflows are given task-dependency graphs that the manager edits rather than creates from scratch; it dispatches to a pre-existing pool of AI and simulated-human workers rather than creating subagents at runtime; and its five metrics (preference, constraint, goal, stakeholder, runtime) cover multi-objective optimization but not least-privilege empowerment, modality-based expert selection, or asynchronous and background scheduling. MultiAgentBench [14] evaluates the emergent collaboration and competition of a multi-agent system under fixed, predefined topologies, not the delegation ability of a single manager. Collaborative Gym [22] and TheAgentCompany [23] target human–agent collaboration and realistic workplace tasks, and analyses of multi-agent failure modes [24] motivate better orchestration evaluation. We position ClawArena-Team against these by holding the team fixed and varying only the manager, so that score differences isolate management skill rather than the underlying capability of the workers.
2.0.0.3 Management primitives in isolation.
Two of the management dimensions ClawArena-Team scores have been studied outside the management context. Progent [12], MiniScope [13], and the mandatory-access-control framework of [25] formalize over-privileging and multi-agent confused-deputy risks, but treat least privilege as an external enforcement layer that the (often single) tool-calling agent is not trusted to perform itself, rather than as a measured ability of the agent. LM routers such as RouterBench [26] and RouteLLM [27] route queries by capability or cost tier rather than by modality; MEXA [28] selects modality-task experts only as an internal step scored by end-task accuracy, and MLLM-Tool [29] scores modality-aware selection at the granularity of an API rather than a modality-specialist subagent. None scores per-decision routing to expert subagents or self-granted least privilege as a first-class, execution-checked management dimension, and none brings these dimensions together under a single manager, which is the gap ClawArena-Team fills (Table ¿tbl:tab:related?).
4.5pt
@l ccccccc@ & Single-LM & Runtime & Modality & Least- & Async/bg & Dynamic & Exec.
Benchmark & manager & subagent & routing & privilege & scheduling & workflow & scoring
SWE-bench [3] & & & & & & &
GAIA [4] & & & & & & &
\(\tau\)-bench [6] & & & & & & &
ST-WebAgentBench [8] & & & & & & &
MultiAgentBench [14] & & & & & & &
Manager Agent [15] & & & & & & &
Progent / MiniScope [12], [13] & & & & & &
&
RouterBench / MEXA [26], [28] & & & & & &
&
ClawArena [30] & & & & & & &
(ours) & & & & & & &
3 The ClawArena-Team Benchmark↩︎
3.1 Overview↩︎
ClawArena-Team is a controlled-comparison benchmark for subagent management: every manager commands the same fixed, locally-served subagent pool, so score differences reflect management quality rather than worker quality.
The pool serves three model keys via local vLLM, with llm and vlm backed by gemma-4-31b-it and omni by gemma-4-e4b-it. The benchmark
comprises 41 scenarios across law, medicine, engineering, business, and science, totaling 258 multi-turn evaluation rounds. Each scenario is a workspace task where the main agent completes a series of user tasks, and between some rounds the workspace
receives staged updates (new or replaced files) that change subsequent answers. Workspaces are large (170.5 MiB; 28.9 M tokens, \(71.9\%\) workspace content and \(27.9\%\) updates)
and heterogeneous (text, code, office documents, images, audio, video) across at least eight top-level directories, some of which are decoys. What “managing” such a scenario actually requires of the main agent \(M\) is
formalized next as \(M\)’s capability surface.
3.2 Capability surface↩︎
We formalize subagent management as a principal–agent problem under information and capability asymmetry: a main agent \(M\) (the principal) faces a multi-turn task whose information and modalities exceed what it can directly consume, and must accomplish it by managing a set of subagents \(\{a_i\}\) (the agents) instantiated from the same pool. \(M\)’s capability surface decomposes into six concrete operations:
Creation. \(M\) creates a subagent with a system prompt, a model key, a tool subset, and a workspace path whitelist (a subset of its own).
Modality routing. Because \(M\) natively perceives only text, images and video must be routed to a vlm subagent and audio to an omni.
Least-privilege empowerment. Tool and path grants should match what the subagent actually needs; over-granting is wasteful and unsafe.
Scheduling. Subagents can run in the foreground or background, as new sessions or continued (resumed) sessions, and in parallel; background tasks notify \(M\) on completion.
Dynamic workflows. A programmatic workflow tool lets \(M\) author multi-subagent orchestration (parallel and pipeline stages) at runtime over the same pool.
Integration. \(M\) must fuse subagent returns into a correct deliverable rather than relay them; only the integrated answer is scored, so a faithful relay of correct subagent reports can still fail.
How every scenario forces these operations to be exercised, rather than routed around, is the subject of the next subsection.
3.3 Tasks, staged updates, and design criteria↩︎
A scenario provides (i) a workspace, (ii) a sequence of user questions, and (iii) ground-truth checks. Of the 258 rounds, 44 (\(17.1\%\)) are preceded by staged updates (72 update groups, 255 files); later rounds depend on earlier subagents’ outputs, requiring sustained management of an evolving task rather than a one-shot solution. Every scenario satisfies ten hard constraints (Table 1) that jointly force genuine management; each constraint targets a distinct management failure mode that a single LM could otherwise route around (skipping delegation, ignoring an update).
| ID | Constraint | What it forces |
|---|---|---|
| C1 | Unreadable regions | Delegation and summarization rather than direct read. |
| C2 | Substantive staged updates | Answer revision across rounds, not append-only memory. |
| C3 | Full tool surface used | Incremental edits and shell verification, not read-only. |
| C4 | \(\geq 8\) directories with decoys | Permission restraint on irrelevant paths. |
| C5 | Non-text modality rounds | Mandatory delegation to a vlm or omni subagent. |
| C6 | Parallel-required tasks | Concurrent subagent scheduling, not serial fan-out. |
| C7 | Session-reuse pairs | New vs.continued session decision in the same scenario. |
| C8 | Cross-round dependencies | Sustained state management across an evolving task. |
| C9 | Modality decoys | Resistance to plausible-but-wrong text shortcuts. |
| C10 | Machine-checkable answers | Execution-based verification, no LLM judge. |
6pt
Several of these constraints, C5 (modality grounding) and C9 (modality decoys) in particular, require workspace anchors that text cannot reproduce. Images include tables, forms, diagrams, and medical regions of interest; audio includes single- and multi-speaker recordings, with a voice-profile matrix to defeat single-timbre shortcuts; video covers algorithm visualizations, screen recordings, and data animations. The workspaces also mix many text formats (CSV, YAML, JSON, SQL, logs, email, calendars) with decoy directories that may be directly accessible to \(M\) yet are irrelevant, testing whether \(M\) withholds them from subagents (C4). Hand-assembling 41 scenarios that each satisfy all ten constraints is not viable; we synthesize them programmatically, as described next.
3.4 Construction↩︎
We synthesize ClawArena-Team end-to-end. Workspace corpora, multimodal assets, ground-truth values, and execution checks all come from the same version-controlled scripts, so corpus and answer are emitted together and the benchmark is regenerable from source (Appendix 12). Two ideas organize the methodology. Workflow-driven parallel authoring decomposes a batch into per-scenario contracts that a controller fans out to isolated subagent-authors and reassembles behind objective gates. An end-to-end test flywheel adds a real baseline run of the subagent-management harness as the final acceptance gate; the run is scored by the rule formalized in §3.5. A scenario is finished only after this baseline run leaves every failure classified as true-difficulty (modality decoy taken, fabricated identifier, over-grant) rather than false-kill (brittle regex, order-locked field). The construction pipeline is itself an instance of subagent management: a single controller that decomposes work, selects model tiers, confines parallel authors under least-privilege contracts, and integrates their deliverables behind gates. ClawArena-Team is in effect authored by the capability it measures.
3.5 Scoring↩︎
That scoring rule is execution-based, following a “verify the result, not the method” principle: each round ships a shell command whose exit code (with optional output matching) determines pass/fail. No LLM judge is used, which avoids both rejecting legitimate alternative solutions and judge drift. We report five per-run components and one composite, all in \([0,1]\):
TCR (Task Completion Rate): mean pass rate over user questions.
TPP (Tool-Permission Precision): per subagent, the fraction of granted tool types that are actually used.
ROC (Read-Only Compliance): per subagent, \(0\) if a read-only subagent was granted a mutating tool, else \(1\).
WPP (Workspace-Permission Precision): per subagent, accessed files divided by granted files.
MCA (Modality-Choice Accuracy): per subagent, whether a vlm actually reads image/video and an omni reads audio (llm scores \(1\)).
The composite Subagent-Management Score is \[\mathrm{\small Sms}\;=\; \mathrm{TCR}\times\frac{\mathrm{TPP}+\mathrm{ROC}+\mathrm{WPP}+\mathrm{MCA}}{4}. \label{eq:sms}\tag{1}\] Because the management factor lies in \([0,1]\), Sms \(\leq\) TCR always: management can only discount task correctness, never inflate it. A model that produces correct work while managing sloppily is penalized, and a model that manages perfectly but fails the task scores zero, encoding that good management without correct deliverables is worthless.
| Model | TCR | TPP | ROC | WPP | MCA | Cost ($) | Rounds | |
|---|---|---|---|---|---|---|---|---|
| Proprietary | ||||||||
| claude-fable-5 | 74.4 | 76.4 | 98.8 | 49.2 | 97.9 | 60.0 | 92.8 | 192/258 |
| gemini-3.5-flash | 69.8 | 69.8 | 95.9 | 45.7 | 96.7 | 53.8 | 23.7 | 180/258 |
| gpt-5.5 | 63.6 | 79.9 | 98.7 | 47.5 | 95.2 | 51.0 | 43.3 | 164/258 |
| gpt-5.4 | 63.2 | 77.3 | 98.8 | 40.8 | 97.6 | 49.7 | 19.5 | 163/258 |
| gemini-3.1-pro | 65.5 | 76.4 | 97.3 | 36.6 | 92.9 | 49.6 | 20.5 | 169/258 |
| claude-sonnet-4-6 | 61.6 | 77.3 | 99.4 | 43.2 | 95.2 | 48.5 | 39.7 | 159/258 |
| Open-weight | ||||||||
| glm-5.2 | 66.3 | 67.4 | 95.4 | 44.4 | 96.7 | 50.4 | 22.9 | 171/258 |
| kimi-k2.6 | 64.3 | 76.8 | 92.0 | 41.9 | 93.9 | 49.0 | 28.7 | 166/258 |
| deepseek-v4-pro | 58.9 | 73.9 | 98.5 | 46.3 | 96.5 | 46.4 | 1.7 | 152/258 |
| qwen3.6-27b | 60.9 | 72.5 | 95.7 | 40.7 | 96.2 | 46.4 | 15.0 | 157/258 |
| gemma-4-31b | 56.6 | 79.0 | 97.7 | 37.9 | 95.5 | 43.9 | 3.5 | 146/258 |
| glm-4.7-flash | 34.5 | 22.4 | 85.8 | 11.7 | 57.2 | 15.3 | 0.8 | 89/258 |
4pt
4 Experiments↩︎
To isolate the main agent’s own management ability, we hold the subagent pool and the execution-based scoring fixed and vary only the manager model. We evaluate twelve main-agent models on the full 41-scenario ClawArena-Team benchmark in a single evaluation run, then analyze it through four lenses. (1) Is the benchmark discriminative across models, and does it leave room above current flagships? (2) Within the six-operation capability surface (§3.2), where does the management bottleneck actually lie? (3) Does main-agent API cost track management quality, or do cheap models reach the same management score? (4) When leaderboard scores cluster, do underlying orchestration behaviors cluster too, or do they diverge? The leaderboard (§4.2) answers (1); the three findings that follow within §4.2 answer (2), (3), and (4) in turn.
4.1 Setup↩︎
Main-agent models. We evaluate twelve main-agent models over all 41 scenarios. Proprietary models include Anthropic claude-fable-5 and claude-sonnet-4-6, gemini-3.5-flash and
gemini-3.1-pro, and two gpt-5 versions accessed through codex. Community-hosted open-weight models include kimi-k2.6 and deepseek-v4-pro via OpenRouter, and glm-5.2 via the
official z.ai API. Self-hosted open models served through local vLLM include qwen3.6-27b, gemma-4-31b, and glm-4.7-flash.
Subagent pool and scoring. Every run uses the same fixed subagent pool (§3) and the same execution-based scoring (§3.5). Per-turn generation is capped to bound runaway
output, and the main-agent context budget is advisory rather than hard (Appendix 9). API cost is computed from token usage at provider or OpenRouter list prices (Appendix 11); self-hosted
models are priced at their OpenRouter reference rates for cross-model comparability. claude-fable-5 is evaluated as shipped, with a vendor-recommended refusal \(\rightarrow\) claude-opus-4-8 fallback [31]; refusal-handling details are in Appendix 9.
Evaluation protocol. Each scenario is a sequence of multi-turn rounds, and a round is one complete interaction in which the manager may issue anywhere from a single subagent call to several dozen; a complete run over the benchmark (41 scenarios, 258 rounds, 44 of them preceded by staged updates) is therefore substantially costlier than a single-inference benchmark. All twelve models are evaluated in a single run under identical scenarios, subagent pool, and protocol, so that any difference in the leaderboard is attributable to the manager model alone.
4.2 Main results↩︎
Table 2 reports the leaderboard, grouped into proprietary and open-weight models and sorted by Sms within each group. The new flagship claude-fable-5 leads on Sms(\(60.0\%\)) and TCR (\(74.4\%\)), with gemini-3.5-flash second on Sms(\(53.8\%\)).
The newly released glm-5.2 is the strongest open-weight manager (\(50.4\%\), fourth overall), overtaking kimi-k2.6 (\(49.0\%\)) to top the open-weight group, behind
only three proprietary models. The twelve-model Sms range is \(44.7\) points, but it is concentrated in two gaps: a \(6.2\)-point jump from the runner-up to
the flagship at the top, and a \(28.6\)-point cliff from the bottom of the cluster (gemma-4-31b, \(43.9\%\)) down to glm-4.7-flash (\(15.3\%\)). Between these two endpoints, ten models sit in a tight \(9.9\)-point band (\(43.9\)–\(53.8\%\)). Only the flagship
clears Sms \(=54\%\): every other manager loses roughly half its score to imperfect task correctness and privilege management, and even the leader leaves \(40\%\) on the table.
Together, Table 2, Figures 2–4, and Table 3 answer three questions about manager behavior. First, the management bottleneck is privilege granting rather than perception: the two “easy” axes are nearly saturated but the two privilege-precision axes are not. Second, cost and management quality are decoupled: cost spans over \(100\times\) while Sms spans under \(4\times\), and several high-cost models are dominated by mid-cost ones. Third, leaderboard scores cluster while behaviors diverge: the middle ten models sit in a \(9.9\)-point Sms band but their orchestration behaviors differ by more than an order of magnitude. We unpack each in turn.
4.2.0.1 Finding 1: the management bottleneck is privilege granting.
The two “easy” management axes are nearly saturated: read-only compliance (ROC) is at least \(92\%\) and modality-choice accuracy (MCA) is at least \(92\%\) for all eleven capable models, because both have clear objectives: not granting a mutating tool to a read-only worker, and routing images to the vision model. The discriminating axes are the two privilege-precision metrics. Tool-permission precision (TPP) sits around \(70\)–\(80\%\) and workspace-permission precision (WPP) never reaches \(50\%\) for any model (Table 2): subagents are routinely granted roughly twice the files they touch and more tools than they use. Management is therefore far from solved; the failure concentrates in least-privilege empowerment—both a safety concern (over-broad grants enlarge a misbehaving subagent’s blast radius) and a cost concern (irrelevant paths inflate context).
4.2.0.2 Finding 2: cost and management quality are decoupled.
Figure 2 plots Sms against main-agent API cost (Cost in Table 2). Cost spans over \(100\times\) ($0.8 to $93 per run) while
Sms spans under \(4\times\). The new leader claude-fable-5 is also the most expensive (\(\approx\)$93 per run): it buys the top score, but at
roughly \(4\times\) the cost of gemini-3.5-flash ($23.7) for only \(+6\) Sms points. Below the top, the two decouple sharply.
deepseek-v4-pro reaches Sms \(=46.4\%\) at just $1.7 thanks to a steep cache-read discount, and the costly gpt-5.5 ($43.3), sonnet-4-6 ($39.7), and
kimi-k2.6 ($28.7) are each dominated by gemini-3.5-flash, whereas the open-weight glm-5.2 ($22.9) lands on the frontier, delivering the top open-weight Sms at a quarter
of the flagship’s cost. Cheap open models sit on the frontier; outside the lone flagship, spending more does not buy better management.
4.2.0.3 Finding 3: scores converge while behaviors diverge.
Below the fable-5 flagship, ten models cluster within a \(9.9\)-point Sms band (\(43.9\)–\(53.8\%\), with
glm-4.7-flash far below), but their orchestration behaviors differ by more than an order of magnitude. The subagent forbidden-access rate (mean per subagent) ranges from \(0.48\) (gpt-5.5) to \(5.78\) (deepseek-v4-pro) among capable models and \(11.44\) for glm-4.7-flash (Figure 3). Managers overwhelmingly default to the llm key and under-use vlm and omni (Figure 4): when they route by modality they do it correctly (hence high MCA), but create few
specialist subagents. Dynamic-workflow use ranges from \(8\) to \(112\) invocations, and background scheduling and session continuation are unevenly adopted (Table 3); most models lean on new foreground runs. The weakest model (glm-4.7-flash) both creates the fewest specialists and almost never uses workflows or background scheduling, consistent with its collapse in
Table 2. A single leaderboard number conceals these differences—precisely why ClawArena-Team reports fine-grained management metrics, not task success alone.
| Model | new+fg | cont+fg | new+bg | cont+bg | wf | total |
|---|---|---|---|---|---|---|
| claude-fable-5 | 325 | 56 | 99 | 7 | 94 | 581 |
| gemini-3.5-flash | 439 | 140 | 31 | 5 | 68 | 683 |
| gpt-5.5 | 250 | 23 | 59 | 3 | 48 | 383 |
| glm-5.2 | 383 | 12 | 48 | 2 | 74 | 519 |
| gpt-5.4 | 274 | 23 | 61 | 3 | 13 | 374 |
| gemini-3.1-pro | 295 | 68 | 61 | 9 | 112 | 545 |
| kimi-k2.6 | 319 | 6 | 59 | 0 | 52 | 436 |
| claude-sonnet-4-6 | 212 | 3 | 119 | 2 | 103 | 439 |
| deepseek-v4-pro | 411 | 5 | 60 | 7 | 78 | 561 |
| qwen3.6-27b | 275 | 23 | 50 | 8 | 77 | 433 |
| gemma-4-31b | 389 | 5 | 15 | 0 | 33 | 442 |
| glm-4.7-flash | 152 | 0 | 8 | 0 | 8 | 168 |
4pt
4.3 Error analysis↩︎
Scenario-level scores (Appendix 15, Figure 7) show that difficulty is uneven and model-specific: no scenario is solved by all models, and the full-pass rate (all rounds in a scenario correct) is near zero, indicating that the hard part is sustained management across an evolving, multimodal workspace rather than any single round. We ground these patterns in three case studies (Appendix 17), to which per-scenario transcript evidence is deferred; three recurring failure modes emerge.
Managers take text shortcuts on modality-decoy rounds. Managers collapse a multimodal round into a text shortcut by trusting a plausible-but-wrong text transcript next to the real asset, rather than delegating to a vision or audio specialist. The C9 decoys are designed for this, and Appendix 17 (smart-home anomaly) shows a concrete instance where the wrong humidity reading is taken from a flagged transcript.
Stale beliefs persist across staged updates. Managers fail to propagate a staged update into a later round, either because they advance to the next question before rewriting an earlier deliverable or because they treat the update as new context without re-checking the prior answer. The product-launch warroom case in Appendix 17 captures this pattern: a pending correction is acknowledged in conversation but never reaches the deliverable.
Over-granting does not buy correctness. Managers over-grant workspace paths and tools, but the loose scoping does not improve TCR; subagents simply receive paths they never read and tools they never call. This is the structural signature of Finding 1 in case-level form.
5 Conclusion↩︎
We introduced ClawArena-Team, a benchmark that isolates a single LM’s subagent-management ability by fixing the worker pool and scoring execution outcomes alongside fine-grained per-subagent management metrics. Across twelve models and \(41\) scenarios, the management bottleneck is privilege granting rather than perception (workspace-permission precision never reaches \(50\%\)), and cost and management quality are decoupled (API cost spans over \(100\times\) while Sms spans under \(4\times\), with several mid-cost models dominating higher-priced ones). Behind a clustered \(9.9\)-point Sms band, the middle ten models’ orchestration behaviors (forbidden accesses, workflow adoption, background scheduling) diverge by more than an order of magnitude. A natural extension scales to broader model coverage and stronger subagent pools, tracking management ability as worker capability improves.
Ethical Considerations↩︎
ClawArena-Team uses synthetic scenarios and assets; any resemblance to real entities is incidental. The benchmark surfaces a safety-relevant behavior—over-broad privilege grants by manager agents—which we report to encourage least-privilege design rather than to enable misuse. No human-subjects data is used.
6 Limitations↩︎
Worker-quality coupling. The fixed subagent pool is what makes management the only variable across runs (§3), but it ties our findings to a particular worker capability. A stronger pool could let a sloppier manager succeed by absorbing the slack; a weaker pool could force even careful managers to over-grant or over-route. Both are open questions ClawArena-Team does not currently answer.
Execution-based scoring blind spot. All checks gate on machine-verifiable artifacts. This deliberately rejects LLM-judge drift but also under-rewards correct reasoning that produces an unverifiable deliverable, such as a thoughtful conditional answer that the shell command then deems wrong because it does not match an expected literal form.
Sample size and implementation notes. The current study evaluates twelve main-agent models on 41 scenarios; broader coverage is ongoing and will refresh all tables. claude-fable-5 is evaluated as shipped with the
vendor-recommended refusal\(\rightarrow\)opus-4-8 fallback (§4), so its score is a composite rather than the base model in isolation. Self-hosted-model API costs are imputed from
OpenRouter list rates and are approximate.
7 Data Sample Overview↩︎
Each scenario directory contains the workspace, the question sequence, staged updates, and execution checks. All 258 rounds use execution checks (\(100\%\)), each declaring an expected exit code and a timeout (mean \(82.9\) s) and referencing scenario scripts and the workspace. Updates split into new (\(61.1\%\)) and replace (\(38.9\%\)) groups,
averaging \(3.54\) files per group.
8 Framework Details↩︎
The main agent is exposed CapitalCase tools (Read/Write/Edit/Bash/Grep/Glob) plus subagent-management tools
(CreateSubagent/RunSubagent/ListSubagents/InspectSubagent) and a Workflow tool. System-level signals (environment, token thresholds) are injected via <system-reminder>
blocks and background-task completions via <task-notification> blocks. Read degrades out-of-modality files to a textual placeholder and suggests delegation, so a text-only main agent must route multimodal content to a vlm/omni subagent.
8.0.0.1 Subagent control surface.
CreateSubagent fixes a subagent’s system prompt, model key (llm/vlm/omni), tool subset, and path whitelist (a subset of the main agent’s own
paths); RunSubagent executes it in the foreground or background and may either start a new session or continue an existing one (session reuse). The Workflow tool accepts a compact JavaScript-style DSL with agent(),
parallel(), and pipeline() primitives that drive multiple subagents over the same pool, enabling programmatic parallel and staged orchestration. Read applies soft/hard size thresholds (notice then truncation)
to bound context; all path access is realpath-checked and symlink escapes are rejected and counted as forbidden accesses (the MAF/SAFt statistics).
9 Implementation Details↩︎
The subagent pool is served locally with vLLM and held identical across all evaluated main agents (Table 4). Per-turn generation is capped at \(24{,}000\) tokens to bound runaway generation.
The \(200\)k main-agent context value is an advisory upper bound (not a hard cutoff in the evaluated runs); effective context depends on each model, whereas subagent budgets are uniform across runs by construction. Main
agents are reached through provider-native or OpenAI-compatible APIs (Gemini, OpenRouter, a ChatMock bridge for codex, and a local server for self-hosted models).
| Key | Model | Server | Native modalities |
|---|---|---|---|
| llm | gemma-4-31b-it | local vLLM | text |
| vlm | gemma-4-31b-it | local vLLM | text, image, video |
| omni | gemma-4-e4b-it | local vLLM | text, image, audio, video |
9.0.0.1 claude-fable-5 refusals and fallback.
claude-fable-5 ships with safety classifiers that can decline a request, returning stop_reason: refusal as a successful response; the vendor-recommended pattern is to retry the refused turn on another Claude model [31]. We evaluate fable-5 as shipped, with an automatic refusal \(\rightarrow\) claude-opus-4-8 fallback; \(124\) such fallbacks fired over the run, concentrated in a single security and PII-heavy scenario (security_pcap_triage).
Because that scenario also exposed a scenario-level asyncio deadlock unrelated to fable (it likewise stalls under opus-4-8 but completes under sonnet-4-6), its fable-5 entry was recovered by
running opus-4-8 directly with a per-round timeout, yielding only \(1/6\) rounds and dragging fable-5’s score down on that single scenario. fable-5’s reported cost (Appendix 11) prices all main-agent tokens at fable-5 list rates and is therefore an upper estimate: refused turns are unbilled and fallback credit refunds the prompt-cache cost of switching.
10 Scoring Details↩︎
The management factor weights TPP, ROC, WPP, and MCA equally. ROC and MCA are near-saturated because they encode clear-cut decisions; TPP and WPP carry most of the discriminative signal. We deliberately use no LLM judge: checks gate on exact numeric/string/regex matches or recomputation, and method constraints are applied only when a question explicitly requests a behavior (an instruction-following gate).
10.0.0.1 Correctness variants we omit from the headline.
Two alternative correctness summaries were computed during development but excluded from the main Smsfor parsimony. A per-scenario averaging of TCR (mean of per-scenario completion rates) tracks the headline TCR almost identically, so the two convey the same information; we report only the headline. A full-pass rate (the fraction of scenarios in which every round is correct) is near zero for every evaluated model, which makes it useful as a difficulty signal but uninformative as a per-model headline; we record it in the data release but exclude it from Sms.
11 Pricing Methodology↩︎
Main-agent cost is \(\mathrm{MIT}\cdot p_{\mathrm{cw}} + \mathrm{MCR}\cdot p_{\mathrm{cr}} + \mathrm{MOT}\cdot p_{\mathrm{out}}\), where MIT/MCR/MOT are input/cache-read/output token totals and \(p_{\mathrm{cw}}\) uses the listed cache-write price when one is published—\(1.25\times\) input for claude models (fable-5, opus-4-8,
sonnet-4-6) and the smaller dedicated rates for the two gemini models—and the input price otherwise. Subagents always run on the fixed local pool and are not priced. Prices are provider or OpenRouter list rates (Table 5); because the underlying token counts are post-hoc estimates (Appendix 14), the resulting costs are likewise estimates. claude-fable-5 is priced entirely at its own list rates, so its
cost is an upper estimate (refused turns are unbilled and fallback credit refunds the cache cost of switching to opus-4-8; §4).
| Model | in | out | cache read | cache write | cost ($) |
|---|---|---|---|---|---|
| claude-fable-5 | 10.00 | 50.0 | 1.00 | 12.50 | 92.8 |
| claude-opus-4-8 | 5.00 | 25.0 | 0.50 | 6.25 | 46.4\(^\ddagger\) |
| claude-sonnet-4-6 | 3.00 | 15.0 | 0.30 | 3.75 | 39.7 |
| gemini-3.5-flash | 1.50 | 9.00 | 0.15 | 0.083 | 23.7 |
| gpt-5.5 | 5.00 | 30.0 | 0.50 | — | 43.3 |
| gpt-5.4 | 2.50 | 15.0 | 0.25 | — | 19.5 |
| gemini-3.1-pro | 2.00 | 12.0 | 0.20 | 0.375 | 20.5 |
| kimi-k2.6 | 0.68 | 3.41 | 0.34 | — | 28.7 |
| deepseek-v4-pro | 0.435 | 0.87 | 0.0036 | — | 1.7 |
| qwen3.6-27b | 0.289 | 2.40 | 0.289\(^\dagger\) | — | 15.0 |
| gemma-4-31b | 0.12 | 0.36 | 0.09 | — | 3.5 |
| glm-5.2 | 1.40 | 4.40 | 0.26 | — | 22.9 |
| glm-4.7-flash | 0.06 | 0.40 | 0.01 | — | 0.8 |
12 Benchmark Construction: Workflow-Driven Synthesis under a Test Flywheel↩︎
ClawArena-Team is built rather than collected: every scenario—workspace corpora, multimodal assets, ground truth, and execution checks—is procedurally synthesized. Two ideas organize the methodology. First, scenarios are produced by workflow-driven parallel authoring: a controller decomposes a batch into per-scenario contracts and fans the work out to isolated subagent-authors. Second, no scenario is considered finished until it survives an end-to-end test flywheel in which a real subagent-management run is the final acceptance gate, and its failures are fed back into the data. There is a deliberate reflexivity here: the construction pipeline is itself an instance of subagent management—a single controller that decomposes work, selects model tiers (empowerment), confines parallel authors under least-privilege contracts, and integrates their deliverables behind objective gates. ClawArena-Team is, in effect, authored by the very capability it measures.
12.1 Why full synthesis↩︎
Full synthesis is chosen over scraping for five reasons. Controllable truth: when content is generated, the ground truth is known by construction, so the answer and the corpus can be emitted by the same script. License
cleanliness: no third-party copyrighted assets enter the release. Reproducibility: generation scripts are version-controlled, so a rerun yields byte-identical or near-identical artifacts. Conflict control:
adversarial designs such as an audio recording that contradicts its transcript, or an image whose caption misleads, require the divergence point to be placed with precision—only possible when both sides are authored. Distribution control:
because the corpus is authored rather than found, its composition can be steered to what the benchmark must measure—modality mix, file-format diversity, domain coverage, difficulty-vector incidence, and the capability tags each round exercises—instead of
inheriting whatever a scrape happens to contain; the resulting distribution is characterized in Appendix 13. A consistent resource priority governs asset production: locally served models first (free at inference), then
deterministic local code and tools (Pillow/matplotlib/graphviz/ffmpeg/office libraries), and paid APIs only as a last resort for photoreal textures that code cannot render.
12.2 The authoring substrate↩︎
Synthesis is organized as a reproducible authoring layer kept separate from the validated release: generation code is version-controlled while bulky intermediate renders are not, so the pipeline is reproducible rather than hidden. A shared helper library centralizes asset synthesis across modalities behind a uniform interface—token accounting against the same tokenizer the evaluation harness uses; text-to-speech audio; slideshow and procedurally animated video; office and binary container formats (spreadsheets, documents, PDFs, encrypted archives, columnar and embedded databases); and a family of image renderers (charts, engineering diagrams, annotated medical and remote-sensing imagery, synthetic screenshots)—paired with post-hoc validators that reject undersized or malformed assets. Each scenario is a self-contained unit that pairs a frozen design contract, deterministic synthesis code, the check sources, and a reverse-verification harness. Its build entry point is idempotent—it resets and regenerates the entire scenario—and ends by self-checking the corpus against two budget floors derived from the design criteria: the inaccessible-only corpus that the main agent must delegate to read, and the staged-update payload.
12.3 Authoring a single scenario↩︎
A single scenario follows a fixed seven-step recipe: (1) read the spec (the C1–C10 criteria and the format and pitfall checklists); (2) scaffold the idempotent build script; (3) lay down the bulk text corpus to satisfy the context-pressure floor; (4) render multimodal assets (image \(\rightarrow\) audio \(\rightarrow\) video, each double-checked at generation); (5) write the scenario manifest, the question sequence, and the per-round checks, attaching each staged update to the first round that must observe its effect; (6) simulate the golden answer in a temporary copy and confirm every check passes, then write a plausible-but-wrong answer and confirm a check fails; (7) strip any ground-truth residue and pass the whole-set validator.
Two engineering principles keep scenarios robust. Prompt–check interlock: the more a question is written in natural intent (no field names, no tool names, no subagent recipe), the more its check must match by shape (an integer \(\ge 45\), an ISO date, a non-empty string containing a stem) rather than by key name; a recipe-style prompt paired with a shape-blind check is the dominant source of spurious failure. Three-layer checks: every check verifies structure (file exists, skeleton present, JSON parses), then fields (non-empty, correctly typed), then truth (exact numeric match, regex/substring, logical consistency)—never whole-segment semantic comparison, and never an LLM judge. To make these difficulties composable, the methodology maintains a reusable difficulty-vector library (Table 6); new scenarios combine de-recipe with content traps plus at least one structural and one detail vector.
| Vector | What it exercises |
|---|---|
| De-recipe prompt | intent only \(\rightarrow\) shape-blind check |
| Content trap | shallow read yields wrong value |
| Staged supersede | \(u_2\) retracts part of \(u_1\) |
| Concurrency pressure | object count forces fan-out |
| Self-audit of chains | re-query truth vs.cite own note |
| Archived red herring | canonical vs.stale same-name copy |
| Cross-round verbatim | later round must quote earlier one |
| Disagreement arbitration | audio vs.transcript, rationale req. |
4pt
The seventh step is enforced by a per-scenario reverse-verification harness: it materializes the golden deliverables in a temporary workspace and asserts every round passes, then injects targeted perturbations and asserts the corresponding round fails. In one incident-analysis scenario, for instance, the perturbations replace the correct failed-component identifier (its round must then fail) and substitute a transcript’s decoy numeric value for the recording’s true value (a modality trap; that round must then fail). This guards against checks silently degrading into always-pass.
12.4 Workflow-driven parallel authoring↩︎
Each batch of scenarios is produced by a three-phase procedure. Phase A (lock the skeleton) is controller-only: the controller internalizes the design handbook and the prior batch’s post-mortem, writes a batch plan and an authoring guide, hand-builds one proof-of-concept scenario through all gates as the gold reference, and freezes every scenario’s anchors in a short per-scenario contract. Phase B (parallel fan-out) spawns one subagent-author per remaining scenario, bounded at a small concurrency (beyond it, shared-file write contention and progress-tracking cost come to dominate), with file-system isolation for slow asset jobs. Phase C (convergence) returns to the controller, who assembles the shared manifests once, runs whole-set validation, and starts the flywheel.
The chief risk of parallel authoring is ground-truth drift—independent authors silently choosing conflicting anchors. Four mechanisms prevent it: the per-scenario contract as a binding specification (anchor table, workspace layout, per-round question and check anchors, token-budget split); the proof-of-concept referenced as a worked example; a distilled authoring guide so authors need not re-derive the handbook; and an explicit no-touch list of shared files whose violation triggers an immediate revert rather than author self-correction. Model tiers are assigned by task: cheaper models for mechanical padding and I/O-bound jobs, stronger models for check-interlock logic and anchor design. This division—decompose, contract, empower by tier, confine by least privilege, integrate behind gates—is exactly the management surface the benchmark scores, the reflexivity noted above. Crucially, the procedure is portable: nothing in it is specific to ClawArena-Team’s domains or asset types, so the same controller–contract–gate workflow transfers to authoring other execution-checked agentic benchmarks. The orchestration workflow, not any individual scenario, is the reusable methodological artifact we expect to outlast the particular set it produced.
12.5 The end-to-end test flywheel↩︎
Acceptance proceeds through four static gates and one dynamic gate (Table 7). The static gates establish that the data is well-formed; they are necessary but not sufficient. The dynamic gate—a real baseline run of the full subagent-management harness against the fixed local pool—establishes that the data is actually solvable and that checks are neither over-strict nor over-loose. Without a baseline run, one cannot tell whether a scenario is the right difficulty, whether a check is too tight, or whether the ground truth can be recovered at all.
| Gate | Mechanism | Asserts |
|---|---|---|
| G1 build | build script | idempotent; corpus budgets |
| G2 verify | reverse-verify | golden PASS; \(\ge2\) perturb FAIL |
| G3 check | structural validator | schema/whitelist, 0 warnings |
| G4 stats | budget profiler | token budgets meet floor |
| D baseline | baseline harness | solvable; checks calibrated |
4pt
Every baseline failure is then assigned to exactly one of two classes. A true-difficulty failure—an omni pool missing a key spoken figure, a model taking the modality decoy, a fabricated identifier, a directory over-granted to a subagent—is the value of the benchmark and is never loosened. A false-kill—an equivalent answer rejected by a brittle regex, an order-independent field locked to an order, a tolerance gap—is repaired by loosening the check on both the deployed and source copies, after which the perturbation guard must still fail. The flywheel (Figure 5) then repeats until every remaining failure is true-difficulty. The governing health criterion is counter-intuitive: a low completion rate with \(100\%\) true-difficulty failures is far healthier than a high rate propped up by loosened false-kills.
12.6 Iterative refinement and reproducibility↩︎
The released set is not a single-pass artifact but the fixed point of repeated flywheel turns. Across successive authoring iterations the difficulty was ratcheted deliberately: each iteration introduced harder structural and adversarial vectors and was accepted only once its baseline run left every failure classified as true difficulty rather than a check artifact. A final systematic audit re-triaged the checks across the whole set for the over-loose, over-strict, and parsing-miss patterns the flywheel had surfaced, paired with a multimodal re-rendering pass and a full re-check against the fixed local pool. Because every asset, check, and budget floor is scripted and self-verifying, the entire benchmark is regenerable from source, and its construction history lives in version control rather than in any released artifact.
13 Dataset Composition and Statistics↩︎
All statistics in this section describe the full 41-scenario set (clawarena-team); per-wave subsets are not reported. Quantities are computed with a single qwen3 tokenizer for cross-item comparability. At a
glance: 41 scenarios, 258 rounds (44 preceded by staged updates, \(17.1\%\)), 72 update groups over 255 files, \(170.5\) MiB of workspace content, and \(28.9\) M tokens.
13.1 Token and modality composition↩︎
Workspace content dominates token volume while staged updates contribute over a quarter (Table 8). By modality (Table 9), text accounts for most files and tokens, but audio and image dominate raw bytes—confirming a substantial non-text payload that a text-only main agent cannot consume directly.
| Category | Tokens | % |
|---|---|---|
| Workspace | 20,795,310 | 71.9 |
| Updates | 8,065,800 | 27.9 |
| Questions | 38,580 | 0.1 |
| Feedback | 32,409 | 0.1 |
| Total | 28,932,099 | 100.0 |
| Modality | Files | File% | Bytes | Tokens |
|---|---|---|---|---|
| text | 1840 | 75.3 | 61.8 MiB | 18,462,122 |
| image | 188 | 7.7 | 13.0 MiB | 52,640 |
| audio | 29 | 1.2 | 78.6 MiB | 21,750 |
| video | 19 | 0.8 | 6.9 MiB | 38,990 |
| document | 62 | 2.5 | 2.4 MiB | 1,255,036 |
| other | 305 | 12.5 | 7.7 MiB | 964,772 |
4pt
13.2 File types↩︎
The benchmark deliberately mixes many formats so that managers must route diverse content (Table 10): Markdown and logs carry most text tokens, uncompressed .wav audio dominates bytes, and office/data formats
(.docx, .pdf, .parquet, .csv, .ndjson, .eml, .html) appear throughout.
| Ext | Files | Byte% | Tokens |
|---|---|---|---|
| .wav | 29 | 46.1 | 21,750 |
| .md | 1141 | 23.2 | 8,105,280 |
| .png | 188 | 7.6 | 52,640 |
| .log | 35 | 6.9 | 6,629,461 |
| .mp4 | 19 | 4.0 | 38,990 |
| .parquet | 11 | 2.6 | 0 |
| .py | 162 | 1.5 | 716,917 |
| .txt | 92 | 1.2 | 625,338 |
| .html | 28 | 1.2 | 398,293 |
| .csv | 66 | 1.1 | 1,352,259 |
| .ndjson | 14 | 0.7 | 644,257 |
| .docx | 31 | 0.7 | 630,657 |
| 25 | 0.7 | 595,981 | |
| .json | 50 | 0.3 | 253,410 |
| .eml | 115 | 0.3 | 102,344 |
4pt
13.3 Multimodal coverage↩︎
Audio spans 29 files totaling \(3{,}090.6\) s (\(\approx 51.5\) minutes); video spans 19 files totaling \(9{,}106\) frames; images number 188. The largest
files are uncompressed .wav interviews and statements (up to \(4.7\) MiB). Multimodal payload appears in nearly every scenario—e.g.hospital_safety_event_review alone contains 37 images—so modality
routing is exercised broadly rather than in a few special cases.
13.4 Staged updates↩︎
Update groups are \(61.1\%\) new and \(38.9\%\) replace, averaging \(3.54\) files per group (max 15; 255 files total). Replaced
files force the manager to supersede earlier beliefs; new files extend the workspace mid-task.
13.5 Capability-tag taxonomy↩︎
Every round is annotated with controlled capability tags from a 51-tag vocabulary organized into eight sections (Table 11); \(98.4\%\) of rounds carry \(\geq 1\) tag, averaging \(4.48\) tags per round, and all eight sections reach \(100\%\) section coverage. Table 12 lists the most frequent tags and Figure 6 the full distribution. The taxonomy spans the entire management capability surface—delegation/permission, modality routing, update handling, adversarial traps, office/data formats, code execution, and information synthesis—evidencing that the benchmark systematically exercises subagent management rather than a single skill.
4pt
| Tag | Section | Rd% | Scn% |
|---|---|---|---|
| numerical_extraction | Struct | 32.6 | 92.7 |
| verbatim_citation | Struct | 26.0 | 87.8 |
| subagent_delegation | Deleg | 25.2 | 97.6 |
| final_synthesis | Synth | 22.1 | 97.6 |
| discredit_window | Trap | 20.9 | 82.9 |
| cross_round_consistency | Struct | 20.2 | 85.4 |
| triage_planning | Synth | 15.5 | 97.6 |
| session_reuse | Deleg | 15.1 | 61.0 |
| update_merge | Update | 14.3 | 90.2 |
| bash_tool_run | Code | 13.6 | 70.7 |
| permission_restraint | Deleg | 12.4 | 56.1 |
| modality_decoy | MM | 11.2 | 68.3 |
| parallel_subagents | Deleg | 10.5 | 53.7 |
4pt
13.6 Largest scenarios↩︎
By tokens: trading_tz_incident (\(5.09\)M), satellite_change_detection (\(1.50\)M), security_incident_triage (\(1.33\)M), security_pcap_triage (\(1.14\)M), oss_supply_chain_audit (\(1.12\)M). By workspace size:
candidate_background_check (\(9.4\) MiB), smart_home_anomaly_triage (\(9.3\) MiB), trading_tz_incident (\(9.1\) MiB).
The most rounds occur in codebase_migration_review (10) and ml_rl_policy_review (8).
14 Token Usage↩︎
14.0.0.1 Estimation caveat.
All token counts in this paper are computed post hoc by applying a single, unified tokenizer and chat template uniformly across all models, rather than read from each provider’s billing. They therefore approximate—and may deviate slightly from—each provider’s own tokenization and true token consumption. All token totals below, and the costs derived from them (Appendix 11), should be read as estimates for cross-model comparison rather than exact bills.
Tables 13 and 14 report main- and subagent-side token usage. Main context peak (MCM) relative to the \(200\)k advisory value (MCU%) shows that some models approach or exceed it. Subagent totals (SIT/SOT/SCH) confirm that the bulk of token volume is borne by the fixed local pool, which is why only the main agent is priced.
| Model | MCM | MCU% | MIT | MCR |
|---|---|---|---|---|
| claude-fable-5 | 86,918 | 43.5 | 931,909 | 44,114,111 |
| gemini-3.5-flash | 135,105 | 67.6 | 1,881,581 | 109,496,591 |
| gpt-5.5 | 66,431 | 33.2 | 956,355 | 44,838,666 |
| glm-5.2 | 144,550 | 72.3 | 1,268,167 | 59,490,242 |
| gpt-5.4 | 60,327 | 30.2 | 842,628 | 38,437,272 |
| gemini-3.1-pro | 125,773 | 62.9 | 1,405,277 | 72,868,929 |
| kimi-k2.6 | 196,951 | 98.5 | 1,726,445 | 73,504,933 |
| claude-sonnet-4-6 | 150,396 | 75.2 | 1,266,762 | 59,975,699 |
| deepseek-v4-pro | 154,241 | 77.1 | 1,623,004 | 60,776,770 |
| qwen3.6-27b | 250,575 | 125.3 | 1,156,906 | 45,508,262 |
| gemma-4-31b | 90,507 | 45.3 | 905,532 | 35,848,384 |
| glm-4.7-flash | 98,548 | 49.3 | 1,035,903 | 46,581,293 |
| Model | SIT | SOT | SCH |
|---|---|---|---|
| claude-fable-5 | 6,816,130 | 629,375 | 30,191,195 |
| gemini-3.5-flash | 7,095,454 | 946,816 | 28,961,064 |
| gpt-5.5 | 4,602,126 | 426,427 | 18,421,450 |
| glm-5.2 | 5,651,660 | 851,710 | 26,058,307 |
| gpt-5.4 | 5,258,901 | 364,872 | 15,585,928 |
| gemini-3.1-pro | 4,770,118 | 581,159 | 26,702,392 |
| kimi-k2.6 | 5,116,392 | 577,839 | 23,759,396 |
| claude-sonnet-4-6 | 5,994,993 | 657,368 | 11,617,382 |
| deepseek-v4-pro | 5,516,428 | 902,067 | 25,062,801 |
| qwen3.6-27b | 5,065,238 | 706,728 | 26,595,078 |
| gemma-4-31b | 5,685,646 | 506,638 | 31,534,922 |
| glm-4.7-flash | 1,920,045 | 414,666 | 37,489,999 |
15 Per-Scenario Breakdown↩︎
Figure 7 shows Sms for every scenario \(\times\) model, with scenarios ordered by mean difficulty. The figure is regenerated as the model set grows.
16 Tool-Grant and Bash-Mode Statistics↩︎
We additionally log, per model, the distribution of granted tool types (TGC) and the foreground/background split of Bash calls (BSH) together with structured-output usage (SOC); these support the behavioral analysis in §4 and are omitted here for space.
17 Case Studies↩︎
The twelve cases below, presented as Figures 8–10, are drawn directly from the recorded run transcripts and metadata.json files; all numbers and quoted strings are verbatim from
those artifacts. They span single-model deep-dives, head-to-head model comparisons, successes, and characteristic failure modes, concretely grounding the three findings and the management capability surface.
References↩︎
Corresponding authors.↩︎