1 Introduction↩︎

LLM-based agent research has expanded from single reasoning-and-acting loops [@yao2022react; @schick2023toolformer; @nakano2021webgpt; @yang2024sweagent] and role-specialized multi-agent systems [@li2023camel; @wu2023autogen; @hong2023metagpt] to designs that configure, evolve, or generate the workflow itself [@chen2023agentverse; @yuan2024evoagent; @hu2024adas; @zhang2024aflow; @fourney2024magenticone].

Accuracy alone cannot separate workflow organization from protocol advantage. Recent benchmarks broaden coverage to interactive tasks, tool use, web environments, software engineering, and diagnostic signals [@liu2023agentbench; @mialon2023gaia; @ma2024agentboard; @zhou2023webarena; @jimenez2024swebench; @qin2023toolbench]. Yet cross-paradigm comparisons routinely leave inputs, answer contracts, tool surfaces, usage accounting, or trajectory logging uneven across systems. For example, debate and orchestration frameworks explicitly add extra rounds, role-specialized handoffs, or tool-routing policies relative to a single-controller loop [@du2023multiagentdebate; @wu2023autogen; @shen2023hugginggpt]; if such systems are compared only by final accuracy, the observed gap can mix coordination with protocol advantage. Our same-instance contrast in Figure 3 later shows a complementary failure mode: an intermediate handoff can drop a task constraint that is invisible in the final answer alone. This is why analytical traces matter in agent evaluation frameworks [@ma2024agentboard; @harbor_framework]: without them, equivalent-looking systems can spend different tokens, lose different constraints, or handle tool failures differently.

Figure 1 sketches the evaluation setup: benchmark instances enter a shared runtime and evaluator, while only the workflow layer varies across single-agent, fixed MAS, dynamic/evolving MAS, and externally evaluated runtime-generated execution. The goal is not to identify the strongest possible agent controller, but to estimate workflow lift: whether reorganizing a matched controller into a MAS workflow yields reliable marginal gain under explicit protocol boundaries. Accordingly, the paper reports one controlled substrate-internal comparison and one protocol-aligned external runtime case study rather than treating all systems as a single symmetric leaderboard.

We build BenchAgent for this purpose. BenchAgent is an evaluation framework; it wraps existing benchmarks in a common execution interface and records the process signals needed for workflow comparison. BenchAgent places single-agent systems, fixed MAS, and evolving MAS under the same benchmark-normalized interface, sandbox runtime, tool-allocation rules, and process-level logger. We use four workflow labels throughout the paper: single-agent, fixed MAS, dynamic/evolving MAS, and runtime-generated workflow. We call the first three a Substrate-Internal (SI) comparison when the workflow runs inside BenchAgent and exposes its messages, tool calls, usage, and termination state to the same logger. For systems that cannot be reimplemented without changing their runtime semantics, we use a Protocol-Aligned External (PAE) comparison: inputs, evaluator, final-answer format, tool-capability classes, backend model family, and permission profile are aligned, while controller details remain visible through retained traces.

With BenchAgent, we compare a single-agent configuration and representative fixed and evolving MAS with a GPT-4.1 backend. The broad suite covers reasoning, coding, and tool-use benchmarks (see Table 1). On GAIA [@mialon2023gaia], we evaluate the Claude-Code-style runtime workflow (CC-workflow) through a PAE study and compare it with the BenchAgent-based MAS workflows.

The broad benchmark results show that increasing the number of agents or adding explicit coordination does not by itself guarantee positive workflow lift. Fixed and evolving MAS vary in effectiveness, token usage, and latency, and several trail the matched single-agent anchor. The GAIA comparison asks a narrower system-level question: whether a mature runtime workflow reaches a different accuracy–cost trade-off on long-horizon tool-use tasks. It does on Levels 2–3, but this is a deployed-configuration result, not evidence for any single mechanism. Our key contributions are:

• We introduce BenchAgent, a workflow-evaluation substrate that normalizes benchmark loading, tool access, answer contracts, usage accounting, and trajectory logging for cross-paradigm agent comparison.

• Under SI conditions with a GPT-4.1 backend, shared tools, evaluator, and logger, at most one of six tested MAS exceeds the matched single-agent anchor on average, and even that case (EvoAgent, +1.44 points) lies within the Wilson one-run guidance; the remaining five trail by 2.56–11.29 points.

• We report a PAE GAIA snapshot in which the CC-workflow reaches 66.72% overall accuracy, leads on Levels 2–3, and uses fewer recorded tokens than the strongest non-Claude baseline; isolating which design choice drives the gain requires controlled ablations beyond our deployed-configuration comparison.

2 Related Work↩︎

2.1 Agent Workflow Design↩︎

ReAct, Toolformer, WebGPT, SWE-agent, and HuggingGPT/JARVIS show that a single instrumented controller can already serve as a competitive workflow [@yao2022react; @schick2023toolformer; @nakano2021webgpt; @yang2024sweagent; @shen2023hugginggpt]. We therefore treat the single-agent baseline as a matched workflow anchor rather than as a bare model call or as a claim about the strongest possible single-controller design; recent work corroborates that a strong single-agent baseline can match or exceed homogeneous MAS [@xu2026rethinking].

Multi-agent systems make the division of labor explicit. CAMEL and MetaGPT define agent roles and communication procedures [@li2023camel; @hong2023metagpt], AutoGen provides a conversable-agent substrate for fixed or adaptive interaction patterns [@wu2023autogen], and debate-style systems aggregate multiple proposals before finalization [@du2023multiagentdebate; @chan2023chateval]. Li et al. [@li2024moreagents] show that sampling-and-voting improves several LLM benchmarks; DSPy [@khattab2023dspy] instead optimizes LM pipelines through compilation. Dynamic and evolving systems search over or mutate the workflow structure within a designer-specified space, as in AgentVerse, EvoAgent, ADAS, AFlow, and Magentic-One [@chen2023agentverse; @yuan2024evoagent; @hu2024adas; @zhang2024aflow; @fourney2024magenticone]; MaAS extends this by searching over query-dependent agent topologies via an agentic supernet [@zhang2025maas]; MASPO further co-optimizes agent prompts jointly across the pipeline [@wang2026maspo]. Cemri et al. [@cemri2025maswhyfail] catalog MAS failure modes attributable to coordination design; Kim et al. [@kim2025scaling] further argue that net workflow gain requires coordination benefit to exceed communication overhead. These systems motivate our taxonomy, yet prior comparisons rarely run single-agent, fixed MAS, evolving MAS, and runtime workflows under a unified logging and accounting protocol.

2.2 Agent Evaluation Frameworks↩︎

AgentBench, GAIA, AgentBoard, WebArena, SWE-bench, ToolBench, and Silo-Bench [@liu2023agentbench; @mialon2023gaia; @ma2024agentboard; @zhou2023webarena; @jimenez2024swebench; @qin2023toolbench; @zhang2026silobench] extend agent evaluation to interaction, tool use, web/software tasks, analytical traces, and distributed coordination. Harbor [@harbor_framework] is closest in infrastructure, evaluating and optimizing sandboxed agents and models in containerized environments. BenchAgent occupies a different scope: it introduces neither a new task dataset nor a benchmark split, but instead aligns execution interfaces, tool surfaces, usage accounting, and trajectories, so that coordination policies can be compared on existing benchmarks. Table 6 in Appendix summarizes the feature-level distinction from AgentBench and Harbor.

2.3 Runtime-Generated Workflows↩︎

Recent engineering systems push part of workflow design into the execution loop. LangGraph [@langgraph_docs] and CrewAI [@crewai_docs] provide graph orchestration and role-based crews; Anthropic [@anthropic_building_effective_agents; @anthropic_multiagent_research_system; @anthropic_context_engineering] documents subagent creation, context-window isolation, and permission-scoped delegation patterns instantiated in Claude Code [@anthropic_claude_code_product; @anthropic_claude_code_overview; @anthropic_claude_code_subagents]; similar agentic coding interfaces appear in OpenAI Codex and OpenCode [@openai_codex_product; @opencode_product]. These systems are documented primarily through official documentation rather than peer-reviewed publications; we cite them as engineering descriptions, not empirical evidence. Our GAIA experiment asks whether the CC-workflow paired with a GPT-4.1 backend reaches a different accuracy–cost profile from BenchAgent-based fixed and evolving MAS under a documented PAE comparison.

3 Evaluation Protocol↩︎

3.1 Workflow Lift as the Measurement Target↩︎

We evaluate agent systems as workflow organizations, not isolated products. For a benchmark instance \(x\), a workflow \(w\) produces a final answer \(\hat{y}=w(x)\) and, when available, an execution trace \(\tau_w(x)\) containing model calls, tool calls, messages, artifacts, and termination events. A task evaluator \(E(x,\hat{y}) \in \{0,1\}\) measures success, while a cost summary \(c(\tau_w)\) records usage signals such as tokens, latency, tool calls, and delegation structure. The comparison therefore asks not only whether \(\hat{y}\) is correct, but also what trace \(\tau_w\) produced it.

We distinguish workflow categories by how \(\mathcal{A}_t\) (active agents), \(G_t\) (communication or delegation topology), and \(\mathcal{T}_t\) (tool scopes) are specified over time:

• Single-agent workflow. Keeps \(|\mathcal{A}_t|=1\) throughout; a single controller handles the entire trace.

• Fixed MAS. Operates with a predefined \((\mathcal{A},G,\mathcal{T})\), such as solver–critic–aggregator or debate-style handoffs.

• Evolving MAS. Selects or mutates \(G\) at runtime from a designer-specified family \(\mathcal{G}\) (a.k.a.dynamic MAS).

• Runtime-generated workflow. Changes \(\mathcal{A}_t\), \(G_t\), and \(\mathcal{T}_t\) during execution by creating task-specific agents, assigning private context, choosing tool scopes, or adding verification and recovery branches.

Table 10 gives the full terminology map.

This framing does not require the single-agent anchor to be the strongest possible ReAct implementation; instead, it requires the compared workflows to share the same control substrate, so that the measured difference reflects the tested workflow organization. Section 3.3 describes how single-run lift estimates are interpreted under Wilson guidance.

3.2 Comparison Setup: SI and PAE↩︎

BenchAgent provides the shared execution substrate for the SI comparison and normalizes six interfaces—benchmark loading, input and answer formatting, runtime control, tool access, usage and trajectory logging, and evaluator calls. In the broad-benchmark experiment (Section 4.2), Jarvis/HuggingGPT-style execution, LLM-Debate, EvoAgent, AutoGen, CAMEL, and ChatEval are implemented as workflow instances over this substrate rather than as unrelated tool stacks. This substrate renders final scores, token usage, latency, tool calls, message histories, agent identifiers, and stage-level traces comparable across workflows. Appendix 8.2 includes retained prompt excerpts that document the controller instructions and output contract used for the external runtime workflow.

Claude Code cannot be reimplemented inside BenchAgent without altering its runtime semantics. The GAIA experiment (Section 4.3) therefore adopts a PAE comparison. The CC-workflow shares the GPT-4.1 backend family, GAIA validation inputs, answer-only output schema, evaluator, and GAIA-relevant tool-capability classes with the SI baselines, while internal controller details remain only partially visible through retained traces. Table 11 in the Appendix lists which protocol fields are aligned and which cannot be certified from the released artifacts.

3.3 Reporting Protocol and Wilson Guidance↩︎

For each instance, we record task success, token usage, wall-clock latency, and the full trajectory. When traces expose sufficient structure, we also derive process summaries—spawned agents, tool calls, and delegation depth—used for interpretation, not as additional success metrics. Broad-benchmark average accuracy is the unweighted mean over ten benchmark-level accuracies; GAIA overall is weighted by level sizes \((N_1,N_2,N_3)=(53,86,26)\). All numbers are pass@1 single-run results.

We use the Wilson 95% binomial confidence interval [@wilson1927] as a conservative scale for one-run accuracy gaps: for an observed proportion \(\hat{p}\) over \(N\) instances, the interval covers the unknown success probability more reliably than the normal approximation when \(\hat{p}\) is near \(0\) or \(1\) or when \(N\) is small. We report its two-sided half-width per benchmark in Table 5 and treat any pass@1 gap smaller than this half-width as descriptive rather than as stable ordering evidence. Concretely, differences below roughly 17 points on AIME (\(N{=}30\)) or 18 points on GAIA Level 3 (\(N{=}26\)) should not be read as ordering evidence under one run.

4 Experiments↩︎

We instantiate two experiments under the protocol of Section 3. The first experiment (Section 4.2) tests whether fixed and evolving MAS produce positive workflow lift over a matched single-agent anchor on ten broad benchmarks under SI conditions; the second (Section 4.3) places a Claude-Code-style runtime workflow on GAIA under PAE alignment. Each number reported below is a pass@1 result from one end-to-end run: we treat gaps smaller than the Wilson half-widths in Appendix Table 5 as descriptive rather than as stable ordering evidence, and we do not attribute the CC-workflow’s GAIA advantage to a single mechanism without ablation evidence.

4.1 Experimental Setup↩︎

Datasets. The broad SI comparison covers ten benchmarks: MATH, AIME, GSM8K, DROP, BBH, MMLU-Pro, HumanEval, MBPP, HotpotQA, and IFEval. The runtime-workflow study uses the full GAIA validation split [@mialon2023gaia] (\((N_1,N_2,N_3)=(53,86,26)\), weighted by level size for overall accuracy), which stresses multi-step retrieval, file inspection, state preservation, and delayed final-answer release.

Tool allocation. Tool access is benchmark-specific but fixed across compared workflows. In the SI suite, HotpotQA uses the expanded full BenchAgent tool registry, whereas MATH, AIME, GSM8K, DROP, BBH, MMLU-Pro, HumanEval, MBPP, and IFEval expose only python_interpreter beyond final-answer emission. GAIA uses the corresponding full-tool regime: BenchAgent-based systems receive the expanded full registry, and the CC-workflow is matched at the level of GAIA-relevant tool-capability classes under the PAE protocol. Appendix Table 4 summarizes the benchmark-level allocation.

Compared systems. The single-agent anchor is BenchAgent Core—a full substrate run, not a bare LLM call—sharing the same loader, evaluator, and tool registry as all MAS wrappers. It serves as the workflow-lift estimator for the broad-benchmark experiment (Section 4.2), not as a claim about the strongest possible single-agent controller. Fixed MAS baselines include Jarvis [@shen2023hugginggpt], LLM-Debate [@du2023multiagentdebate], AutoGen [@wu2023autogen], CAMEL [@li2023camel], and ChatEval [@chan2023chateval]; EvoAgent [@yuan2024evoagent] represents the evolving MAS category. Implementation notes, transfer sanity checks, and a vanilla ReAct anchor calibration appear in Appendix Tables 21 and 7.

CC-workflow. Evaluated only on GAIA under a PAE protocol: inputs, answer schema, evaluator, backend model family, and GAIA-relevant tool-capability classes are aligned; the controller remains partially external. Full protocol documentation appears in Appendix Tables 11 and 12.

3.8pt

Model and infrastructure. All reported main-text BenchAgent results use GPT-4.1 (gpt-4.1-2025-04-14) [@openai_gpt41_model]. Dataset sampling, tool-surface summaries, GAIA-only backend-generalization checks, and trajectory evidence appear in the appendix; the backend checks are scoped to Qwen3-32B for BenchAgent-compatible GAIA workflows and GLM-5 for the BenchAgent GAIA anchor (Appendix Tables 8 and 9).

4.2 Broad Benchmark Results↩︎

This experiment tests whether fixed or evolving MAS produce workflow lift over the BenchAgent single-agent anchor under matched model, tool surface, evaluator, and logging protocol; results appear in Table 1, with the accuracy–cost view in Figure 2. Appendix Table 7 calibrates the anchor against a standalone vanilla ReAct controller without redefining the comparison target.

5pt

Fixed and evolving MAS do not consistently produce positive workflow lift. The matched single-agent anchor reaches 74.12% benchmark-balanced average accuracy, while EvoAgent reaches 75.56%. These broad-suite averages are descriptive benchmark-balanced means rather than interval estimates on a single shared instance pool; per-benchmark Wilson half-widths in Table 1 carry the row-level uncertainty. EvoAgent is the only MAS with numerically positive average lift, but its +1.44-point gain is smaller than the one-run uncertainty guidance; the remaining MAS fall below the anchor at 62.83%–71.56%. Holding model, tools, evaluator, and logger fixed, adding roles or handoffs did not lift overall performance. Takeaway: at most one MAS exceeds the anchor on average, and that case sits within the one-run uncertainty scale.

Cost varies more sharply than accuracy. Figure 2 shows workflows with similar benchmark-balanced accuracy but different instance-level token usage and latency. EvoAgent’s +1.44-point gain comes at higher token cost; ChatEval is far more token-intensive while trailing the baseline; Camel and Jarvis are lighter and faster but lose accuracy. Takeaway: accuracy-similar workflows can have sharply different token use and latency.

Workflow structure, not agent count, explains MAS variation (Appendix Table 22). Agent count does not explain MAS behavior in our suite: EvoAgent gains on BBH through prompt-scaffold search, LLM-Debate gains on HumanEval and MATH where proposals are verifiable, and ChatEval gains on IFEval under multi-judge instruction checking. The same protocols hurt elsewhere. Takeaway: aggregate MAS behavior follows task-protocol fit rather than agent count.

MAS gains are task-dependent. EvoAgent’s BBH score (94.00% [\(\pm\)​2.4] vs.% [\(\pm\)​4.0] for single-agent) reflects task-specific scaffold selection, not a general multi-agent advantage, and raises a fairness question about prompt-search budget equity. LLM-Debate leads on HumanEval and MATH, where proposals are easy to check; ChatEval scores highest on IFEval but collapses on AIME. MAS help when their protocol matches the task error mode, and add overhead or lose information otherwise. Additional task-level views appear in Appendix Figures 7–10. Takeaway: MAS protocols help when they match a task’s error mode, and hurt otherwise.

4.3 GAIA Runtime Workflow Results↩︎

This experiment focuses on GAIA, where long-horizon tool use tests state preservation, evidence management, and recovery after partial failures; results appear in Table 2, with backend-sensitivity checks in Appendix Tables 8 and 9 and the same-instance contrast in Figure 3. The comparison remains PAE, as described in Section 3.2 and Appendix Table 11, so it compares aligned deployed configurations rather than isolating an internal mechanism.

The CC-workflow advantage scales with task length. The CC-workflow reaches 66.72% (\(\pm\)​7.1) overall, 20.06 points above the strongest non-Claude result (Jarvis, 46.66%). Jarvis leads on Level 1, where short retrieval chains often suffice, but the CC-workflow leads by 26.60 points on Level 2 and 42.31 points on Level 3, gaps larger than the uncertainty scale in Appendix Table 5. Retained accounting also reports fewer tokens and less wall-clock time than the strongest non-Claude baseline, though cost summaries depend on log visibility and provider-side bookkeeping. Takeaway: the CC-workflow leads by 20.06 points overall, with a 42.31-point Level 3 gap.

Retained traces expose structured runtime coordination. The CC-workflow traces show task-specific subagents, reusable evidence artifacts, and verifier-stage events before final-answer release (6.25 subagents on average, max delegation depth 2.5; Appendix Table 25). Figure 3 illustrates the contrast: EvoAgent loses a task-critical constraint and forces an answer, while the CC-workflow preserves state and verifies before finalization. Takeaway: subagent count and delegation depth (6.25 / 2.5 max) track task structure. Appendix 8.2 gives a retained event-level excerpt from the GAIA trajectory logs.

The two failure modes are structurally different. In fixed and evolving MAS baselines, a failed sub-step typically stays in the same linear handoff: a solver makes a claim, an aggregator receives a compressed version, and the final answer can be forced with a missing constraint. In retained CC-workflow traces, failures appear as local tool or state problems addressable by changing tool strategy, re-reading an artifact, or invoking a verifier. Candidate mechanisms—runtime delegation, persistent artifacts, verifier-stage control, and context management—remain confounded; Appendix Table 23 lists the needed ablations. Takeaway: baseline failures get baked into linear handoffs; CC-workflow failures stay local and recoverable.

5 Discussion↩︎

The broad-benchmark results show task-specific failure modes, not a uniform MAS advantage. Debate-like aggregation helps when independent proposals are checkable (LLM-Debate on HumanEval and MATH), while evolutionary search helps when prompt or role variation finds a better reasoning mode (EvoAgent on BBH). The same machinery hurts when tasks require exact instruction following or tight evidence control, because handoffs can compress context and hide constraints. Table 1 is therefore a workflow-lift comparison around a shared SI anchor, not a leaderboard over all single-agent designs.

Runtime-generated workflows require a different evaluation target. The CC-workflow exposes candidate mechanisms—separate context windows, permission scopes, persistent artifacts, and runtime delegation—that remain confounded with provider-side bookkeeping. Evaluations should therefore report workflow topology, context boundaries, tool scopes, and repair gates alongside final scores; otherwise, similar accuracy can hide incompatible coordination policies, costs, and failure modes.

We keep SI and PAE separate: SI isolates workflow wrappers over a common core, while PAE reports what a mature runtime workflow achieves under a documented but less internally controllable protocol. Collapsing them into one leaderboard would mix workflow lift with protocol advantage.

6 Conclusion↩︎

BenchAgent evaluates LLM-agent workflows under aligned execution and instrumentation. Across broad benchmarks, fixed and evolving MAS do not consistently beat the matched single-agent anchor and occupy different accuracy–cost trade-offs. On GAIA, the CC-workflow performs better on harder levels with lower retained-token and wall-clock cost under PAE, but this remains a deployed-configuration comparison rather than a mechanism estimate. We release BenchAgent and trajectory tooling for joint study of workflow generation, context management, tool scoping, and verification as properties distinct from agent count.

7 Limitations↩︎

Our comparison is controlled but not fully causal. The broad-benchmark MAS are re-instantiations, not exact reproductions; the CC-workflow is an external engineering exemplar routed to the same backend, so its advantage may combine workflow generation, context compaction, file and shell tooling, permission scoping, and provider-side bookkeeping. The results compare deployed configurations, not isolated mechanisms.

Mechanism evidence is partial: we report one-run pass@1 results and rely on retained traces that expose only part of Claude Code’s internals. Appendix Table 7 adds a vanilla ReAct calibration on MATH, BBH, HumanEval, and HotpotQA to contextualize the matched anchor. Future work should add repeated trials, controller-sensitivity checks, stricter tool-surface-matched ablations, runtime-workflow ablations, and fuller instrumentation for context packaging and recovery behavior.

8 Evaluation Protocol and Supplementary Evidence↩︎

The appendix reports protocol checks, auxiliary baseline-transfer evidence, runtime-generated workflow traces, and supplementary broad-benchmark analyses.

8.0.0.1 Appendix contents.

This appendix is organized as follows:

• Appendix 8.1 reports dataset sampling, tool regimes, Wilson guidance, framework comparison, ReAct calibration, backend checks, and protocol-alignment tables.

• Appendix 8.2 gives retained prompt and trajectory excerpts for the runtime-workflow setting.

• Appendix 8.3 reports illustrative process signals and separates observed trace evidence from mechanisms that require ablation.

• Appendix 8.4 reports runtime-generated workflow trace statistics and same-task visual evidence.

• Appendix 8.5 summarizes artifact use, release planning, and responsible-use notes.

• Appendix 8.6 provides supplementary broad-benchmark visual analyses.

8.1 Protocol and Fidelity Checks↩︎

Table 3 reports the evaluation size used for each dataset. For any source split with more than 400 instances, we evaluate a fixed-random-seed subset of 400 instances. For source splits with at most 400 instances, we evaluate the full split. The full-split evaluations below 400 instances are AIME (30), HumanEval (131), MBPP (341), and GAIA validation (165). The benchmark-balanced average accuracy in Table 1 is computed over benchmark-level scores rather than by pooling all evaluated instances.

5pt

5pt

8.1.0.1 Global model-call configuration.

Unless otherwise noted, all experiments use MAX_TOKEN_SIZE=8192, temperature=0.2, and top_p=1.0.

6pt

5pt

Table 7 is a controller-strength calibration, not an alternative denominator for workflow lift. It changes the controller relative to BenchAgent Core, so the scores contextualize the matched anchor used in Table 1 rather than replacing the SI comparison, whose target is the effect of changing workflow organization while holding the BenchAgent execution substrate fixed.

5pt

Tables 8 and 9 report GAIA-only backend checks for BenchAgent-compatible workflows. The Qwen3-32B table uses the main-text GAIA metrics to compare the BenchAgent anchor with all fixed and evolving MAS rows inside the same BenchAgent protocol under the global model-call configuration. The GLM-5 table gives a focused BenchAgent backend-strength calibration on GAIA only. The external CC-workflow is intentionally excluded from these backend-swap checks because rerouting that controller to a different backend would change the PAE runtime configuration rather than produce a controlled BenchAgent-internal backend ablation.

5pt

5pt

The backend checks test whether the main workflow-level pattern is specific to one GPT-4.1 setting. They do not introduce a separate conclusion; instead, they provide scoped evidence for the same interpretation reported in the main experiments: MAS lift is not uniformly positive and depends on the interaction among backend reliability, task difficulty, and workflow structure. Under Qwen3-32B, the evaluated MAS rows all trail the BenchAgent anchor overall. Jarvis, Camel, and EvoAgent are closer to the anchor, but Camel’s Level 3 drop and the larger gaps for LLM-Debate, AutoGen, and ChatEval are consistent with the view that added coordination can be sensitive to harder multi-step evidence-control tasks. The GLM-5 BenchAgent calibration gives the complementary single-agent reference point, showing that changing the backend can substantially change absolute GAIA level scores while leaving the workflow-comparison question intact. Together, these checks support a bounded conclusion rather than a new claim: agent count or MAS structure alone is not a reliable source of improvement under the tested protocols.

Table 11 lists the retained protocol evidence for the GAIA comparison. Fields that cannot be certified from retained artifacts are marked explicitly rather than silently normalized.

4pt

3.5pt

8.1.0.2 Claude Code configuration notes.

The retained setup artifacts enable experimental Agent Teams and define four named roles: Planner, Solver, Verifier, and Writer. The prompt forbids modifying project code, deleting files, or running destructive/network commands, and requires the Writer to emit final answers and logs under outputs/ and logs/. No MCP or external servers are recorded for these runs. Local proxy credentials and API tokens are deliberately omitted from this paper.

3pt

4pt

4pt

4pt

8.1.0.3 Claude Code retained-log token accounting.

The retained LiteLLM-style records make the visible token boundary auditable for the PAE run. We therefore report CC-workflow cost as retained-log accounting rather than as provider-independent billing or a strict resource-budget ablation against BenchAgent. The counted usage includes model-visible instructions, the GAIA mission prompt, visible system-reminder context, tool schemas, tool-call JSON, subagent/delegation model calls, auxiliary title-generation calls, and tool results once returned to later prompts. Local shell execution is not tokenized until its output is fed back to the model. In the deduplicated batch, the logs contain 128 subagent model calls, 206 calls that emitted tool calls, 236 calls whose prompts include historical tool_use/tool_result context, and 22 title-generation calls. We did not observe an actual compaction-summary event in this batch; invisible provider-side policy remains outside the retained logs and is not used for mechanism attribution.

8.2 Prompt and Trajectory Excerpts↩︎

This subsection records compact prompt and trajectory excerpts used to audit the reported runs. Table 16 summarizes the BenchAgent Core prompt contract from bench/mas_arena/agents/prompts.yaml and bench/mas_arena/agents/bench_agent.py. Tables 17 and 18 extend the record to the reproduced fixed and evolving MAS wrappers. Table 19 gives three retained single-task trajectory excerpts from the response and structured-log artifacts, and Table 20 adds further excerpts covering the ChatEval debate structure and the CC-workflow verifier-release gate. Long task text, local paths, and hidden tool schemas are shortened only for readability.

4pt

Tables 17 and 18 extend the prompt-contract record to the other compared workflows. The fixed MAS table makes the “roles and topology specified before execution” characterization from Table 10 concrete: each system names its agents, assigns distinct or homogeneous instruction sets, and fixes the communication procedure before any task input is seen. The EvoAgent table documents the base-prompt seeds and the LLM-driven crossover and mutation operations, clarifying that the evolutionary search is confined to the prompt-scaffold space while the BenchAgent execution substrate and tool surface remain fixed across all candidates.

3.5pt

4pt

3pt

Table 20 extends the trajectory record to the fixed and runtime-generated workflow paradigms in more detail. The ChatEval-1 trace illustrates how the three-expert debate converges toward a single answer across rounds for a GAIA book-title retrieval task, connecting directly to the token and model-call overhead reported in Table 14. The Runtime-2 and Runtime-3 traces come from sub-agent perspectives within the same CC-workflow run (6.json): Runtime-2 records the Solver role reasoning and forwarding answers to the Verifier, while Runtime-3 records the Writer role receiving Verifier-confirmed packets and enforcing the output contract. Together they provide the artifact-level support for the “verifier-stage control” mechanism entry in Table 23.

3pt

Table 21 reports an auxiliary transfer sanity check for selected MAS baselines. It compares official implementations or officially reported task-level numbers against matched non-BenchAgent re-instantiations on six shared datasets. These numbers are not the BenchAgent results in Table 1; the check is scoped to task-level plausibility because the original reports and our transferred runs do not always share the same underlying base model generation. Its purpose is to verify that the transferred wrappers remain in a plausible task-level range before they are placed under the controlled BenchAgent protocol.

4.5pt

The independent transfers fall within a small-to-moderate range of the official reference numbers, but this table is only a sanity check. The controlled evidence in the main paper comes from Table 1, where the compared systems are evaluated under the shared BenchAgent execution substrate.

8.3 Illustrative Process Signals and Mechanism Evidence↩︎

Table 22 gives process context for broad-benchmark patterns in Table 1. Because the retained broad-benchmark artifacts do not expose a unified per-wrapper process summary, the measured-quantity column reports illustrative counts from retained GAIA traces for the same wrappers. These rows are used to interpret plausible workflow behavior, not to claim that the GAIA counts causally explain the broad-benchmark scores.

3.5pt

Table 23 separates observed trace signals from causal claims that require controlled ablation for the CC-workflow GAIA result.

2.5pt

Table 24 quantifies where errors are first introduced within each wrapper, using the binary-search failure attribution logs from retained GAIA runs. For each failed instance, the binary-search probe identifies the earliest conversation step at which the agent’s response diverges from the ground truth. Step 1 corresponds to the first agent turn (i.e., the initial generation before any iterative correction); later steps indicate that the wrapper’s internal loop produced at least one intervening response before the error was committed or persisted.

5pt

The failure-step profile is consistent with two complementary observations. First, both wrappers commit the majority of analyzed errors at the first agent turn, suggesting that the tested wrapper structures often do not compensate for an initial-turn knowledge or retrieval gap. Second, while ChatEval’s debate loop does occasionally defer or surface an error at a later round (steps 3–13 account for 16.8% of ChatEval failures), this correction window is limited and does not translate into a net accuracy gain on GAIA relative to the simpler Jarvis handoff. This supports the bounded process reading in Table 22: in these retained runs, ChatEval’s additional coordination expands token usage without a proportionate observed error-correction benefit.

8.4 Runtime-Generated Workflow Evidence↩︎

Table 25 reports structured-event statistics from the retained Claude Code dumps that preserve explicit team-creation and role-message structure. “Subagents” counts distinct non-lead spawned agents, “Max Depth” is the longest directed delegation chain from the team lead, “Verifier Triggers” counts unique task-level non-idle verifier emissions, and “Explicit Repair Loops” counts verifier events that route control back to the solver or request correction.

6pt

These statistics support the narrower process claim made in the main text: the retained runtime-generated workflow traces repeatedly instantiate explicit subagents, reach delegation depth up to 4, and expose verifier-stage events in two structured runs. They do not support an exhaustive count of all recovery loops.

Table 26 extracts the team-initialization sequence from 6.json item 17 (the first item with all four roles instantiated). The trace shows the team lead deciding at runtime how many agents to create, what roles to assign, and what prompt each role receives—a workflow structure that does not exist as a static template but is constructed during the GAIA run itself.

3.5pt

The team-initialization sequence illustrates two key properties of the runtime-generated workflow. First, the workflow structure (4-role pipeline with planner \(\to\) solver \(\to\) verifier \(\to\) writer routing) is not hard-coded in any configuration file but is emitted live by the team lead in a single conversation turn, conditioned on the GAIA task batch. Second, each role’s prompt is embedded at spawn time and specializes the subagent’s behavior from the first message it receives—the verifier, for instance, is instructed from the outset to return failing answers to the solver rather than simply forward everything to the writer. This is structurally distinct from fixed MAS designs in which role assignments and routing are predetermined before any task is seen.

Table 27 continues the trace from team initialization to full execution, following one GAIA instance (task_id: 48eb8242, “How many nonindigenous crocodiles were found in Florida from the year 2000 through 2020?”, ground truth: 11) through each pipeline stage. The trace is drawn from 6.json items 15, 22, 24, and 30, corresponding to the Planner, Solver, Verifier, and Writer turns respectively for this instance.

3.5pt

The execution trace exposes three runtime behaviors that are not present as logged state transitions in the fixed MAS instantiations evaluated here. First, inter-role communication is explicit and visible: each handoff from Planner \(\to\) Solver \(\to\) Verifier \(\to\) Writer appears as a SendMessage event with a typed payload, making the state at each boundary auditable from the retained logs. Second, the verifier applies a format gate at step E7, distinguishing answer delivery from a simple pass-through chain; under the role prompt, a failed check would route control back to the Solver. Third, the Writer applies a skip-if-exists guard at step E10, preventing double-writes even if the same verified answer arrives via a separate message path. These state transitions are the trace-visible correlates of the candidate mechanisms in Table 23; they support the existence of verifier-stage control and explicit state handoff in this retained execution, while leaving their causal contribution to accuracy for ablation.

Table 28 records the full spawn-time role prompts for each of the four agents. These prompts are extracted verbatim from the prompt parameter of each Agent call in 6.json item 17 (steps 2–5 of Table 26). Because the prompts are issued at spawn time and not updated within a run, they represent the complete behavioral contract for each role throughout the session: the only subsequent instructions each agent receives come from the shared mailbox.

4pt

Comparing Table 28 with the reproduced fixed MAS prompt contracts in Table 17 reveals a structural difference in how behavioral constraints are specified. Fixed MAS prompts (Jarvis, ChatEval, LLM-Debate, CAMEL) are written as general-purpose expert personas: they describe a type of reasoning (mathematical, logical, critical) that applies to any task, and the role assignment does not change across instances. The CC-workflow prompts, by contrast, are operationally specific to the current batch: the Planner prompt names the exact JSONL file (level2_part15.jsonl) and prescribes the exact tools (head, sed, tail); the Solver prompt specifies the exact target for routing (“send answer, task_id, and reasoning to the Verifier”); the Verifier prompt specifies an explicit fallback path (“if not, return to Solver noting the issue”); and the Writer prompt encodes an idempotency contract (“if outputs/<task_id>.txt exists, SKIP”). This specificity is only possible because the team lead generates these prompts in real time, after reading the task batch and before any agent begins work.

A second distinguishing feature is the scope of the behavioral contract. In fixed MAS designs, the system-level prompt defines the full behavioral contract; agents may receive task-specific content via the conversation, but their behavioral role does not change. In the CC-workflow, the spawn-time prompt is the full behavioral contract for each role: there is no separate system-level prompt, and the only post-spawn input each agent receives is the mailbox traffic from other roles. The Solver in Table 28, for instance, is aware from its first token that it must route its output to the Verifier specifically—it does not receive this instruction as part of the task, and it cannot deviate from it without violating the prompt it was initialized with. This is the mechanism by which role specialization persists throughout a run without a central dispatcher repeatedly re-stating the routing rules.

8.4.0.1 Scope and limitations of the retained trace evidence.

The trace evidence collected in Tables 26–28 and in the preceding trajectory excerpts supports a narrow set of claims. First, the team-initialization trace confirms that the CC-workflow creates its agent structure dynamically and assigns role-specific prompts at the point of instantiation, not from a fixed template. Second, the task-execution trace confirms that role-to-role handoffs are mediated by explicit SendMessage events, that the verifier stage receives the answer before the writer stage receives it, and that the writer enforces an idempotent write guard. Third, the spawn-time prompt table confirms that behavioral specialization for this particular GAIA batch is embedded in the initial prompt for each agent, not relayed through subsequent messages.

The retained traces are intentionally used for bounded process claims. The detailed tables are drawn from one GAIA batch partition (level2_part15.jsonl), so they establish that this runtime-generated structure occurred, not that the same four-role structure is universal across all CC-workflow batches. Because the team lead generates structure conditioned on the task batch, different inputs could yield different role counts or routing graphs. Trace visibility is also partial: spawn-time prompts and inter-role SendMessage events are visible, while provider-side bookkeeping such as context compaction, token caching, and background retry logic is outside the retained artifacts. The “explicit repair loop” count in Table 25 is therefore a lower-bound event count, not a claim that no unobserved repair behavior occurred.

The evidence tables therefore support the descriptive claim in the main text: the retained CC-workflow traces instantiate explicit subagents, implement a verifier-stage gate, and preserve routing state through visible handoffs. They do not assign the accuracy advantage to one mechanism. In particular, the logs alone cannot separate the effects of the verifier gate, the skip-if-exists idempotency guard, spawn-time prompt specificity, context packaging, and other runtime behaviors. Controlled ablations that disable individual design choices while holding all others fixed would be required for that causal attribution, as noted in Table 23.

8.5 Responsible Research and Artifact Use↩︎

8.5.0.1 Artifact use.

This paper uses public benchmarks and baseline descriptions for research evaluation, with dataset sampling, tool regimes, protocol alignment, and retained trace evidence documented in Appendices 8.1–8.4. We follow the original license and access terms for the benchmarks, baselines, and tool artifacts used in the study, and we do not redistribute restricted source data, credentials, private paths, or provider tokens.

8.5.0.2 Release plan.

During anonymous review, we do not publicly release non-anonymized code, local configuration, or raw traces that could expose private infrastructure, credentials, local paths, or non-public runtime details. Upon acceptance, we plan to release the evaluation code, analysis scripts, anonymized or non-sensitive aggregate statistics, and trajectory tooling needed to reproduce the reported tables and audit the retained process evidence where redistribution is permitted.

8.5.0.3 Risks and AI assistance.

The main empirical risks are the one-run pass@1 setting, the PAE nature of the CC-workflow comparison, and the possibility that agent-count summaries obscure workflow, tool-surface, and context-management effects; these risks are discussed in the main-text limitations and mechanism-evidence notes. AI assistants were used for language editing, code assistance, and drafting support. The authors reviewed and take responsibility for the final claims, evidence, and manuscript text.

8.6 Additional Broad-Benchmark Analyses↩︎

Appendix Figures 6–10 collect the supplementary broad-benchmark views. Figure 6 summarizes the three aggregate rows used in the main-text cost discussion, while Figures 7–10 show task-level accuracy, workflow lift, per-benchmark rankings, and per-workflow benchmark-level lift profiles.

---

1. Work was done during Yuhang’s internship in Westlake University.↩︎