1 Introduction↩︎

Multi-agent LLM systems decompose tasks across specialized agents, tool use, retrieval, and code execution [1]–[3]. This design can improve task coverage, but it also creates long failure trajectories that are difficult to interpret from final answers alone. A failed trace may contain redundant searches, unproductive retrieval, execution errors, weak evidence, or repeated coordination decisions. Final-answer accuracy summarizes the endpoint of the trace; it does not characterize the process that produced that endpoint or indicate where intervention might have been useful.

This paper treats wasted computation as an observability problem. We define wasted computation as tokens, tool calls, execution attempts, and latency spent on runs that do not produce a usable, grounded final answer. The study does not measure hardware energy directly. Instead, it uses cost-correlated proxies available during execution, including token counts, tool-call counts, retry events, evidence gaps, execution errors, and loop indicators. The central claim is that multi-agent LLM systems require failure-aware observability in addition to final-answer evaluation.

We instantiate this claim in a three-agent question-answering system composed of an orchestrator, a search component, and an execution component. Runs are stored as structured event traces. During execution, the orchestrator emits triage labels when trace patterns indicate repeated actions, low information gain, missing evidence, or tool-failure streaks. Offline analysis then derives reliability, grounding, semantic-support, and budget metrics from the same event stream. The goal is not to replace benchmark accuracy, but to explain why computation was spent productively or wastefully before the final outcome.

The contributions of this paper are:

• A failure taxonomy for tool-using multi-agent LLM systems that distinguishes execution failure, orchestration loops, evidence failure, no-final termination, and cost waste.

• A failure-aware observability framework that maps each failure mode to structured online metrics and offline analysis signals.

• An empirical, level-stratified analysis on GAIA validation traces showing how early warnings, grounding signals, and budget proxies relate to failed computation.

2 Background and Motivation↩︎

Tool-using LLM systems such as ReAct-style agents interleave reasoning, retrieval, and action [1]. Multi-agent frameworks extend this pattern by assigning subtasks to interacting components [3]. These systems are commonly assessed by final task success on benchmarks such as GAIA [4]. Final success is an essential metric, but it compresses a multi-step process into a single outcome and therefore provides limited guidance for operational diagnosis.

Distributed systems observability uses structured traces to connect observed symptoms with internal execution paths [5]. Multi-agent LLM systems require a related but distinct form of trace analysis. Their costs are incurred through model calls, tool calls, retrieval, and execution; their failures may be semantic rather than exceptional; and apparently valid final answers may lack evidence support. This motivates an observability layer in which failure modes are explicitly represented as measurable trace patterns.

Recent work studies adjacent mechanisms such as agentic fallback for retrieval [6], tool-augmented memory for long-context QA [7], dynamic retrieval augmentation [8], multi-agent root-cause investigation pipelines [9], comparative retrieval and memory augmentation under constrained compute [10], and iterative multi-agent decision loops in domains such as trading [11]. Prior work also analyzes cross-platform skill equivalence in educational taxonomies [12], highlighting that evaluation criteria and labels can shift across settings. These works motivate stronger agent capability and evaluation methodology. Our focus is narrower: we define trace-level observability signals that explain how existing multi-agent runs spend computation and where failure becomes visible.

3 System and Trace Records↩︎

3.1 Agent Architecture↩︎

The evaluated system contains three roles: orchestration, search, and execution. At each step, the orchestrator selects one structured action from a small action space covering search, page retrieval, attachment reading, code execution, and final response generation. The search component provides retrieval summaries and source references. The execution component supports local attachment processing and restricted Python execution. The experiment uses low-cost model configurations for the orchestrator and search components, with deterministic decoding.

The system records online signals independently of final correctness. These signals include budget updates, repeated action signatures, information-change proxies, loop indicators, and evidence availability. The resulting trace can therefore be analyzed both during execution and after completion.

3.2 Structured Trace Records↩︎

The observability layer maintains an ordered record for each run. Each event contains a run identifier, timestamp, event type, agent role, and structured payload. Event types distinguish messages, tool calls, tool results, metrics, and errors. This model supports metric reconstruction without dependence on hidden in-memory state. The persistent storage backend is an implementation detail; the important design choice is that every diagnostic signal can be recovered from the trace itself.

Table 1 summarizes the main mapping from failure modes to observable signals.

4 Failure-Aware Metrics↩︎

Let a run be \(r\) and let \(E_r\) denote its ordered event trace. We compute metrics at two levels: online metrics emitted during the run and offline metrics computed from the completed trace.

4.1 Execution and Tool Reliability↩︎

Tool error rate measures the fraction of tool results whose status is error: \[\mathrm{ToolErr}(r) = \frac{N_{\mathrm{err}}(r)}{N_{\mathrm{tool}}(r)}.\] Here \(N_{\mathrm{tool}}\) is the number of tool-result events in the run and \(N_{\mathrm{err}}\) is the subset with error status. We separately measure execution success and retry outcomes to distinguish unreliable tools from ineffective recovery behavior.

4.2 Orchestration and Loop Signals↩︎

Orchestration signals characterize whether the agentic process is still making progress. The system represents each action by a normalized signature and uses repeated signatures or alternating repeated signatures as loop proxies. It also records coarse information-gain signals, such as new source references or newly extracted evidence. Repeated low-gain steps are treated as stagnation indicators rather than as direct correctness judgments.

4.3 Evidence and Grounding Signals↩︎

Grounding metrics estimate whether a final answer is supported by the trace. Always-on signals record evidence availability and citation consistency. Deeper offline analysis estimates answer-evidence alignment with a lightweight semantic encoder. The reported artifact uses a local LSA-style representation; the same interface can support Sentence-BERT-style embeddings [13]. The sentence-level support proxy is: \[\mathrm{Support}_{\tau}(r) = \frac{1}{|S_a|}\sum_{s \in S_a} \mathbb{1}\left[\max_{c \in C_e} \cos(f(s), f(c)) \ge \tau\right],\] where \(S_a\) are answer sentences, \(C_e\) are evidence chunks, \(f\) is the embedding function, and \(\tau=0.65\) in the reported analysis.

4.4 Selective LLM-as-Judge Audit↩︎

We use an LLM judge only as an offline grounding audit. For each audited trace, the judge receives the original question, the final answer, and the recorded evidence. A strict prompt asks whether the evidence supports the answer and returns a support score in \([0,1]\). This audit measures grounding, not benchmark correctness. The judge does not use prior knowledge to answer the task and does not control the agent trajectory. We use deterministic decoding and cache responses to avoid repeated calls.

4.5 Wasted Computation Proxy↩︎

For a configurable cost vector, run cost can be approximated as \[C_r = \alpha T_r + \beta H_r + \gamma R_r + \delta X_r,\] where \(T_r\) is total tokens, \(H_r\) is tool calls, \(R_r\) is retries, and \(X_r\) is execution attempts. A wasted-computation ratio can then be defined as \[\mathrm{WCR} = \frac{\sum_r C_r \cdot \mathbb{1}[\mathrm{unusable}(r)]}{\sum_r C_r}.\] In the current experiment, dollar-cost coefficients are left unset. We therefore report token and tool-call counts directly and treat them as computation-cost proxies rather than as monetary or energy measurements.

5 Experimental Setup↩︎

We evaluate on the complete GAIA 2023 validation snapshot available in this workspace. The snapshot contains 165 tasks: 53 level-1 tasks, 86 level-2 tasks, and 26 level-3 tasks. We execute all 165 tasks with the same caps: 8 orchestrator steps, 8 tool calls, 30-second model-request timeouts, and a 25,000-token soft ceiling.

Each level is executed as an isolated batch to avoid mixing controlled traces with older exploratory runs. Trace metadata records the task level, task identifier, model settings, budget limits, timeout, batch name, and source-code fingerprint. Offline analysis reconstructs outcome, evidence, tool, budget, and triage metrics from the structured records.

We define a usable final result as an orchestrator final tool result with status ok. Failed or no-final outcomes include final tool results with status error and runs with no final result message. This outcome definition is intentionally operational: it asks whether the system produced a usable final artifact. It does not claim that the artifact is correct. Grounding quality and benchmark correctness are analyzed separately.

We use two capped audits, both offline. First, a 10-trace grounding audit uses gpt-4.1-mini to compare the agent’s final answer with recorded evidence; this reference comparison uses 10 calls, 8,260 input tokens, and 467 output tokens. Second, a 30-trace correctness audit samples usable finals across levels. For each correctness-audit trace, the grader receives the question, the agent’s final answer, and the GAIA reference “Final answer” for that task, then returns a binary correct/incorrect judgment plus confidence. The grader is instructed to treat the reference answer as canonical and to avoid using external knowledge. Both audits are cached and capped to avoid becoming always-on evaluation strategies.

6 Results↩︎

Figure 1 summarizes the complete validation-snapshot run. On correctness, we report a selective offline audit over 30 runs that produced usable finals (10 per level). Using gpt-4.1-mini as the grader against GAIA validation reference answers, 12/30 audited usable finals are judged correct (level-1: 6/10; level-2: 4/10; level-3: 2/10). We treat this as a sanity check that separates “usable final produced” from benchmark correctness; it is not a full accuracy estimate over all tasks or over failed runs.

6.1 Cross-Level Run Behavior↩︎

Table 2 reports the full-run metrics by task level. Failed or unusable outcomes occur in all three levels: 22 of 53 level-1 runs, 33 of 86 level-2 runs, and 12 of 26 level-3 runs. Average token use rises sharply with level, from 8,152 tokens at level 1 to 16,389 at level 3.

Evidence availability is high in levels 2 and 3, but support does not increase monotonically with task level. This distinction is important: a run may retrieve or read evidence while still failing to produce a trace-supported final answer. The absolute support values should be read as conservative proxy scores rather than human-verified entailment rates. Citation gaps also remain visible: evidence is present without citations in 28.3%, 36.0%, and 46.2% of level-1, level-2, and level-3 runs, respectively. These observations support separating evidence presence from answer support.

6.2 Failure Signals↩︎

Trace labels attribute unusable outcomes to distinct mechanisms. Level-1 failures include insufficient evidence, repeated-action loops, and max-step termination; level-2 and level-3 failures are dominated by repeated-action and max-step termination. LOOPING_WARNING appears in 16 level-1, 31 level-2, and 9 level-3 runs. Lower-frequency labels expose low information gain, tool-failure streaks, evidence gaps, execution without usable output, and blocked near-duplicate search. These labels identify candidate intervention points rather than automatic stop conditions.

6.3 Metric-Layer Comparison↩︎

Figure 2 compares full-run correlations with unusable outcomes against the selective 10-trace grounding audit. Across all 165 runs, any triage label (\(r=0.550\)) and tool-error rate (\(r=0.391\)) are positively associated with unusable outcomes. Sentence support (\(r=-0.594\)) and answer-evidence similarity (\(r=-0.568\)) are negatively associated with unusable outcomes. In the cached judge audit, the same semantic metrics are positively associated with judge support. These descriptive correlations clarify the metric layers without treating any proxy as a causal explanation or an accuracy estimate.

7 Discussion↩︎

The full run supports a layered observability design. Cheap trace signals expose tool instability and stalled trajectories during execution. Semantic metrics separate evidence presence from answer support during deeper offline analysis. Selective LLM-as-judge evaluation is useful for auditing those proxies, but its model-call cost makes continuous use undesirable.

The most useful interpretation of an early warning is not “stop immediately.” Some warned runs recover, so warnings should be treated as intervention opportunities. A production system could respond by changing retrieval strategy, requiring stronger evidence before finalization, reducing the model or tool budget for a low-probability continuation, or routing the trace to a more expensive judge only when cheap signals disagree.

The results also show why final-answer accuracy alone is too coarse for agent engineering. A no-final trace, a weakly grounded answer, and a correct answer that required many redundant calls have different operational meanings. The observability layer makes those differences measurable.

8 Limitations↩︎

This study has several limitations. The evaluation uses one agent configuration and one GAIA validation snapshot; it does not establish generality across models, prompts, or benchmarks. The 10-trace grounding audit and 30-trace correctness audit are descriptive, not gold-standard evaluations. Token and tool-call counts are cost proxies; the study does not measure energy, carbon emissions, wall-clock latency, or provider billing. Embedding similarity can miss entailment, numerical correctness, and cases where evidence is relevant but phrased differently. Finally, the analysis is observational and does not yet evaluate whether an online intervention policy improves accuracy, cost, or both.

9 Conclusion↩︎

This paper introduced a failure-aware observability framework for diagnosing wasted computation in tool-using multi-agent LLM systems. The framework maps tool, execution, orchestration, evidence, and budget failures to structured trace signals. In a 165-trace GAIA validation-snapshot run, unusable outcomes are explained by evidence gaps, repeated-action loops, tool issues, and budget exhaustion rather than by a single failure mode. Cheap metrics can identify wasteful trajectories early, semantic metrics can deepen grounding analysis, and selective LLM-as-judge audits can validate the analysis without becoming an always-on cost. The next step is to turn these diagnostics into controlled intervention policies and measure whether they improve cost-adjusted task success.

References↩︎