Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility
May 07, 2026
Abstract
This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory. To ensure that tool efficiency is well-defined, we also introduce marginal tool utility, a new quantitative metric defined per tool call indicating whether a tool is useful or whether it can be safely removed from the tool suite without affecting accuracy while increasing tool efficiency; in this paper, we determine the sign of marginal tool utility for each tool call in a trajectory using LLM-as-a-Judge. While much prior work has been done to develop techniques that improve tool use by LLMs and design evaluation methods measuring efficiency indirectly using accuracy as a proxy, our work is centered on measuring efficiency directly via the quantitative metric proposed in this paper in post hoc trajectory analyses. It is our intention that this work contributes to the frontier of LLM evaluation research as a springboard for future benchmark designs and agent harness engineering (specifically with regards to creating lean tool suites) that optimize for metrics that complement but are distinct from accuracy.
1 Introduction↩︎
Large Language Models (LLMs) are increasingly used in real-world tasks such as software engineering [1]–[3], mathematical reasoning and autoformalization [4]–[6], and scientific research [7], [8]. However, should LLMs remain simply as language modeling artifacts [9] without access to external up-to-date knowledge [10] and no way to act or modify its environment [11]–[13], they can hardly be called agents [14]. To address this, LLMs are equipped with tools [15], which have become an integral part of the LLM agentic workflows that we see today [16].
While tools are generally useful, not all tools are equally useful [17], [18], this finding acknowledged even by frontier industry labs through launching features such as tool search [19]. To that end, much prior work focuses on developing techniques that improve tool use [20] and on creating benchmarks that such techniques can evaluate against and optimize for [21]. A common ground shared by existing work is that the evaluation metric used is almost always outcome accuracy [22], [23], even if accuracy is not the primary metric of interest, rather it being used as a proxy to measure the efficiency of tool use [21].
Given this, we notice a gap in the literature: there does not yet exist a direct way to measure tool efficiency, which we define as the rate of useful tool calls, given an LLM agent trajectory. While recent works like TRM by [24] take the first steps towards defining reward functions centered on intermediate tool invocations for reinforcement learning to improve tool call quality, we take this idea one step further by enabling post hoc analyses on agent trajectories with an aggregate tool efficiency metric, and keeping to solely inference-time methods easily adoptable by application-layer engineering teams to assist them in designing leaner and more maintainable agent tool suites.
In attempting to define tool efficiency, we must first define what usefulness means with respect to a tool call. We arrive at the idea of marginal tool utility, which is a metric relevant to each particular tool call instance in an agent trajectory that directly determines the usefulness of these tool calls, and whose aggregate per tool determines the usefulness of the tool itself. This definition must be supported by empirical results, specifically those obtained from tool ablations and observing the change in accuracy (or lack thereof) depending on which tool was ablated. To that end, we used the APEX-SWE Observability benchmark, a benchmark in which today’s frontier models still struggle to achieve above \(40\%\) accuracy as reported by [25], and a benchmark that by default includes tools that we hypothesize (and later prove) are necessary (useful) as well as unnecessary (removable without affecting accuracy).
As such, our contributions in this paper are four-fold:
Introduce marginal tool utility \(\Delta_\alpha\) defined per tool call, specifically for pass/fail tasks. Aggregate tool utility \(\sum \Delta_\alpha\) defined per tool predicts whether said tool affects the agent’s accuracy when ablated.
Introduce tool efficiency \(\eta_\alpha\), where a useful tool call \(\alpha_i\) is one with \(\Delta_\alpha(\alpha_i) > 0\).
Implement a simple and cost-efficient method for calculating \(\eta_\alpha\) for pass/fail tasks by using LLM-as-a-Judge to determine the sign of each \(\Delta_\alpha(\alpha_i)\).
Empirically support the correctness of our definitions and LLM-as-a-Judge evaluator through tool ablations on the APEX-SWE Observability benchmark; critically, adding tools with \(\sum\Delta_\alpha \leq 0\) does not increase the accuracy of the agent, while adding tools with \(\sum \Delta_\alpha > 0\) does increase the accuracy of the agent, all as expected.
2 Related Works↩︎
LLM evaluations. There exists a plethora of benchmarks to assess an LLM agent’s accuracy in completing various tasks, including SWE-bench [26], Terminal-Bench [27], BrowseComp [28], and OSWorld-Verified [29]. Benchmarks specifically designed to evaluate an agent’s ability to use tools to correctly complete tasks include MCP-Atlas [30], ToolBench [31], StableToolBench [32], and Toolathlon [33]. While they differ in their exact evaluation method, the aforementioned benchmarks all focus on evaluating accuracy. Even benchmarks to specifically assess the quality of tool use like WTU-EVAL by [21] use accuracy as a proxy to determine whether tool usage is done correctly. The quantitative metrics that we introduce in this paper allow future benchmarks to bypass accuracy altogether, or use them as complementary metrics to accuracy.
LLM tool call optimizations. Many methods have been proposed by existing work to optimize LLM tool usage: reward models [24], [34], self-play reinforcement learning [35], and tool retrieval [36], [37]. Other interesting related work include ART [38] that approach tool call optimization not necessarily from the lens of improving tool usage, but using tool outputs to improve generation dynamically. Several works like [18] conclude that some tasks benefit from tool usage while others do not, suggesting that tool usefulness is task-dependent and not equal across all tools. Similar to [24], we extend this idea and hypothesize that usefulness is not equal across all tool calls, though instead of developing reward models for reinforcement learning, we derive aggregate metrics to be used by application-layer developers conducting post hoc analyses on their agent trajectories to optimize their agent harness [39] without fine-tuning [40]. Moreover, marginal tool utility and tool efficiency can add another dimension to future dynamic tool synthesis methods like Test-Time Tool Evolution [41], which currently calculates semantic similarity between planning steps and tool descriptions to dynamically select tools from a tool library or to generate and refine new tools.
Methods improving agentic efficiency. To the best of our knowledge, tool efficiency is not a well-defined nor quantitative metric prior to this paper. However, this does not mean that there has been no prior work that is focused on improving efficiency in LLM agentic workflows in the broad sense. For instance, Recursive Language Models [42] and FrugalGPT [43] focus on minimizing API cost, LLMLingua [44] on minimizing inference latency, and an instruction-refinement framework by [45] on maximizing Cost-Aware Pass Rate (which is a metric introduced in the same paper). We consider the metrics these methods aim to optimize efficiency-adjacent, reserving the term efficiency for a metric that calculates the rate of useful items as is accepted in physics, where efficiency is the ratio of useful energy to total energy input of a system [46].
3 Methodology↩︎
To enable direct and explicit measurements of efficiency, we first introduce marginal tool utility, particularly in the case of pass/fail tasks. We then introduce tool efficiency, where the usefulness of a tool call corresponds to the sign of marginal tool utility of said tool call.
3.1 Problem Setup↩︎
We formalize a multi-turn LLM agent trajectory as an ordered sequence of token sequences. Assuming the ReAct paradigm [16], which is the paradigm used for the agent harness in APEX-SWE Observability [25], an LLM \(\pi\) alternates between generating reasoning tokens and executing tool calls.
Formally, consider LLM \(\pi\) as well as system message \(s\) and user message \(u\). Given prompt \(p \triangleq (s, u)\), LLM \(\pi\) executes multiple tool calls sequentially and outputs reasoning tokens in between tool calls to synthesize the information it has access to at each timestep. This iterative process terminates once LLM \(\pi\) reaches the maximum allowable step count or calls a terminal tool to produce final answer \(y\). Hence, we define agent trajectory \(\tau\) as
\[\tau \triangleq (p, t_1, r_1, o_1, ..., t_N, r_N, o_N, t_{N+1}, y)\]
where each \(t_i \space (1 \leq i \leq N+1)\) denotes a sequence of reasoning tokens, each \(r_i \space (1 \leq i \leq N)\) denotes a tool request, each \(o_i \space (1 \leq i \leq N)\) denotes the tool output corresponding to \(r_i\), and \(N \in \mathbb{N}\) denotes the total number of steps taken by the agent that is no greater than the maximum step count set as a hyperparameter. We define a tool call as a paired tool request and tool output, or formally,
\[\alpha_i \triangleq (r_i, o_i)\]
To validate our paper’s proposed definitions, we must define the expected correctness of final answer \(y\). In APEX-SWE Observability, final answer \(y\) is the final code patch generated by the agent to solve its automatic program repair task [47]. Correctness is determined by unit tests following the \(\text{FAIL\_TO\_PASS} / \text{PASS\_TO\_PASS}\) methodology inspired by SWE-bench [26]. We define each unit test as function \(f_j \space \colon \space \mathbb{N}^n \rightarrow \{0,1\}\), where \(f_j(y) = 0\) when final answer \(y\) causes unit test \(j\) to fail and \(f_j(y) = 1\) when final answer \(y\) causes unit test \(j\) to pass. The expected correctness of final answer \(y\) over trajectories generated by LLM \(\pi\) is thus
\[\label{eq:e-correctness} \mathbb{E}_{\tau \sim \pi}[\mathbb{I}(f_1(y) \cdot ... \cdot f_M(y) = 1)]\tag{1}\]
where \(M \in \mathbb{N}\) denotes the total number unit tests such that \(1 \leq j \leq M\), and \(\mathbb{I}(\cdot)\) is the indicator function.
3.2 Defining Marginal Tool Utility↩︎
Given trajectory \(\tau\) of an agent tasked to solve a pass/fail task, the marginal tool utility of the \(i^{\text{th}}\) tool call \(\Delta_\alpha(\alpha_i)\) is defined as the difference between the likelihood that the task is solved correctly given tool calls up to and including \(\alpha_i\) and the likelihood that the task is solved correctly given tool calls up to and excluding \(\alpha_i\). Among other methods, this can be determined using repeated policy \(\pi\) rollouts or using LLM-as-a-Judge [48], [49] comparing the before and after trajectories. The latter is more computationally efficient and practical than the former, hence we opt to use LLM-as-a-Judge in this paper, specifically to determine \(sgn(\Delta_\alpha(\alpha_i))\), where \(sgn(\cdot)\) is the signum function. We argue that this is an acceptable method as a tool call \(\alpha_i\) is useful if and only if upon its execution the likelihood of correctness increases \((\Delta_\alpha(\alpha_i) > 0)\). Hence, the additional information afforded by repeated policy \(\pi\) rollouts, that being the exact likelihood difference before and after the tool call, is unnecessary for our purposes.
Formally, consider judge LLM \(\pi^{(J)}\) as well as system message \(s^{(J)}\) and user message \(u^{(J)}(\alpha_i, \tau)\). Given prompt \(p^{(J)} \triangleq (s^{(J)}, u^{(J)}(\alpha_i, \tau))\), judge \(\pi^{(J)}\) generates final answer \(y^{(J)}\), which is a structured output that can be
represented as a triplet \((L, C, R)\), where \(L = \{0,1\}\) denoting two possible marginal tool utility labels (positive or non_positive), \(C = [0,1]\) denoting a float confidence score, and \(R = \mathbb{N}^n\) denoting tokens that read as the judge’s rationale for its classification.
The judge trajectory \(\tau^{(J)}\) is thus simply
\[\tau^{(J)} \triangleq (p^{(J)}, y^{(J)})\]
Given judge \(\pi^{(J)}\), the \(i^{\text{th}}\) tool call \(\alpha_i\), the trajectory up to and excluding the \(i^{\text{th}}\) tool call \(\tau_{1, ..., i-1}\) (i.e., before trajectory), and the trajectory up to and including the \(i^{\text{th}}\) tool call \(\tau_{1, ..., i}\) (i.e., after trajectory), we get \(y^{(J)}_i\). In particular,
\[\begin{cases} L_i = 1 \iff \Delta_\alpha(\alpha_i) > 0 & \text{tool call is useful}\\ L_i = 0 \iff \Delta_\alpha(\alpha_i) \leq 0 & \text{otherwise} \end{cases}\]
Notice that marginal tool utility is defined per tool call. For a particular trajectory \(\tau\), a tool is useful if and only if it has more useful \((\Delta_\alpha(\alpha_i) > 0)\) calls than it has non-useful \((\Delta_\alpha(\alpha_i) \leq 0)\) calls. Equivalently, for all tool calls \(\alpha_i\) of a particular tool,
\[\sum \Delta_\alpha(\alpha_i) \begin{cases} > 0 & \text{tool is useful}\\ \leq 0 & \text{otherwise} \end{cases}\]
We can also define this aggregate for a set of independent trajectories. This is only advisable in practical settings if the independent trajectories share the same tool suite and each corresponds to a task comparable to each other (e.g., each corresponds to an observability task using the same agent harness within the same benchmark).
For reproducibility, we outline the specific system message \(s^{(J)}\) and user message \(u^{(J)}\) in Appendix 8, and note that the particular model used for judge LLM \(\pi^{(J)}\) is GPT‑5.4 on Microsoft Azure [50].
3.3 Defining Tool Efficiency↩︎
Having determined \(sgn(\Delta_\alpha(\alpha_i)) \space \forall i\), we can define tool efficiency as the ratio of the number of useful tool calls to the total number of tool calls in a trajectory. Formally, for trajectory \(\tau\) with \(N\) tool calls,
\[\eta_\alpha(\tau) \triangleq \frac{|\{\alpha_i | \Delta_\alpha(\alpha_i) > 0 \space \}|}{N} \text{, 1 \leq i \leq N}\]
We can thus also define the mean tool efficiency of a set of \(L\) independent trajectories
\[\overline{\eta_\alpha}(\tau^{(1)},...,\tau^{(L)}) = \frac{\eta_\alpha(\tau^{(1)}) + ... + \eta_\alpha(\tau^{(L)})}{L}\]
We hypothesize, and empirically show in Section 4, that tool efficiency increases when a non-useful tool is removed from the tool suite. This is not trivially implied by the definition of tool efficiency, most significantly because the total number of tool calls \(N\) can vary given a change in the tool suite.
3.4 Experimental Setup↩︎
We conduct our empirical experiments to show that the concept of marginal tool utility aligns with what we expect to see in practical settings when performing tool ablations. Particularly, if a tool has positive aggregate tool utility, the expected correctness of the agent’s final answer increases with the inclusion of the tool in the agent’s tool suite; if a tool has non-positive aggregate tool utility, the expected correctness of the agent’s final answer remains constant or decreases with the inclusion of the tool in the agent’s tool suite. We show that these hypotheses are supported by the empirical correctness results.
We evaluate frontier LLMs against the 25 public observability tasks of APEX-SWE [25] available on HuggingFace. [25] noted that this public subset is representative of the whole set of 100 observability tasks, of which 75 are private.
In APEX-SWE Observability, the default agent harness includes native agent tools (bash, search_files, read_file, apply_patch, update_plan) and three read-only MCP tools exposed via bash: Grafana/Loki for logs, Mattermost for discussions between human developers, and Plane for software issues/tickets/specifications. Being familiar with how observability tasks for production software are often solved, we hypothesize that Grafana/Loki is a useful MCP tool with positive aggregate tool utility, while Mattermost and Plane are non-useful MCP tools with non-positive aggregate tool utility.
To prove that hypothesis, we conducted tool ablations to obtain three variants of the agent harness:
default: Full suite of MCP tools (Grafana/Loki, Mattermost, Plane).grafana: Only Grafana/Loki MCP is available.no-mcp: Absolutely no MCP tools are available.
Through these ablations, we are able to verify (1) whether the inclusion/omission of Mattermost and Plane affects accuracy, and (2) whether the inclusion/omission of Grafana/Loki affects accuracy. In Section 4, we present the task accuracy results we obtained and cross-reference them with the aggregate tool utility results of the three MCP tools. We did not conduct ablations on the native agent tools as they are either necessary for the task to be completed (e.g., apply_patch) or for the MCP tools to be executed (e.g., bash).
Agent specifications. We used GPT‑5.3‑Codex on Microsoft Azure [51] and Gemini 3.1 Pro on Google Agent Platform (formerly Vertex AI) [52] to create two independent agent instances. Therefore, we obtained a total of \(25 \cdot 3 \cdot 2 = 150\) distinct agent trajectories.
Compute specifications. The proprietary LLMs were accessed via their respective provider APIs and thus run on their respective providers’ compute clusters. The experiments were otherwise run on AWS Batch using EC2 CPU workers, each as
an isolated containerized job with Docker-in-Docker enabled. Jobs used 16 vCPUs, 110 GB RAM, and a 600 GB gp3 root volume. The AWS Batch compute environment used the r7i.4xlarge, r7i.8xlarge, and m7i.8xlarge instance
types.
4 Results↩︎
In this section, we report the results we obtained from all \(150\) agent trajectories, namely task accuracy, marginal and aggregate tool utility, and mean tool efficiency. We show that task accuracy increases with the inclusion of a tool with positive aggregate tool utility, and tool efficiency increases with the removal of tools with non-positive aggregate tool utility.
4.1 Task Accuracy↩︎
We find that task accuracy does not significantly change when both Mattermost and Plane are removed (default vs grafana) and that task accuracy noticeably decreases when Grafana/Loki is removed (grafana vs
no-mcp), as seen in Tables 1 and 2.
| Variant | Pass | Fail | Accuracy | ||
|---|---|---|---|---|---|
| default | 8 | 17 | 0.32 | ||
| grafana | 9 | 16 | 0.36 | ||
| no-mcp | 6 | 19 | 0.24 |
| Variant | Pass | Fail | Accuracy | |||
|---|---|---|---|---|---|---|
| default | 9 | 16 | 0.36 | |||
| grafana | 9 | 16 | 0.36 | |||
| no-mcp | 5 | 20 | 0.20 |
A breakdown of which specific tasks passed/failed can be found in Appendix 9. Comparing default and grafana, when a task is completed correctly for one it is also generally completed
correctly for the other. A similar relationship exists for tasks that were completed incorrectly.
From these results, note that we can qualitatively claim that Grafana/Loki is a useful tool, while Mattermost and Plane are not. This claim aligns with the nature of the tasks: an agent completing an observability task benefits more from reading logs (primary source of information) than from reading discussions or feature specifications (secondary sources of information). Furthermore, the realities of software development mean that these secondary sources may instead mislead the agent towards investigating parts of the codebase that are actually not broken as misinterpretations of the bug by human developers may be reflected in the discussions and feature specifications may be outdated even during feature implementation.
4.2 Marginal Tool Utility↩︎
Using the judge LLM, we first determine the sign of marginal tool utility of each tool call in each trajectory. We also analyze the output rationales from the judge LLM for its classifications. Then, we calculate the aggregate tool utility of each tool across all trajectories grouped by model and variant.
In Section 4.1, we see that Grafana/Loki affects accuracy when added/removed from the tool suite, and that neither Mattermost nor Plane do. We thus expect the following aggregate tool utilities:
\[\begin{cases} \sum_{\text{grafana}} \Delta_\alpha(\alpha_i) > 0 \\ \sum_{\text{mattermost}} \Delta_\alpha(\alpha_i) \leq 0 \\ \sum_{\text{plane}} \Delta_\alpha(\alpha_i) \leq 0 \end{cases}\]
Investigating the \(50\) trajectories belonging to the default variant (\(25\) generated by GPT-5.3-Codex and another \(25\) by Gemini 3.1
Pro), the marginal tool utility results in Tables 3 and 4 are consistent with that expectation, supporting our proposed definition and implementation. For default with
GPT-5.3-Codex, the aggregate tool utility of Grafana/Loki is \(+25\), Mattermost is \(-35\), and Plane is \(-30\). For default with Gemini 3.1
Pro, the aggregate tool utility of Grafana/Loki is \(+5\), Mattermost is \(-17\), and Plane is \(-28\). For grafana, the aggregate tool utility
of Grafana/Loki is \(58-29=+29\) with GPT-5.3-Codex and \(23-20=+3\) with Gemini 3.1 Pro (no corresponding tables).
In determining the signs of marginal tool utilities, the judge was prompted to output its rationale. The most common rationales to justify \(\Delta_\alpha > 0\) classifications are as follows: Grafana/Loki logs exposed concrete failures (exact error logs that hint at the likely fix); more targeted queries for Grafana/Loki logs at later steps narrowed the investigation and provided more focused signals; and Mattermost and Plane context helped orient the agent towards useful conversation and/or requested feature context. To justify \(\Delta_\alpha \leq 0\) classifications: generic, irrelevant, or noisy context from logs, discussions, and specifications distracted the agent; malformed queries by the agent wasted steps on retries; and valid queries but those resulting in empty outputs wasted steps on pivoting to valid queries that actually gave non-empty outputs.
| Tool | \(\Delta_\alpha > 0\) | \(\Delta_\alpha \leq 0\) | Mean Confidence (\(\Delta_\alpha > 0\)) | Mean Confidence (\(\Delta_\alpha \leq 0\)) |
|---|---|---|---|---|
| Grafana/Loki | 52 | 27 | 0.819 | 0.890 |
| Mattermost | 7 | 42 | 0.734 | 0.927 |
| Plane | 23 | 53 | 0.775 | 0.928 |
| Tool | \(\Delta_\alpha > 0\) | \(\Delta_\alpha \leq 0\) | Mean Confidence (\(\Delta_\alpha > 0\)) | Mean Confidence (\(\Delta_\alpha \leq 0\)) |
|---|---|---|---|---|
| Grafana/Loki | 26 | 21 | 0.825 | 0.870 |
| Mattermost | 3 | 20 | 0.673 | 0.930 |
| Plane | 3 | 31 | 0.607 | 0.917 |
As seen in Figures 1 and 2, tool calls with \(\Delta_\alpha > 0\) occur more often near the beginning of trajectories, and tool calls with \(\Delta_\alpha \leq 0\) occur more often in the middle. Analyzing the agent trajectories in detail, most early calls are high-signal Grafana/Loki calls (so are most early positive marginal tool utility calls), middle calls are mostly spent querying noisy sources and performing broad conversation/specification discovery, and later calls generally surface more specific evidence to support the agent’s code patch generation.
4.3 Tool Efficiency↩︎
Given the signs of marginal tool utility determined by the judge LLM for each tool call, we calculate tool efficiency for each trajectory in default and grafana. Then, we calculate the mean tool efficiency grouped by model and
variant. Since the no-mcp variant has no MCP tools, tool efficiency is undefined for that particular variant. Our results are found in Tables 5 and 6.
| Variant | Useful Call Count (All) | Total Call Count (All) | Mean Tool Efficiency |
|---|---|---|---|
| default | 82 | 204 | 0.359 |
| grafana | 58 | 87 | 0.720 |
| Variant | Useful Call Count (All) | Total Call Count (All) | Mean Tool Efficiency |
|---|---|---|---|
| default | 32 | 104 | 0.367 |
| grafana | 23 | 43 | 0.593 |
Tool efficiency increases when Mattermost and Plane are removed. This finding is also consistent with the conclusion that these two MCP tools are unnecessarily included in the agent’s tool suite and thus can be considered redundant given the nature of the task to solve.
5 Discussion↩︎
We identify several promising directions for future work the community can engage in that leverages marginal tool utility and tool efficiency.
Minimizing sub-optimal middle calls. The trend that middle calls are often sub-optimal (see Section 3.2) is a useful finding that may inspire future LLM reinforcement learning techniques, similar to those by [24], [53]–[55], with marginal tool utility as a component of the reward function. Alternatively, marginal tool utility can be used for inference-time optimizations, like designing agents that can backtrack [56] should the trajectory thus far show some number of consecutive tool calls with non-positive marginal tool utility.
Designing leaner tool suites. As seen through our tool ablations of APEX-SWE Observability, tool suites can be more bloated than necessary, as has been surfaced by [57] albeit from a different perspective. The tool efficiency metric can be directly used to assess the overall health of a tool suite such that it remains as lean and maintainable as possible, and the aggregate tool utility of each tool to decide whether to preserve the tool in improved versions of the harness. This enables the creation of tool suites that contain tools that serve mutually exclusive purposes. Tool descriptions and/or execution bodies [41] alone may not indicate redundancy between tools, as we have seen in Tables 1 and 2, hence marginal tool utility and tool efficiency can serve as extra sources of information to uncover such redundancies.
More informed harness engineering. We observe in Section 4 that using different backbone models for agents can yield the same task accuracy with different tool efficiencies. We posit that this is primarily a product of different model training methodologies as the agent harness remains a constant variable, but an important finding to note nonetheless as this supports the view that harness engineering must be informed by each agent’s particular backbone model. In other words, a one-size-fits-all approach to harness engineering may be sub-optimal, hence workflows that are designed to automate harness engineering [58] can benefit from our proposed metrics.
Self-improving agents indexing on tool efficiency. Taking automated harness engineering one step further, the agents themselves can recursively self-improve [59], [60] without relying on another agent that optimizes the harness offline [61], [62]. Not only can this online improvement be based on the usual metrics like task accuracy, tool efficiency and marginal tool utility can be leveraged mid-execution to provide rich reward signals to optimize context (including tool descriptions, input schemas, and output data shapes) or update model weights.
6 Limitations↩︎
While confidence scores returned by an LLM as decoded tokens are not the most rigorous [63], [64], the judge’s confidence score per classification in Tables 3 and 4 indicate that the judge is generally more confident on non-positive classifications. We did not investigate why this is so, nor did we attempt to balance the confidence scores, as we view this work as outside the scope of this paper. Alternatively, a different LLM-as-a-Judge implementation could have been used, such as one replacing the language modeling head of the judge with a binary classification head [24], though this requires using an open-weight LLM.
Most notably, we only ran tool ablations for read-only tools (the three MCPs are all read-only). We did not remove tools with write privileges (e.g., apply_patch) from the agent’s tool suite as we suspect that that would have prevented the agent from completing the tasks at all. Having said that, we seek to find ways in which we can assess the marginal tool utility of write tool calls, either directly or indirectly (the latter case by investigating how read-only tool calls are affected by what was written using the write tool calls), in future experiments.
7 Conclusion↩︎
We introduce tool efficiency and marginal tool utility, both being new quantitative metrics that are relevant in determining the usefulness of tools and thus in constructing the leanest possible tool suite for an LLM agent given a particular task. Our findings suggest that intuitive judgments regarding a tool’s usefulness as well as empirical tool ablation results using accuracy as a proxy in discerning usefulness/necessity of different tools agree with conclusions drawn from analyzing marginal tool utility and tool efficiency values.
All in all, we look forward to what our new definitions enable in the development of LLMs and LLM agents. We hope that the literature matures not only in terms of new methods, architectures, and workflows, but also in new evaluation dimensions against which we can measure the holistic quality of these new artifacts. As well-defined quantitative metrics are more tractable to optimize for, we believe that evaluation/metrics research will accelerate progress in LLM research and engineering further, providing the necessary direction and focus guiding iteration cycles of novel methods.
Societal Impacts. As large language models and agents built on them become more capable, many have raised concerns regarding our over-reliance, even dependence, on them. While this work does not introduce a new artifact directly relevant to such concerns, this work does enable new artifacts to be more capable than their existing versions today. We want to acknowledge that, and it is our hope and belief that agents and humans are able to work in tandem, each tackling a dimension of work the other is less-suited for. The onus is on us, those at the frontier of the development of this technology, to understand and thus educate the public on what those dimensions are.
The author conducted this work in her capacity as Founding Research Engineer at Foam. The author thanks Perla Gamez and Luke Mercado for their support throughout the research process. All funding for this work comes from Foam.
8 Prompt for LLM-As-A-Judge for Marginal Tool Utility Sign↩︎
\(\newline\)
You are an expert evaluator of agent trajectories completing a SWE bugfix observability task.
Your task is to determine whether a tool call increases the likelihood the bugfix task is solved correctly. This is measured by a metric called marginal tool utility (MTU).
Concretely, you are tasked to determine the sign of marginal tool utility (MTU) for one tool call.
Definition:
MTU(i) = p(task solved correctly | tool calls 0..i) - p(task solved correctly | tool calls 0..i-1)
Equivalently:
MTU(i) is positive if the likelihood of the bugfix task being solved correctly strictly increases after tool call i is added into the trajectory. MTU(i) is non-positive otherwise.
You are given:
BEFORE context: state of trajectory before tool call i executes.
AFTER context: state of trajectory after tool call i completes.
The specific tool call and tool result.
Output strict JSON only:
{
"label": "positive" | "non_positive",
"confidence": float between 0 and 1,
"rationale": "brief explanation"
}
Use "positive" only when the call clearly increases solve likelihood.
Use "non_positive" when likelihood is unchanged or reduced.
\(\newline\)
Decide MTU sign for this tool call.
Target tool call:
tool_call_id: {TOOL_CALL_ID}
tool_name: {TOOL_NAME}
arguments: {ARGUMENTS_TRUNCATED}
tool_result_excerpt: {TOOL_RESULT}
BEFORE context (through tool calls 0..i-1 completion):
=== BEFORE START ===
{BEFORE_CONTEXT}
=== BEFORE END ===
AFTER context (through tool calls 0..i completion):
=== AFTER START ===
{AFTER_CONTEXT}
=== AFTER END ===
Return strict JSON only.
9 APEX-SWE Observability Per-Task Correctness Breakdown↩︎
References↩︎
Alternate email address:
nyx@berkeley.edu↩︎