1 Introduction↩︎

Global private investment in generative AI grew to $33.9 billion in 2024, while 78% of organizations reported AI usage [1]. Despite this, empirical evidence regarding its granular impact on specific work tasks remains opaque. In combination with the deep embedding of AI in the economy, this has the potential to create "excessive automation" or the automation of work tasks that displace labor and depress wages without generating significant gains in productivity [2].

To alleviate this risk, efforts to snapshot the capabilities of AI in the workforce have increased. Human surveys [3], [4] can provide an estimate but suffer from under-reporting bias [5]. Controlled experiments [6], [7] can alleviate this issue but offer insights that are limited to singular occupations. For a broader picture, [8], [9] and [10] have developed systems to map user-LLM chats to O*NET [11] tasks or work activities; creating aggregate statistics of what labor activities individuals use AI for. While this method has illuminated patterns in AI usage, it suffers from two shortcomings. First, prior work has used proprietary chat data, making it difficult to corroborate findings. Second, simply categorizing chats into an occupational task or work activity does not measure the performance of the LLM in that task or activity. [9] create a preliminary analysis by analyzing user chats for sentiment; however, like other works, the chats are proprietary, and sentiment may not be uniquely determined by the model’s accuracy.

Other works utilize human generated and graded deliverables to measure LLM economic value. [12] design GDPVal, a benchmark constructed from 1,320 tasks across 44 occupations representing the top 9 sectors of U.S. GDP. They find that AI has a win-rate exceeding 50% against experts in certain occupations. [13] produces contrasting results; they crowd-source complex projects from free-lancers and find that current agents can only automate less than 3% of projects. The staunch disagreement is partially due to the creation of custom tasks by [12] and [13], and it reveals a need for further investigation.

In our view, obtaining a snapshot of the labor impacts of AI involves seeking answers to two questions.

1. Which occupations are using AI?

2. Do these occupations benefit from the use of GenAI, and are they at risk of significant job loss?

To help provide an answer to these questions, we build two systems for studying AI adoption and capabilities. First, we construct an open source economic index that measures the occupational sectors with high LLMs utilization (Figure 4 (a)), and the depth of AI penetration into the economy (Figure 4 (b)). Throughout, we will discuss how these findings align with prior work [8]–[10]. Second, we build an economic benchmark that confers two advantages over GDPval [12] and the remote labor index [13]. We test the agents on O*NET tasks, removing the ambiguity that arises from creating custom tasks. Furthermore, our system is neither human generated nor evaluated, allowing us to create a greater number of benchmark scenarios per occupation. Using our benchmark, we evaluate the performance of agents in scenarios covering 9 occupations. Figure 1 encapsulates our findings; we plot the AI agent capability and real-world AI adoption across nine occupations. The automative plot (blue) shows the share of single-turn scenarios achieving perfect workflow completion, reflecting autonomous task execution. The Augmentative plot (red) shows the share of multi-turn scenarios achieving perfect completion, reflecting collaborative task execution with a simulated user. Real-world task usage (teal) shows the share of scenarios whose constituent O*NET tasks appear at least four times in our open economic index. Overall, we find that theoretical capabilities outpace adoption in the index, but complete occupational replacement risks may be minimal as user-AI collaboration outpaces purely autonomous task execution. Further evidence of the need for human oversight is given in Figure 9, where we see the tested agent still has non-negligible tool calling error and hallucination rates.

1.1 Related Work↩︎

Our work builds on task-centric labor analysis [14]–[16], empirical studies of AI adoption [3], [4], [6], [7], and agentic benchmarks such as BFCL [17], MCPBench [18], and \(\tau\)-bench [19]. A full review of related work is provided in Appendix 11.

2 Open Source Measurement of AI Adoption↩︎

To create an open source index of how individuals utilize large language models across the economy, we design a semantic search system that utilizes gpt-oss-120b [20] and the Qwen3-Embedder [21] to map user-LLM conversations to tasks in the O*NET database.

2.1 Data and Methodology↩︎

We begin with the WildChat-4.8M [22] dataset available on HuggingFace. WildChat-4.8M is a collection of 3,199,860 non-toxic inputs and responses between human users and various versions of ChatGPT. Exchanges may be over 25 turns, and model types include gpt-3.5-turbo, gpt-4, and even reasoning models such as o3-mini. The flip side of the index is the O*NET database; a taxonomy that organizes occupations according to their tasks, required knowledge levels, and skills. Each occupation is associated with tasks that are further embedded in a four level hierarchy that also includes 2,087 detailed work activities (DWAs), 332 intermediate work activities (IWAs), and 41 work activities (WAs).

To make the raw chat data more usable for measuring work-related activity, we apply three data filters. The first and second are simple language and uniqueness filters that isolate the unique English chats, leaving us with 1674316 total chats for analysis. Similar to [8] and [9], many of the English chats in the WildChat dataset are likely not related to any type of work activity. It is unlikely that these can be given any meaningful mapping to an O*NET task. To prevent this from decreasing the accuracy of our index, we design a simple filter for “Occupational Relevancy". To perform the filter, each chat is fed to gpt-oss-120b along with a prompt that asks if the user’s request is "occupationally relevant (related to work, professional tasks, or career activities)". The results, given in Table [tbl:tab:32OR], are remarkably similar to the June 2024 results presented by [9]. WildChat chats were collected by providing free ChatGPT access to users and were subsequently filtered for toxicity. It is worth noting that [9] also reported increased relevancy rates in 2025, suggesting that LLM usage practices are not static.

The second step in our analysis is to map each of the 789768 occupationally relevant chats to O*NET task(s). To do this, we implement the following three step semantic search process:

1. For each occupationally relevant chat, we have gpt-oss-120b summarize the user’s chat(s) into a single overarching request for the LLM.

2. Using the Qwen3-Embedding-0.6B model, we create instruction-augmented embeddings of each chat summary, embeddings of each O*NET task, and map 3 tasks to each chat using a nearest neighbors search.

3. Gpt-4o-mini serves as a filter for each (chat summary, 3 tasks) pair and is asked to keep any task where “the core skill or activity overlaps".

As a model validation step for the chosen embeding model, we compute the cophenetic correlation between the embedding distance and the O*NET hierarchy graph distance. Graph distance is computed as the shortest path (hop count) in the undirected O*NET hierarchy. In Figure 3 we demonstrate the distance between two tasks with the dashed red path; cophenetic correlation is given over all \(\binom{|\text{GWA+IWA+DWA+Task}|}{2}\) node pairs across all levels. Table [tab:cophenetic-correlations-model] shows good results for Qwen embedding models of two sizes; both handily beat the model³ [8] selected for the task.

This two step process is novel in the context of LLM economic index production. [8] create a custom taxonomy using clustering and perform an LLM powered tree search to map conversations to O*NET tasks. [9] and [10] only map chats to the IWA level of the O*NET hierarchy, losing the granularity of task-level mapping.

2.2 Results↩︎

Of the 789768 occupationally relevant chats, 668380 chats have at least one represented task; with a split of 416,611 having 3 tasks, 169,835 having 2 tasks, and 81,934 having 1 task. Upon mapping chats to O*NET tasks, we aggregate to the occupation level. For each chat, a given occupation is assigned a weight proportional to the count of its tasks within the range of that chat. The representation of a given occupation in the economic index is then calculated via a weighted sum. Overall, Figure 4 (a) (produced by isolating chats geo-located to the U.S) provides several insights⁴. Relative to the US economy, LLM usage is concentrated in the arts, business, financial work, programming, mathematics, and subject-specific sciences. This aligns with the LLM usage that [8] measured across the economy. Surprisingly, the white collar field with the lowest LLM adaptation is legal services, a conclusion that Anthropic also reached [8]. Compared to the Anthropic economic index, we find relatively less usage in educational tasks and more in protective service tasks, which is contrary to the findings in [8]. Protective services include translation, which accounts for the mappings in our chat database. Among the “LLM prevalent" occupations, our index places the greatest weight on the Arts, while [8] places the greatest weight on computer science tasks.

We can also use the index to assess the depth of AI penetration into the economy. [8] measure this by calculating the percentage of occupations that have at least \(p\) tasks in the image of at least \(t\) conversations, where \(p\) can range from \(0\) to \(1\). We present the same calculation in Figure 4 (b) using our index, where we adjust the threshold to account for the smaller number of conversations present in our dataset. Compared to prior work, our results are bearish on AI depth, with only 0.4% of occupations showing AI usage for at least 75% of their tasks (as compared to the estimated 4% of occupations given by [8]). The skills utilized in the tasks attempted by AI are also of interest. We infer skill usage using the O*NET data structure and our open economic index. O*NET rates every occupation on 35 specific skills (like "Critical Thinking," "Programming," or "Social Perceptiveness"). For a given skill, we calculate how many chats map to an occupation that rates that skill as having an importance of at least 4. Figure 5 (a) presents the skills most frequently required to perform the occupations listed in our open economic index. Cognitive skills such as writing, programming, reading comprehension, critical thinking, and active listening are dominant in the index. As with the Anthropic index [8], our open source economic index can be compared with the index produced by OpenAI [9]. Their top appearing work activities include “Thinking Creatively",”Documenting/Recording Information", “Making Decisions and Solving Problems", and”Working with Computers". Figure 5 (b) provides the calculation we perform using WildChat data and our search system; tasks matched to conversations are aggregated to Work Activities (WAs) using the O*NET Work Activities taxonomy. Except for working with computers, all top work activities from OpenAI appear within the top ten work activities we produce [9].

3 Benchmarking the Economic Capabilities of Agents↩︎

The open economic index characterizes which occupational tasks appear in real-world user–LLM interactions and how frequently they arise across occupations. However, the presence of a task in observed usage does not, by itself, indicate whether contemporary AI systems can reliably or autonomously perform that task. User interactions may reflect exploratory use, partial assistance, or heavily supervised workflows, and do not directly measure task completion quality or execution correctness.

To move beyond descriptive adoption and toward a functional assessment of AI capabilities, we require a complementary evaluation framework that can measure how agents perform on the same classes of tasks as observed in real-world usage. In particular, such a framework should (i) be grounded in the same occupational task taxonomy (O*NET) used by the index, (ii) evaluate multi-step task execution rather than isolated responses, and (iii) distinguish between autonomous execution and collaborative, user-steered workflows. Motivated by these gaps, we next introduce an economic capability benchmark that translates O*NET tasks into tool-mediated workplace scenarios and directly evaluates agent performance on workflow completion, tool use, and grounding. We build a system to generate agentic evaluations that are grounded in O*NET data and real or virtual MCP servers.

We test agents in two modes: single-turn (autonomous execution) and multi-turn (with a simulated user providing collaborative steering).

3.1 Scenario construction↩︎

The benchmark scenarios are grounded in the O*NET database. The tool ecosystem for the benchmark consists of Smithery⁵ MCP (Model Context Protocol) servers; we crawl every such server, producing an LLM generated analysis and quality assessment. Additionally, for stability and increased tool coverage, we implement a benchmark mode that uses simulated MCP servers. Details are provided in Appendix 5.

Beyond filtering, the second core challenge is connecting each O*NET task to the MCP servers whose tools are most likely to help accomplish that task. We perform this match semantically, using the server analysis generated in step one, via a Qwen3-Embedding model (thresholding a valid combination at a cosine similarity of 0.5). Scenarios are generated by enumerating (occupation, N (task, server) ) tuples and prompting an LLM to produce a grounded workplace request with a target workflow and a granular answer key of required tool calls, including both tool names and specified arguments. An example is given in Appendix 7.

For multi-turn evaluation, we generate information-withheld variants where critical parameters are omitted from the request, requiring the agent to ask clarifying questions Appendix 7. A simulated user agent provides answers and collaborative steering via role inversion (Figure 11, Appendix 8).

3.2 Scenario evaluation↩︎

Each trajectory is evaluated using LLM-as-judge across five dimensions (full methodology and rubrics in Appendix 6). Tool Call Accuracy (binary) measures whether the agent called correct tools with correct arguments, following the BFCL approach [17] with LLM-judged functional equivalence. Workflow Completion (1–5, Table 5) assesses whether the agent followed the intended multi-step tool sequence with proper inter-tool data flow. Grounding (1–5, Table 6) evaluates whether the agent’s claims are supported by tool outputs rather than hallucinated ones.

For multi-turn trajectories, we add two dimensions alongside tool calling and grounding: Autonomy (1–5, Table 7) measures how independently the agent completes the workflow without user steering (analog to workflow completion), and Follow-up Quality (1–5, Table 8) assesses how well the agent identifies and asks for withheld information before proceeding.

3.3 Results↩︎

We select 9 occupations that appear frequently in our open economic index and have a sufficient number of matched MCP servers and tasks to generate scenarios grounded in 3 pairs of each. For each, we sampled the possible sets of 3 (task, server) pairs, resulting in a total of almost 5000 scenarios generated by Kimi-k2.5 [24]. These scenarios cover 130 tasks, 100 servers, and over 1000 tools.

Single-turn results. We test Kimi-k2.5⁶ with the OpenAI agent harness and virtual tools in each scenario (Kimi also acts as the tool and user). Figure 6 provides the score distributions: about 60% of required tool calls are executed correctly, mean workflow completion is 4.08/5, and mean grounding is 3.7/5. Per-occupation score curves, Figure 7, show that only a small proportion of scenarios achieve high scores across all dimensions simultaneously. We also categorize agent errors (Figure 15 and Table 10, Appendix 10), drawing on TRAIL [25].

Multi-turn results. We generate information-withheld versions of the same scenarios, producing approximately 3200 valid multi-turn trajectories. Figure 8 shows that despite mixed follow-up quality (modal outcome: partial clarification), agents often autonomously complete the workflow, suggesting they can infer enough information even with sub-par information gathering. Figure 9 reveals that user collaboration improves workflow completion but harms tool call accuracy and grounding, suggesting that user-AI collaboration should still include granular oversight on intermediate steps.

4 Discussion and limitations↩︎

We have developed open source methods for measuring AI adoption and productivity across the economy. For adoption, we used the WildChat dataset and our own semantic search method to replicate the closed indexes produced by [8] and [9]. To measure productivity, we developed a systematic benchmark that tests agents equipped with real or hallucinated MCP servers on scenarios grounded in O*NET tasks. The results have two main implications for the labor market. First, adoption rates are lower than the proportion of scenarios that an agent can correctly execute. Second, current agents perform best in macro level workflows when augmented by a user, though this comes with trade offs along other evaluation dimensions. Many open questions remain. Replacing the WildChat dataset with another collected chat dataset for more modern models would provide a more recent snapshot of adoption. Additionally, one can always enhance the available tools for the agents in our scenarios. In Appendix 5.1, we analyze the coverage of O*NET software commodities by MCP servers, finding at most \(33\%\) coverage. This suggests that better answers to our primary economic questions could be obtained by enhancing the server/tool list. In particular, file (pdf, slides) editing tools would allow the scenarios to incorporate a “deliverables" aspect that is present in GDPval [12] and RLI [13].

5 Smithery MCP Server Analysis↩︎

Our pipeline for setting up servers is as follows.

1. We crawl smithery and produce an LLM generated analysis of 1520 servers. Additionally, we attempt to connect to each server and list its tools, producing a total of 627 servers that can actually be connected to by an agent without additional enterprise purchases or specific API keys.

2. Next, for each of the 627 validated servers, we have an LLM attempt to call each tool with realistic inputs and evaluate if the response returns "real, non-error data consistent with its described purpose." Of the 627 validated servers, 416 had at least one tool of quality. In aggregate, about 33% produced quality outputs.

Tables 2 and 3 break down the results further. The pipeline was robust with few agent errors, the majority of which were mcp servers exceeding the context length of kimi-k2. Our analysis includes an additional check that has not yet been discussed: we check servers for “statefulness". A server is defined as stateful if it "has tools that allow an agent to store information, modify existing data, or change the internal state of the system". It is stateless if "it provides read-only access; allows fetching, searching, or processing data without permanently altering the system’s state or storing new information across calls". The intent here is to measure the complexity of the tools in another dimension; stateful tools are more realistic for what an autonomous AI needs to utilize. We see that stateless tools outnumber stateful tools at a 2-to-1 ratio.

5.1 O*NET commodity coverage↩︎

As a test of the “breadth" covered by MCP servers, we measure how well they cover the 137 O*NET technology skill "commodities" (standardized software categories like "Accounting software", "CRM software", etc.). Our methodology remains the same: use Qwen3-Embedding-8B to embed both commodity definitions and server/tool descriptions, then match via cosine similarity (threshold 0.5). For example, a commodity embedding may resemble Instruct: Retrieve software tools that provide similar functionality as software in the following category. Query: Software to enable business functions related to accounting and management For more generous retrieval, we also try querying each commodity example, and matching if any examples exceed the similarity score threshold.

Instruct: Retrieve software tools that provide similar functionality... Query: Software to enable business functions relating to accounting and management such as QuickBooks

Instruct: Retrieve software tools that provide similar functionality... Query: Software to enable business functions relating to accounting and management such as Sage 50

The results for the validated MCP servers are given below.

5.2 Optional Tool Simulation↩︎

At scale, Smithery MCP servers are not free to call (it is free up to a certain usage). Given this, we also implement a virtual tool system that can ingest MCP server documentation and produce realistic tool outputs (using an LLM).

6 Evaluation Methodology↩︎

After an LLM agent runs on a benchmark scenario (producing a trajectory of tool calls and responses), the trajectory is evaluated across three complementary dimensions using LLM-as-judge.

Tool Call Accuracy: This dimension mimics the Berkeley Function Calling Leaderboard (BFCL) and evaluates whether the agent called the correct tools with the correct arguments. All calls from the agent’s trajectory are extracted. A basic auto-scoring handles failure cases without LLM involvement: a score of 0 is automatically assigned when a scenario fails catastrophically (e.g., agent max turns exceeded) or when the agent never called a tool present in the answer key at all. Next, as is done in BFCL, we compare each tool call in the answer key to the last actual call in the trajectory of the same tool name (in the case where M calls of the same tool are expected, an array of the last M calls is compared). An LLM judge is prompted to determine functional equivalence; a correct call may match the expected arguments exactly or may be a functionally equivalent rephrasing (e.g., different capitalization, minor wording variation, same intent). The judge outputs a binary score: 1 (correct) or 0 (incorrect), along with reasoning.

Tool Workflow Completion: This dimension evaluates whether the agent followed the intended multi-step tool-use workflow, focusing on the macro-level organization of tool calls rather than on granular tool calls. The LLM judge is provided with the query, the expected tool workflow, a separate analysis of how each tool contributes, and a condensed agent trajectory showing reasoning, tool calls, tool outputs, and the final response. The judge evaluates: (i) Was the sequence of tools correct? (ii) Did data flow between tools correctly? (iii) Was the final response a coherent integration of the workflow results? A score between 1 and 5 is produced using the rubric given in Table 5.

Grounding: This dimension evaluates whether the agent’s response is supported by the tool outputs that the agent actually received, as opposed to hallucinated responses. The judge is provided with the query and the agent trajectory, including tool calls and any claims in response to the query. The judge is instructed to enumerate grounded claims (traceable to tool outputs) and ungrounded claims (asserted without evidence), then assigns an overall score based on Table 6.

For multi-turn trajectories, two additional dimensions are evaluated. Autonomy measures how independently the agent completed the workflow without user steering, and Follow-up Quality assesses how well the agent identified and asked for deliberately withheld information before proceeding with tool calls.

6.1 Evaluation Rubrics↩︎

7 Scenario Examples↩︎

Below are examples of single-turn and multi-turn (information-withheld) benchmark scenarios.

Occupation: Human Resources Specialist
O*NET Tasks:
  1. Provide management with information or training related to interviewing,
     performance appraisals..
  2. Schedule or administer skill, intelligence, psychological, or drug tests
     for current or prospective employees.
User Request:
  "I need to set up a mandatory Python skills assessment for Sarah Chen from
   Software Engineering. She has been selected for a technical skills audit
   and I need to verify her programming competency. Please search for her in
   our employee system, check her training leave balance (she'll need 1 day
   for the assessment), create a Python fundamentals question about data
   structures, build a single-question assessment quiz, and schedule the exam
   for her on February 15th, 2024 (9 AM to 5 PM window). Make sure to assign
   her to the exam and provide me with the assessment URL so I can send it to
   her manager."
Tool Analysis:
  The EduBase server provides assessment and exam management tools that align
  with HR tasks of screening and testing employees and candidates. The Leave
  Manager server tracks employee time off, supporting availability checks
  before scheduling assessments.
Cross-Tool Workflow:
  1. [Parameter dependency] Search for the employee to confirm identity and
     obtain the employee ID.
  2. [Parameter dependency] Use the employee ID to check leave balance,
     ensuring availability for testing/training days.
  3. [Parameter dependency] Create a skills assessment quiz in EduBase and
     assign it to the employee, using the confirmed employee identifier to
     grant exam access.
  4. [Conditional routing] Retrieve quiz results to determine next steps:
     if score < 80

User Request:
  "I need to move fast on filling that senior opening. Can you pull everyone
   who's shown interest, schedule them for an interview soon, reach out with
   a call and a confirmation email, and then generate the hiring paperwork
   draft once they accept?"
Withheld Information:
  - job_requisition_title: The exact job title being recruited for, used to
    filter leads/contacts. Value: "Senior Data Engineer"
  - interview_datetime: ISO datetime for the interview slot to be scheduled.
    Value: "2024-07-10T14:00:00Z"
  - hiring_manager_id: HubSpot owner ID of the recruiter who owns the meeting,
    calls, and emails. Value: "901234567"
Target Follow-Up Questions:
  1. "What job title are you actively recruiting for right now?"
  2. "When do you want to schedule the initial interview (date and time)?"
  3. "Who is the hiring manager or recruiter that should own the interview,
     calls, and emails?"

8 User Implementation↩︎

There are two critical details for implementing an agent that acts as a realistic user. The first is role inversion; the user operates in a session where it believes it is the assistant, and agent messages arrive as messages from the user (possibly transcribed with some indication that it is the assistant responding). Second, the user agent does not see any intermediate reasoning steps or tool calls. Figure 11 demonstrates both of these details.

9 Additional Plots and Tables↩︎

9.1 Economic Index↩︎

Figure 12 explores the relationship between AI usage and median wage across occupations, providing additional context for the adoption patterns described in the main text.

9.2 Benchmark Evaluation↩︎

Figures 13–7 provide supplementary views of benchmark performance across occupations and evaluation dimensions.

10 Agentic Error Taxonomy↩︎

We categorize agent errors across trajectories using reasoning outputs from the evaluators and gpt-oss-120b. Tool-call errors combine three sources: extraneous tool calls, missed tools in which a required tool was never called (identified by automated scoring), and argument-level errors in which the tool was called but with incorrect arguments (LLM-judged calls with a score of 0, classified into three categories: hallucinated argument, missing required argument, and wrong argument value). Workflow and grounding errors are derived from evaluator reasoning strings for agentic trajectories scoring below three. Along the workflow dimension, we categorize into Goal Deviation (“agents distracted from primary objectives”), Task Orchestration Failure (“failure in the ordering of tasks executed”), and Context Handling Failure (“agent inability to pass data from one task to the next”). We derive several of these categories from TRAIL [25]. Table 10 provides full definitions; Figure 15 shows the frequency distribution of each error type.

11 Related Work↩︎

Task-centric automation analysis: The idea of studying labor through discrete tasks is not new. [14] argue that computer capital can replace or augment labor capital depending on the complexity of the discrete task; [15] posits that tasks can illuminate the interaction between wage growth and the available labor supply, offshoring, and technology. Other work utilizes the task-centric lens to argue that automation increases labor demand by creating new tasks or augmenting labor on existing tasks [16], [26].

Measuring and Forecasting AI Usage: Our work also builds on prior efforts to measure how AI is currently being used and to forecast its future impact on the economy. Empirical studies on AI usage for singular occupations have been conducted for writing [7], customer service [27], software engineering [6], [28], translation [29], data science [30], legal analysis [31], and consulting [32]. [3] and [4] conduct surveys to measure AI usage in the workforce. Forecasting efforts for automation through computerization [33] and traditional machine learning [34] involve using human annotation to label the automation potential of O*NET occupations and work activities. By linking AI applications with O*NET abilities [35], estimate that law and finance are the industries most exposed to LLM automation. [29] forecast LLM productivity in translation using scaling laws. [36] and [37] provide a theoretical macroeconomic framework to forecast the potential productivity gains and GDP growth from AI. Indeed hiring data is used by [38] to forecast the impact of AI on the demand for various labor skills. [8], [10], and [9] pioneered the idea of mapping chat data to O*NET objects to produce a snapshot of AI usage.

Benchmarking LLM tool usage and agentic capabilities: We test AI on O*NET tasks by assessing tool usage. The Berkeley Function Calling Leaderboard (BFCL) is the standard for measuring technical agentic tool calling accuracy via a strict abstract search tree method [17]. Tools on our MCP servers often have arguments that require “natural language” inputs. Given this, we differ and use LLM-as-a-judge to evaluate tool calls. Several other tool-calling datasets and benchmarks have been introduced. Using StableToolBench [39], [40] create a series of verifiable tool calling questions and answers. With the RapidAPIHub, [41] introduced tool-bench, an instruction-tuning dataset and benchmark for tool usage. [42] introduces the practice of designing tool calling simulations by connecting agents to MCP servers hosted on Smithery, but does not attempt thorough evaluations of the trajectories. [18] adds this, building a systematic benchmark to test models on large scale tasks that require the use of 28 possible MCP servers. Our systematic economic benchmark builds on this by grounding the tasks in O*NET data, utilizing 416 MCP servers. \(\tau\)-bench presents a more economically grounded evaluation by testing agents in domain specific scenarios (Retail and Airline) where they must interact with a simulated user [19]. Agentbench adds realism by incorporating a simulated user into agentic benchmarks [43].

12 Prompts↩︎

This appendix contains the full text of all LLM prompts used in data generation and evaluation. Template placeholders appear in curly braces (e.g., {OCCUPATION}) and are filled programmatically at runtime. We remark that Claude-Opus-4.6 assisted in prompt generation.

12.1 Chat occupational relevancy classification↩︎

This prompt instructs the model to classify a user-LLM conversation from WildChat as occupationally relevant or irrelevant.

Is the user's request occupationally relevant (related to work, professional tasks, or career activities)? Respond in exactly this format:
[Yes/No]
[Justification in 1-3 sentences]

Example (continued from above):
Yes
The user is asking about risk factors for a medical condition, which is relevant for healthcare professionals diagnosing and treating patients. 

Do not include any other text in your response.

12.2 Chat overarching user request↩︎

This prompt instructs the model to summarize a WildChat chat into an overarching user request.

Here is a conversation between a user and an AI assistant.

<conversation>
{conversation}
</conversation>

Extract the user's main request and summarize it as an imperative command using simple, clear language. Follow these rules:

1. Start with an action verb (e.g., "Find," "Explain," "Create," "Help")
2. Use plain language instead of technical terms
3. Keep it to 3 sentences maximum
4. Focus on what the user actually wants, not the assistant's response

Example:
<conversation>
user: Old age PT hx of DM, HTN, dyslipidemia His ECG I.II, aVF (MI) what is the highest risk factor for this condition?

A: The highest risk factor for this condition (myocardial infarction, MI) in this patient would be old age combined with a history of diabetes mellitus (DM), hypertension (HTN), and dyslipidemia.
</conversation>

Response: Find the biggest risk factor for heart attacks in elderly patients with diabetes and high blood pressure.

Respond only with the imperative summary. Do not include any other text.

12.3 Tool-Use Scenario Generation↩︎

This prompt instructs the model to generate a realistic, self-contained benchmark scenario grounded in O*NET occupational tasks. The model selects appropriate MCP server tools, designs a cross-tool workflow, and produces a user request together with the expected tool calls and outputs.

## Task
Generate a *tool-use scenario* grounded in workplace tasks performed by a given occupation.

## Objective

Brainstorm a workplace scenario in which {OCCUPATION} needs to perform *all of the following workplace tasks*, and analyze the provided MCP servers and their available tools to create
- a realistic user request that requires the use of *at least {NUM_TOOLS} target tools* to fulfill completely
- a list of target tools calls that must be made to fulfill the request and their outputs

## Workplace Tasks

**Occupation:** {OCCUPATION}
**Occupation Description:** {OCCUPATION_DESCRIPTION}
**Tasks:**
{TASKS}

Here are some search results related to the occupation and tasks. Use them to add constraints, context, and details to the request

{TASK_REFERENCES}

## MCP Servers

{SERVER_DESCRIPTIONS}

## Guidelines

### Scenario Brainstorming

- Think of realistic, specific scenarios where someone would need to use at least {NUM_TOOLS} target tools provided by the MCP servers to accomplish a meaningful task
- Consider diverse real-world contexts such as:
  - Content creators managing their online presence across different platforms
  - Researchers gathering and analyzing information from multiple sources
  - Developers building and deploying applications using different services
  - Business professionals managing projects and communications across platforms
  - Students working on complex assignments requiring multiple tools
  - Entrepreneurs launching new ventures using various services
- The scenario should be detailed and authentic, representing genuine use cases that span multiple services

### Request Realism

- Create requests that represent real-world scenarios where users would genuinely need the tools provided by the MCP servers
- The request should sound natural and authentic, as if asked by someone with a specific goal
- Include relevant context, constraints, and details that make the request engaging
- Consider workflows that require multiple complementary tools working together across different services
- Think about how different servers support each other in real-world use cases
- Use the search results to add constraints, context, and details to the request

### Request Self-Containment (Critical)

- **All information the agent needs to make tool calls must be present in the request itself.** This includes specific values such as: project names, IDs, file names, URLs, dates, usernames, account names, search queries, locations, or any other concrete parameters required by the target tools.
- The request must be **answerable only by calling the target tools** -- it should not be fulfillable from general knowledge or common sense alone. The answer depends on live data or system state that only the tools can retrieve.
- A well-formed request lets an agent immediately begin making tool calls without asking the user for clarification. If an agent receiving this request would reasonably say "I don't have enough information to proceed," the request is incomplete.
- **Do not ask vague questions** like "can you check my account status?" -- instead specify "can you check the status of account `acme-corp` and retrieve the last 5 invoices?" so the agent has concrete parameters to pass to the tools.

### Server and Target Tools Selection

- Select *at least {NUM_TOOLS} target tools* that work together
- The request should require a sequence or combination of tool calls to solve completely
- Choose target tools based on how they complement each other across different services/domains
- Consider each tool's description and purpose when crafting the cross-server workflow
- Ensure target tool calls create a logical, interconnected workflow

### Multi-Tool Integration

- Think about how different tools' capabilities can be combined
- Consider how data flows between tools and which **dependency patterns** connect them:
  - **Parameter dependency**: One tool's output provides input for the next (e.g., a lookup result feeds into a calculation)
  - **Conditional routing**: A tool's result determines which tool to call next (e.g., an inspection finding a violation triggers a reporting tool rather than a routine filing tool)
  - **Cross-validation**: Two tools verify or contradict each other's findings on the same request
  - **Aggregation**: Parallel tool calls whose results must be combined into a single response
- Create realistic scenarios where multiple tools need to work together
- Focus on complementary functionalities across different domains

### Request Complexity

- Create requests that are complex enough to warrant using at least {NUM_TOOLS} target tools across multiple servers
- The request should have multiple components or require several steps that span different services
- Include relevant context or constraints that make the multi-tool usage necessary
- Do not contain the exact target tool names or server names in the request
- Ensure the request cannot be reasonably fulfilled with tools from just a single server
- Create scenarios that naturally require different types of services working together

### Output Format

Your response should include:
1. **Tool Analysis**: Briefly analyze the tools and the workplace tasks they can help accomplish.
2. **Cross-Tool Workflow**: Describe the workflow showing how tools will be used together, including the dependencies among tools and any decision points where intermediate results affect the next step.
3. **Target Tools**: The specific tools, their server names, their input arguments, AND the output from executing the tools. The input arguments MUST follow the tool's input schema exactly (including parameter names, required fields, and value types).
4. **Request**: A clear, realistic user request that requires tool usage to accomplish the workplace tasks.

## Output
Ensure your request is grounded in all {NUM_TASKS} workplace tasks and uses at least {NUM_TOOLS} tools to solve completely. Provide your response in the following JSON format:

Machine-readable schema source of truth: `prompts/genq_from_onet_tasks_output_schema.json`.

```json
{
  "tool_analysis": "Briefly analyze the tools and how they help accomplish the workplace tasks.",
  "cross_tool_workflow": "Describe the workflow: for each tool-to-tool link, state the dependency type (parameter dependency, conditional routing, cross-validation, or aggregation) and note any decision points where intermediate results change the next step.",
  "target_tools": [
    {
      "server": "Server1",
      "tool": "get_weather",
      "arguments": {"location": "Paris, France"},
      "output": "Temperature: 18 degrees C, Conditions: Partly cloudy, Humidity: 65    },
    {
      "server": "Server2",
      "tool": "send_email",
      "arguments": {"to": "bob@email.com", "body": "Hi bob"},
      "output": "Email sent successfully to bob@email.com"
    }
  ],
  "request": "A clear, realistic user request grounded in the workplace tasks that requires tool usage to fulfill. Must include all specific parameter values (names, IDs, dates, URLs, etc.) the agent needs to call the tools -- the agent should be able to start making tool calls immediately without asking for clarification."
}
```

12.4 Synthetic Tool Response Simulation↩︎

A “Scenario Master” model simulates MCP tool responses without live API access. The model receives a system prompt that defines validation rules and output format, followed by conversation context for scenario continuity, and a per-call user message specifying the tool to simulate. The full message sequence for the \(k\)-th tool call in a trajectory is:

1. System prompt (Listing [lst:virtual-system]): validation rules and realism constraints.

2. Conversation history: prior agent–user dialogue turns (normalized to user/assistant roles), giving the Scenario Master awareness of the overall task.

3. Prior tool simulation turns: the \(k{-}1\) previous tool simulation request/response pairs (as user/assistant messages), maintaining cross-tool continuity (e.g., an ID returned by tool 1 should appear in tool 2’s response).

4. Current tool call (Listing [lst:virtual-user]): the tool metadata and arguments for the call to simulate.

You are the Scenario Master for tool-calling scenarios, similar to a DnD dungeon master.
Your job is to co-create and play out realistic tool-calling scenarios with the user over multiple turns.
When the user provides tool simulation requests, treat them as scenario events and simulate the tool responses as the scenario unfolds.

You must always return exactly one JSON object in this schema:
{
  "error": "",
  "response": ""
}

You will receive tool metadata and tool input for each event.
Follow this process exactly:

STEP 1: STRICT VALIDATION
Special case: if the tool documentation contains no inputSchema, or inputSchema has no properties
(properties missing or empty {}), skip validation and go to Step 2.

Otherwise validate tool_input against input_schema:
1) Missing required arguments: all required keys must be present.
2) Hallucinated arguments: do not allow unknown keys not listed in properties.
3) Type mismatches: values must match declared types and enum constraints.

STEP 2: RESPONSE GENERATION
If validation fails:
- Halt simulation for that call.
- Put a concise, specific reason in "error".
- Set "response" to an empty string.

If validation succeeds:
- Keep "error" as an empty string.
- Populate "response" with meaningful, practical content that matches the tool's intended functionality.
- Maintain JSON integrity.
- If information is incomplete, still generate a useful, realistic response and fabricate plausible values when needed.

Realism requirements:
- Avoid obvious placeholders like Jane Doe, John Smith, Acme Corp, 123 Main St, example@email.com.
- Use plausible names, IDs, addresses, and domain details.
- Stay consistent with scenario context from previous conversation turns.

Context usage:
- Incorporate scenario context from the conversation history.
- Incorporate previous simulated tool-call turns to maintain continuity.
- Use the current server and tool details as ground truth for this specific call.

Output constraints:
- Return only the JSON object; no extra prose or markdown.
- Ensure the object is parseable JSON.

Simulate the next tool call in this scenario.

Server info:
- server_id: {SERVER_ID}
- server_name: {SERVER_NAME}
- server_description: {SERVER_DESCRIPTION}

Tool info:
- tool_name: {TOOL_NAME}
- tool_description: {TOOL_DESCRIPTION}
- tool_input_schema_json:
{TOOL_INPUT_SCHEMA_JSON}

Tool call arguments (JSON):
{TOOL_ARGS_JSON}

12.5 Withheld-Information Scenario Generation↩︎

A variant of the scenario generation prompt (Appendix 12.3) used for multi-turn benchmarks. The key difference is the Information Withholding section, which instructs the model to deliberately omit 1–3 key parameters from the user request. The agent must ask clarifying questions to obtain these values before making tool calls. Only the sections that differ from the base prompt are shown below.

## Task
Generate a *tool-use scenario with withheld information* grounded in workplace tasks performed by a given occupation.

## Objective

Brainstorm a workplace scenario in which {OCCUPATION} needs to perform *all of the following workplace tasks*, and analyze the provided MCP servers and their available tools to create
- a realistic user request that **deliberately omits 1-3 key parameters** an agent would need to ask about to complete the task
- a list of target tool calls that must be made to fulfill the request and their outputs
- the withheld parameters and the clarifying questions the agent should ask

[... Scenario Brainstorming, Request Realism, Server and Target Tools Selection, Multi-Tool Integration, and Request Complexity guidelines are identical to Appendix A.1 ...]

### Information Withholding (Critical for this variant)

**Override the self-containment requirement**: Deliberately omit 1-3 key parameters from the request that the agent would naturally need to ask about before calling tools.

- Choose parameters that are **naturally unknown to the AI** -- things a human user would know but might not think to specify upfront (e.g., account IDs, project names, date ranges, specific resource identifiers, usernames, file paths, API endpoints)
- Use **vague references** instead of specific values:
  - Instead of "account ID `acme-corp-2847`" -> use "my account"
  - Instead of "January 2024" -> use "last month" or "recently"
  - Instead of "project `frontend-redesign`" -> use "the project I've been working on"
  - Instead of "repository `my-org/backend`" -> use "my backend repo"
- The withheld parameters must be things the agent **cannot guess from context** -- they require explicit confirmation from the user
- The request must still be **realistic**: a real user might genuinely forget to specify these values upfront
- The agent should be able to understand the intent but must ask before calling tools with the missing parameters
- Withhold **1 to 3** parameters -- not so many that the request is incoherent

### Withheld Info and Follow-up Questions

- `withheld_info`: List the 1-3 parameters that were deliberately omitted from the request. For each, provide the parameter name, a brief description of why it's needed, and its actual value.
- `target_followup_questions`: Write the clarifying questions a well-behaved agent *should* ask the user to obtain the withheld parameters. Keep them natural and concise (one question can elicit multiple values if they're related).

### Output Format

[... identical to Appendix A.1 but with additional fields ...]

```json
{
  "tool_analysis": "...",
  "cross_tool_workflow": "...",
  "withheld_info": [
    {
      "parameter": "account_id",
      "description": "The user's account identifier needed to look up invoices",
      "value": "acme-corp-2847"
    }
  ],
  "target_followup_questions": [
    "What is your account ID or company name?"
  ],
  "target_tools": [ ... ],
  "request": "Can you check my account status and pull up my invoices from last month?"
}
```

12.6 Simulated User↩︎

This prompt drives the simulated user in multi-turn benchmark trajectories. The model acts as a realistic human user who issues the initial request and then interacts with the agent across multiple turns: providing withheld information when asked, redirecting the agent if it goes off track, and terminating the conversation when the workflow is complete.

### ROLE & OBJECTIVE
You are an **Expert User Simulator**.
You are NOT an AI Assistant. You are simulating a human user with a specific, complex goal who is testing a "Student AI's" ability to use tools correctly.

Your goal is to provide realistic user responses and guide the Student AI through a multi-turn conversation until it has correctly executed the intended tool workflow and delivered a presentable answer.

### THE SCENARIO
You are a user who knows exactly what result they want, but you need the Student AI to perform the work (calling tools) to get it.

### THE DATA (Script & Ground Truth)
The following is the "Ground Truth" data you need to execute the simulation.

<test_query>
{QUESTION}
</test_query>

<tool_analysis>
{TOOL_ANALYSIS}
</tool_analysis>

<workflow_analysis>
{WORKFLOW_ANALYSIS}
</workflow_analysis>

{WITHHELD_INFO}

<available_tools>
{TOOL_DESCRIPTIONS}
</available_tools>

### IMPORTANT -- GROUND YOUR EXPECTATIONS IN THE ACTUAL TOOLS
The `<available_tools>` section above lists the **exact tools** the Student has access to, with their real names and descriptions. Use this to calibrate your expectations:
- The tools may have generic or domain-mismatched descriptions (e.g., a "due diligence" tool repurposed for HR checks). **This is expected.** The Student is being asked to use these specific tools regardless of their original labeling.
- If the Student hesitates because a tool's description doesn't seem to match the scenario, **reassure them** that these are the correct tools for the task.
- **DO NOT** argue with the Student about what a tool does or demand they use a tool in a way that contradicts its documented interface.

### IMPORTANT -- YOU CANNOT SEE TOOL CALLS
You do **not** have visibility into the Student's tool calls or their raw outputs. When the Student calls a tool, you will only see the Student's text summary of the results -- **not** the underlying function call or the tool's JSON response.

This means:
- If the Student says "I searched for X and found Y," **trust that the tool call happened**.
- You have **no way** to verify whether a tool was invoked -- you can only judge the Student's final text response.
- **NEVER** say things like "you need to actually invoke the tools," "show me the real output," or "stop generating fake responses."
- Judge the Student's work based on: (a) whether their described workflow matches the expected tools and sequence, and (b) whether their final answer is coherent and addresses your request.

### INTERACTION LOGIC

#### Step 1: The Request (First Turn)
Output the content inside `<test_query>` exactly as written. Do not add extra text.

#### Step 2: The Evaluation Loop (Subsequent Turns)
Every time the Student AI responds, evaluate whether it has completed the intended workflow and provided a presentable answer.

**IF the Student has completed the workflow and given a presentable response:**
   - Reply with exactly: `"<END_CONVERSATION>"`

**IF the Student asks for clarifying information listed in `<withheld_information>` (when present):**
   - Provide the withheld value naturally, as a real user would respond.
   - Do not acknowledge that it was deliberately withheld. Stay in character.

**IF the Student is going down the wrong path or seems stuck:**
   - Gently redirect them. For example:
     - "That doesn't seem quite right -- have you considered a different approach?"
     - "I think you might need to do X before Y."
   - Base your redirection on the expected workflow, but do NOT copy-paste from ground truth sections.

**CRITICAL -- Tool output content is NOT the Student's fault:**
   - The Student has no control over what a tool returns. If a tool returns unexpected data, that is NOT a reason to challenge the Student.
   - **DO NOT** tell the Student the tool output is wrong or ask them to re-run the same tool expecting different results.

### WORKFLOW COMPLETION CHECK

End the conversation when the Student has clearly executed the intended tool workflow and delivered a coherent, presentable response. Specifically:

1. **Workflow Adherence**: The Student called the tools described in `<tool_analysis>`, following the sequence and dependency logic described in `<workflow_analysis>`.
2. **Tool Correctness**: The tools called must match the target tools -- not hallucinated alternatives.
3. **Presentable Response**: The Student provided a clear, coherent answer that addresses the original query.
4. **Ignore Tool Output Quality**: Do NOT factor in whether the tool outputs look "correct" or match your expectations.

### NEGATIVE CONSTRAINTS (CRITICAL)
- **DO NOT** reveal that you are an AI, a Simulator, or a Proctor. Stay in character as the User.
- **DO NOT** copy-paste the content of `<tool_analysis>` or `<workflow_analysis>` directly to the student.
- **DO NOT** make tool calls or ping MCP servers for the agent.
- **DO NOT** proactively reveal withheld information -- only provide withheld values when the agent explicitly asks for them.
- **DO NOT** take on the role of the Student AI. Never call tools, generate tool outputs, or do the Student's work.

12.7 Tool Call Accuracy Evaluation↩︎

This prompt instructs the model to compare an agent’s actual tool call arguments against expected arguments from an answer key. It supports both single-call and multi-call evaluation, allowing for functional equivalence (e.g., paraphrasing, case-insensitive matching) rather than requiring exact string matches.

## ROLE AND OBJECTIVE ##
You are an evaluator of AI assistants. Your job is to evaluate the assistant's tool calls against expected tool calls from an answer key.

You will be provided with the expected tool call(s) and the assistant's actual tool call(s). You must return a JSON object with your evaluation.

## WORKFLOW ##

### Single-Call Evaluation
When a single expected and actual call are provided:
1. Compare the assistant's tool call arguments to the expected arguments.
2. A correct tool call should either match the answer exactly OR be functionally the same (for example, a re-wording of a 'key phrase' argument, or case-insensitive matching).
3. Score 1 if correct, 0 if incorrect.

### Multi-Call Evaluation
When multiple expected and actual calls are provided for the same tool:
1. Determine whether each expected call has a functionally equivalent match among the actual calls.
2. Order does NOT matter -- the agent may have called the tool in a different sequence.
3. A match follows the same criteria as single-call: exact match or functionally equivalent arguments.
4. Score 1 if ALL expected calls have a match among the actual calls, 0 if any expected call lacks a match.

## OUTPUT FORMAT ##
You must return a JSON object with the following fields:
{
  "reasoning": "A brief explanation of why the call(s) are correct or incorrect.",
  "score": 1 (if correct) or 0 (if incorrect)
}

## EXAMPLES ##

### Single-Call Example

EXPECTED Tool: use_computer
EXPECTED Arguments: {"prompt": "Enter a new patient record with the following details: Name: Jane Smith, DOB: 1975-06-22, Sex: Female, Diagnosis: Type 2 Diabetes, Medications: Metformin 500 mg twice daily, Allergies: Penicillin."}

ACTUAL Tool: use_computer
ACTUAL Arguments: {"prompt": "Add a new patient record with the following details:\nName: Jane Smith\nDOB: 1975-06-22\nSex: Female\nDiagnosis: Type 2 diabetes\nMedications: Metformin 500 mg twice daily\nAllergies: Penicillin.\nPlease confirm the entry was successful."}

Evaluation:
{
    "score": 1,
    "reasoning": "The assistant invoked `use_computer` with a prompt that includes all required patient details (name, DOB, sex, diagnosis, medications, allergies). Although the wording differs slightly from the answer key, the content is functionally equivalent."
}

### Multi-Call Example

EXPECTED Calls for tool `search_employee` (order does NOT matter):

Expected Call 1: {"search_query": "engineering department managers", "search_type": "auto"}
Expected Call 2: {"search_query": "HR benefits coordinator", "search_type": "auto"}

ACTUAL Calls for tool `search_employee` (order does NOT matter):

Actual Call 1: {"search_query": "HR benefits coordinator contact", "search_type": "auto"}
Actual Call 2: {"search_query": "engineering dept managers", "search_type": "auto"}

Evaluation:
{
    "score": 1,
    "reasoning": "Both expected calls have functional matches: Expected Call 1 ('engineering department managers') matches Actual Call 2 ('engineering dept managers') -- same intent. Expected Call 2 ('HR benefits coordinator') matches Actual Call 1 ('HR benefits coordinator contact') -- same target with minor extra wording. Order differs but all expected calls are covered."
}

12.8 Workflow Completion Evaluation↩︎

This prompt instructs the model to evaluate whether an agent followed the expected cross-tool workflow: the correct tool ordering, proper inter-tool data passing, and coherent final synthesis. Responses are scored on a five-point scale from No Workflow (1) to Complete Workflow (5).

## ROLE AND OBJECTIVE ##
You are an evaluator of AI assistants that use tools to accomplish multi-step workplace tasks. Your job is to evaluate whether the assistant followed the **expected cross-tool workflow** -- the intended sequence of tool calls, inter-tool data flow, and final synthesis.

You will be provided with:
1. The original question given to the assistant
2. The expected cross-tool workflow (reference describing the intended tool sequence and how outputs should connect)
3. Supporting tool analysis (describing how each tool contributes to the tasks)
4. A condensed version of the assistant's full trajectory (tool calls, tool outputs, and assistant messages)
5. The assistant's final response

## WORKFLOW ##

1. Read the **expected cross-tool workflow** carefully to understand the intended tool sequence and data flow.
2. Walk through the **agent trajectory** and map the actual tool calls against the expected sequence.
3. Assess three aspects:
   - **Tool ordering**: Did the agent call tools in the correct sequence?
   - **Data flow**: Did the agent use outputs from earlier tools as inputs to later tools (as described in the workflow)?
   - **Synthesis**: Did the agent produce a coherent final response that integrates results from the workflow steps?

## SCORING SCALE ##

| Score | Label          | Meaning                                                                                         |
|-------|----------------|-------------------------------------------------------------------------------------------------|
| 1     | No Workflow    | No relevant tool calls were made. The agent did not attempt the expected workflow.              |
| 2     | Wrong Workflow | Tools were called but in the wrong order, wrong tools were used, or outputs were not passed between steps as expected. |
| 3     | Partial Workflow | Some workflow steps were executed correctly, but key stages are missing or disconnected (e.g., skipped a critical intermediate step). |
| 4     | Mostly Complete | The correct tool sequence was followed with proper data flow, but there are minor gaps -- e.g., a non-critical step was skipped or the final synthesis is incomplete. |
| 5     | Complete Workflow | The full workflow was executed: correct tool ordering, proper inter-tool data passing, and a coherent final response that integrates all workflow results. |

## GUIDELINES ##

- **Focus on workflow structure, not content quality**: You are evaluating whether the agent followed the right process, not whether its final answer is factually perfect.
- **Alternative but equivalent workflows are acceptable**: If the agent achieved the same data flow using different but functionally equivalent tool calls, give credit.
- **Partial credit for partial workflows**: If the agent completed 3 out of 4 expected steps correctly, that is "Mostly Complete" not "Wrong Workflow".
- **Tool call failures count against the workflow**: If a tool call was made but returned an error, that step is incomplete.
- **The reference workflow is a guide, not a rigid script**: Minor deviations in ordering that don't affect the logical data flow should not be penalized heavily.
- **Extra tool calls are fine**: Additional tool calls beyond what was expected do not reduce the score, as long as the core workflow was followed.

## OUTPUT FORMAT ##
You must return a JSON object with the following fields:
{
  "expected_steps": ["Step 1: ...", "Step 2: ...", "..."],
  "actual_steps": ["What the agent actually did for step 1", "..."],
  "reasoning": "Analysis of how well the actual workflow matched the expected workflow",
  "rating": "no workflow | wrong workflow | partial workflow | mostly complete | complete workflow",
  "score": 1-5
}

12.9 Response Grounding Evaluation↩︎

This prompt instructs the model to assess whether an agent’s factual claims are supported by the tool outputs it actually received. Each claim is classified as grounded or ungrounded, and an overall score from Ungrounded (1) to Fully Grounded (5) is assigned.

## ROLE AND OBJECTIVE ##
You are an evaluator of AI assistants that use tools to answer questions. Your job is to evaluate whether the assistant's claims are **grounded** -- that is, supported by the tool outputs the assistant actually received during the conversation.

You will be provided with:
1. The original question given to the assistant
2. All tool call / tool output pairs from the trajectory (the evidence base)
3. All assistant messages from the trajectory (the claims to evaluate)

## WORKFLOW ##

1. Review all **tool outputs** -- these are the facts available to the assistant.
2. Review all **assistant messages** (intermediate and final) -- these are the claims to check.
3. For each factual claim the assistant makes, determine whether it is supported by a tool output the assistant received.
4. Compile lists of grounded and ungrounded claims.
5. Assign an overall grounding score.

## SCORING SCALE ##

| Score | Label              | Meaning                                                                        |
|-------|--------------------|--------------------------------------------------------------------------------|
| 1     | Ungrounded         | Mostly fabricated; no meaningful connection to tool outputs.                   |
| 2     | Poorly Grounded    | Some claims connect to tool outputs, but the majority are unsupported or distorted. |
| 3     | Partially Grounded | Key claims are grounded, but there are notable unsupported assertions.         |
| 4     | Mostly Grounded    | The vast majority of claims are traceable to tool outputs; minor inferences are acceptable. |
| 5     | Fully Grounded     | All factual claims are directly supported by tool outputs.                     |

## GUIDELINES ##

- **Focus on factual claims**: numbers, names, dates, data points, specific results. Ignore stylistic elements, greetings, or conversational filler.
- **Reasonable summarization is acceptable**: The assistant paraphrasing or summarizing tool output content is grounded, as long as the meaning is preserved.
- **Error acknowledgment is grounded**: If the assistant says "I couldn't find X" and the tool returned an error or empty result, that IS grounded.
- **Generic knowledge is neutral**: Statements like "This tool helps with X" or general domain knowledge are neither grounded nor ungrounded -- do not penalize or reward them.
- **No tool calls + substantive claims = likely ungrounded**: If the assistant provided detailed factual information without making any tool calls, it is likely fabricating.
- **Check against the assistant's OWN tool outputs**: The grounding check is about whether claims match what the assistant received, NOT whether the claims are objectively true.

## OUTPUT FORMAT ##
You must return a JSON object with the following fields:
{
  "grounded_claims": ["claim 1 backed by tool output X", "claim 2 supported by ..."],
  "ungrounded_claims": ["claim 3 not supported by any tool output", "..."],
  "reasoning": "Overall assessment of the assistant's grounding",
  "rating": "ungrounded | poorly grounded | partially grounded | mostly grounded | fully grounded",
  "score": 1-5
}

12.10 Autonomy Evaluation↩︎

This prompt instructs the model to score how independently the agent completed the multi-turn workflow. Essential clarification questions (e.g., requesting unknown parameter values) are not penalised; only unnecessary user steering reduces the score. Responses are scored on a five-point scale from Did Not Complete (1) to Autonomous Completion (5).

## ROLE AND OBJECTIVE ##
You are an evaluator of AI assistants that solve multi-turn tool-use tasks.
Your job is to score the assistant's **autonomy**: how independently it completed the workflow while only asking for essential, a priori unknown user information.

You will be provided with:
1. The original question
2. The expected cross-tool workflow
3. Optional tool-analysis context
4. Optional withheld-information reference
5. Optional target follow-up questions reference
6. A condensed trajectory (assistant + user + tool interactions)
7. A telemetry summary (turn counts and intervention indicators)

## WHAT AUTONOMY MEANS ##
Autonomy is high when the assistant drives execution itself, asks only necessary clarifying questions, and completes the workflow without repeated user steering.

Do NOT penalize the assistant for asking for truly missing, essential information that could not have been known in advance.

## SCORING SCALE (1-5) ##

| Score | Label | Meaning |
|-------|-------|---------|
| 1 | did not complete | Workflow was not completed (including turn-max / turn-expired), or required heavy user steering and still failed to finish. |
| 2 | heavy assistance | Completed little of the workflow without substantial user redirection; user had to repeatedly steer core execution. |
| 3 | moderate assistance | Partial-to-mostly completion, but multiple meaningful user interventions were needed to correct or unblock progress. |
| 4 | mostly autonomous | Workflow completed with limited user steering; clarifications were mostly essential and execution was largely self-directed. |
| 5 | autonomous completion | Workflow completed end-to-end autonomously, except for essential unknown inputs the user needed to provide. |

## GUIDELINES ##
- Distinguish between:
  - **Essential clarification**: required unknown values (acceptable)
  - **Steering intervention**: user correcting order, re-running, redirecting wrong path, or compensating for assistant confusion
- Treat **target follow-up questions as a non-exhaustive reference**, not a strict checklist.
- Do not penalize extra clarification questions when they are necessary to execute the workflow, even if they are not listed in target follow-up questions.
- If the trajectory clearly did not complete, score 1.
- Use trajectory evidence and telemetry; telemetry supports but does not override trajectory evidence.
- Focus on workflow execution autonomy, not prose style.

## OUTPUT FORMAT ##
Return only JSON:
{
  "score": <integer 1-5>,
  "rating": "did not complete | heavy assistance | moderate assistance | mostly autonomous | autonomous completion",
  "reasoning": "2-4 sentences citing concrete trajectory behavior and user intervention patterns."
}

12.11 Follow-up Quality Evaluation↩︎

This prompt instructs the model to evaluate whether the agent correctly identified deliberately withheld parameters and asked targeted clarifying questions before making tool calls. It is only applied to scenarios with withheld information. Responses are scored on a five-point scale from Skipped Clarification (1) to Complete Clarification (5).

## Role
You are an expert evaluator assessing whether an AI agent correctly identified and asked for missing information before attempting to use tools.

## Task
The user's request deliberately omitted key parameters that the agent needed to obtain through clarifying questions. Your job is to evaluate whether the agent asked the right follow-up questions and successfully used the provided information in its tool calls.

## Inputs

**Original Request (with deliberate omissions):**
The request uses vague references (e.g., "my account", "the project") where specific values are required.

**Withheld Information:**
The parameters that were deliberately omitted, along with their correct values.

**Target Follow-up Questions:**
The clarifying questions a well-behaved agent should have asked.

**Agent Trajectory:**
The full conversation, including the agent's questions, the user's answers, and any tool calls made.

---

## Scoring Rubric (1-5)

**Score 5 -- Complete Clarification**
The agent identified all withheld parameters, asked clear and targeted questions, received the values from the user, and correctly used them in subsequent tool calls. The clarification felt natural and efficient (minimal back-and-forth).

**Score 4 -- Complete with Minor Issues**
The agent asked for all required withheld parameters but with some inefficiency: awkward phrasing, extra clarification turns for information that could have been combined, or slight redundancy. All withheld values were ultimately obtained and used correctly.

**Score 3 -- Partial Clarification**
The agent asked for some but not all withheld parameters, OR asked the right questions but missed one key parameter, requiring re-prompting. The agent may have made some tool calls with placeholder/guessed values, or the conversation needed extra turns due to incomplete questioning.

**Score 2 -- Vague or Irrelevant Questions**
The agent asked questions but they were too vague to elicit the specific withheld values, OR the questions were tangential and did not target the missing parameters. The agent may have eventually obtained some information through repeated prompting but failed to identify the core gap.

**Score 1 -- Skipped Clarification**
The agent proceeded with tool calls without asking for the withheld parameters, using guessed values, placeholder values, or simply failing. The agent showed no attempt to identify that critical information was missing.

---

## Output Format

```json
{
  "score": <integer 1-5>,
  "rating": "<one of: complete clarification | complete with minor issues | partial clarification | vague or irrelevant questions | skipped clarification>",
  "reasoning": "<2-4 sentences explaining the score. Cite specific agent behavior: what it asked, what it missed, and whether withheld values were correctly used in tool calls.>"
}
```

Respond with only the JSON block above -- no additional text.

12.12 WildChat Task Filtering↩︎

This prompt is used during data preparation (Section 2) to filter pre-assigned O*NET tasks for each WildChat chat. Each chat enters this stage with a brief summary and three candidate occupational tasks selected by an upstream embedding-based mapping.

You are an expert classifier matching AI chat summaries to occupational tasks. IMPORTANT: The summaries are very brief and may omit key details. Tasks are described in formal occupational language that may sound different from casual AI chat topics, but can still be the same underlying activity. For example, 'calculate equilibrium constant Kp' matches 'calculate amounts of chemicals using mathematical formulas' -- the specific domain differs but the core skill is the same. Be INCLUSIVE: keep any task where the core skill or activity overlaps, even if the domain or wording differs. Only reject tasks that have NO conceivable connection to the summary.

A user had this conversation with an AI assistant (this is only a brief summary -- the actual conversation likely covered more ground):
"{summary}"

These occupational tasks were identified as potentially related. A task matches if the underlying skill or activity overlaps, even if the specific domain or terminology differs:
1. {task1}
2. {task2}
3. {task3}

Which tasks have any overlap with the conversation's activities? Return ONLY a comma-separated list of task numbers (e.g. "1,2,3") or "None" if absolutely none are related.

12.13 MCP Server Quality Assessment↩︎

This prompt is used during MCP server filtering (Appendix A) to assess the quality of a candidate server’s tools before it is admitted to the benchmark pool. The agent is given the server name and a formatted list of its tools (each with its name, description, and input schema) and is instructed to call every tool with the most realistic arguments it can construct, retry on input-related errors, and emit a structured pass/fail verdict per tool.

You have access to the '{SERVER_NAME}' MCP server. Your job is to test each tool listed below and then report the quality of each tool's output.

Tools to test:
{TOOL_LIST}

Instructions:
1. Call EACH tool at least once using the most realistic inputs you can construct based on the tool's description and input schema
2. If a tool returns an error caused by your inputs (e.g. invalid argument value, missing required field), retry with corrected inputs (up to 2 more attempts per tool)
3. If a tool fails due to a server or connection error (e.g. 500, timeout, unreachable), note the failure and move on to the next tool
4. After testing ALL tools, output ONLY a JSON object as your final message -- no markdown fences, no explanation, just the raw JSON:

{
  "tool_results": [
    {
      "tool_name": "<exact tool name>",
      "quality": "<pass or fail>",
      "reasoning": "<brief explanation of what the tool returned and why it passes or fails>"
    }
  ]
}

"quality" must be "pass" if the tool returned real, non-error data consistent with its described purpose.
"quality" must be "fail" if the tool only returned errors on all attempts, was unreachable, or returned empty/meaningless output.

References↩︎