1 Introduction↩︎

The use of large language models (LLMs) is rapidly advancing from conversational assistants toward autonomous agentic systems. These systems equip LLMs with external tools that allow them to interact directly with the real world: sending emails, modifying files, controlling smart home devices, and executing code [1]–[5]. Unlike chatbots that simply produce text for human review, agents can perform consequential and irreversible actions on the user’s behalf. As agents gain greater autonomy, their potential impact and associated risks increase dramatically  [6]–[9].

A major risk in agentic systems is that agents may execute actions that deviate from what the user actually intended, a phenomenon we refer to as misalignment. For example, a user who asks an agent to “request $30 from Daniel” may find that the agent instead calls a “send money” tool, paying the amount to Daniel, an action contrary to the user’s request. The term “misalignment” has sometimes been used to refer to malicious agent behavior [10]–[12], an equally important but distinct concern. In this work, we focus on cases where agents act without malicious intent but on a misunderstanding of or unjustified assumptions about the user’s intent that is expressed through a prompt. Such misalignment can cause significant unintended harm, particularly when the resulting actions are irreversible. Detecting and preventing misalignment before tool execution is therefore crucial for deploying agents in practice.

Existing research has explored multiple approaches to mitigating the risks of agent misalignment. Training-time alignment methods, such as preference optimization and supervised fine-tuning [13], [14], can improve the general behavior of an agent, but cannot cover the full range of tool configurations, environment states, and task contexts encountered at deployment. Prompt-level controls, including carefully designed system instructions and reasoning scaffolds, can further guide behavior, but they still rely on the same underlying model whose judgments they seek to improve [15]–[17]. Sandboxing and capability restriction can reduce harm by isolating tool execution and limiting an agent’s privileges, but they primarily contain the consequences of an action rather than determine whether the action faithfully reflects the user’s intent [18], [19]. Post-hoc verification and recovery methods provide useful auditing or corrections after actions are taken, but their abilities to prevent harm are limited once an irreversible tool invocation has already occurred [20], [21]. Together, these limitations motivate the need for pre-execution runtime guardrails in LLM agents [22], [23]. Positioned between action generation and tool execution, these guardrails serve as an independent decision layer that can detect and prevent misaligned actions before they produce irreversible effects on the environment [24].

However, existing runtime guardrails often rely on an LLM-as-a-judge mechanism to assess whether a proposed action should be allowed [25]–[27]. Without a systematic framework for reasoning about alignment, such judgments remain largely subjective and unguided [28]. They lack explicit decision criteria, which can lead to inconsistent standards between runs and models [29]–[31]. Consequently, the underlying rationale for a judgment is more difficult to interpret and audit.

In this paper, we propose a systematic framework to detect misaligned agent actions motivated by provenance analysis, a well-established discipline that traces the origin and derivation history of an artifact [32], [33]. The key intuition behind our approach is that whether or not an action is aligned with the user’s intent should be justifiable by traceable evidence in the context, i.e., the user’s request, the available tool documentation, and the history of prior interactions. We develop this idea at the level of tool invocation by distinguishing three types of misalignment (Fig. 1): (1) tool-level misalignment: selecting a wrong tool, (2) parameter-level misalignment: selecting tool parameters that are not traceable to the context or not appropriate to address user’s query, and (3) interpretation-level misalignment: selecting one of multiple possible interpretations of the user’s query that is underspecified in the given context. We provide a conceptual framework that formalizes the notion of provenance in agentic settings and the definition of the above types of misalignment. Our framework serves as a theoretical basis for implementing guardrails to check whether or not a tool action is considered aligned or misaligned based on explicit, auditable provenance links between the action and the given context.

Based on the proposed framework, we developed ProvenanceGuard, a multi-stage pipeline implementation of the framework. The pipeline performs a three-stage analysis, with separate stages for tool-level, parameter-level, and interpretation-level misalignment. We evaluated ProvenanceGuard on two benchmarks, Agent-SafetyBench [34] and WorkBench [35], against an LLM-as-a-judge baseline and a single-step variant of ProvenanceGuard, across 10 backbone LLMs.

Our experimental results show that the proposed provenance-based conceptual framework itself substantially improves misalignment detection over direct LLM-as-a-judge prompting, and that the multi-stage design yields significant additional gains, especially on Agent-SafetyBench. Specifically, ProvenanceGuard reduces average error rate on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench. At the same time, these gains do not come with a comparable increase in intervention cost. Compared to the baseline, ProvenanceGuard does not introduce a statistically significant rise in unnecessary interventions on aligned traces, and it lowers intervention rate on task-successful WorkBench traces from 30.5% to 12.8%. These results suggest that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents against misaligned actions.

The main contributions of this paper are as follows:

• A taxonomy of agent action misalignment, distinguishing tool-level, parameter-level, and interpretation-level misalignments (Sec. 3).

• A provenance-based conceptual framework for detecting misaligned agent actions, which frames alignment checking as determining whether proposed tool calls are grounded by traceable evidence in the agent’s context (Sec. 4).

• ProvenanceGuard, a multi-stage pipeline implementation of the framework that operates as a runtime guardrail (Sec. 4.3), together with an evaluation on two benchmarks across 10 backbone LLMs (Sec. 5&6).

• A manually annotated benchmark of agent execution traces derived from Agent-SafetyBench (Sec. 5.1).

2 Preliminaries↩︎

2.1 LLM Agent Workflow↩︎

We consider a typical LLM agent setting in which an LLM interacts with a set of pre-defined tools to complete user tasks (Fig.2). At the beginning of an interaction, a user submits a natural language query to the agent. The LLM serves as the decision-making core of the agent. Based on the query and the current context, the LLM determines which tool to invoke and how to parameterize the tool call in order to access information or modify the environment [1].

After a tool is invoked, the resulting output is returned to the LLM. The model then decides the next step: It may invoke additional tools, ask the user for clarification, or return a final response once the task is completed. Through this iterative loop of reasoning, tool invocation, and observation, the agent interacts with the environment to accomplish the user’s request.

Incorporating an external guardrail module in the workflow is an effective way to reduce the risk of unintended actions [24]. This module acts as a monitoring layer between the actions proposed by the LLM and the actual execution of the corresponding tools. When the LLM proposes a tool call, the guardrail evaluates whether the action is consistent with the user’s query. If the guardrail determines that the proposed action is potentially misaligned, it can block the execution and instead request user confirmation or clarification.

2.2 Underspecification↩︎

When a user interacts with an LLM agent, the natural language query ultimately corresponds to a sequence of concrete actions that an LLM agent needs to execute. However, due to the inherent limitations of natural language and the absence of certain contextual information, a single query can often be interpreted in multiple plausible ways. Each interpretation may correspond to a different sequence of actions that the agent could perform. But not every interpretation is aligned with THE user-wanted sequence of concrete actions. We refer to such situations as underspecification.

A query is underspecified when the information contained in the query, together with the available context, is insufficient to uniquely determine the intended action sequence. In this case, multiple interpretations remain consistent with the user’s request, and the agent cannot reliably infer which interpretation reflects the user’s true intent.

Underspecification is closely related to, but distinct from, ambiguity [36]. Ambiguity is an intrinsic property of natural language: Linguistic expressions can admit multiple meanings due to lexical, syntactic, or semantic factors. Underspecification, in contrast, arises from the interaction between the query and its context. A query is underspecified not merely because it has multiple linguistic interpretations, but mainly because the available information is insufficient to resolve which interpretation should guide the agent’s actions. A query that is underspecified in one context may be well-specified in another with additional contextual constraints.

When a query is underspecified, selecting one plausible interpretation and acting upon it requires the agent to make assumptions about the user’s intent. If these assumptions are incorrect, the resulting actions may deviate from the user’s actual intent, leading to misaligned behavior. Therefore, identifying underspecified queries is a critical step toward ensuring that LLM agents avoid arbitrarily committing to a particular interpretation and instead seek clarification or defer action when necessary.

3 Types of Misalignment↩︎

LLM agents interact with external environments by invoking tools. These tool calls could produce irreversible or high-impact effects, such as transferring money, sending emails, or modifying files in the computer. Unlike incorrect textual responses from chatbots, which are typically visible to the user and can be corrected, misaligned tool executions may cause unintended consequences without the user immediately realizing it.

In this work, we define misalignment at the level of tool invocation. We categorize misaligned actions into three types.

3.0.0.1 (1) Tool-Level Misalignment

The first type occurs when the agent selects a tool that is irrelevant or incompatible with the task in the user query.

3.0.0.2 Example

As shown in Fig.1a, while the user asks the agent to “request $30,” the agent proposes to call the “money_transfer” tool, which transfers money to the person who actually owes the user. The action is clearly inconsistent with the user’s request. The tool invocation itself is unrelated to the intended operation, leading to a tool-level misalignment.

3.0.0.3 (2) Parameter-Level Misalignment

The second type occurs when the agent selects the correct tool but assigns incorrect or ungrounded parameters. The resulting action deviates from the stated task even though the tool itself is appropriate.

3.0.0.4 Example

As shown in Fig.1b, the agent proposes to call the right tool, “money_transfer,” but the parameters it picks do not match the user’s request. This type of error reflects a failure to properly ground the action parameters in the user query or the context.

3.0.0.5 (3) Interpretation-Level Misalignment

The third category directly relates to underspecification. An underspecified query admits multiple plausible interpretations. The misalignment arises when the agent arbitrarily resolves an underspecified query by committing to a particular interpretation and executes actions without clarification. In such cases, the actions may be consistent with one possible interpretation, but the interpretation itself is unjustified and the actions could lead to severe unwanted consequences.

3.0.0.6 Example

As shown in Fig.1c, the user only says to “handle” the request while not providing any concrete steps about how to handle it. Possible interpretations include paying the request, rejecting it, notifying the user, or asking for confirmation. However, the agent interprets this instruction as automatically paying overdue tickets, which involves a strong assumption that was never stated by the user. This is a clearly misaligned action. The appropriate behavior in this situation would be to present the pending requests and ask the user how they should be handled.

4 Technical Approach↩︎

In this section, we introduce our provenance-based approach to detection of misalignment, including a conceptual framework for provenance-based guardrails and its realization in a tool called ProvenanceGuard.

4.1 Background: Provenance Analysis↩︎

Provenance analysis is a well-established discipline that studies the origin and derivation history of an artifact by explicitly tracking how it is produced from source inputs, intermediate transformations, and responsible actors. It has been widely applied in databases, scientific workflows, and security/privacy-critical systems, where understanding how a result is generated is critical for explanation, reproducibility, debugging and auditing [32], [33], [37], [38].

A central concept in provenance analysis is dependency tracing. The W3C PROV data model [37] is one widely-established standard of provenance. It models the causal dependencies between three primary components: Entities (digital artifacts or input data), Activities (processes or functions that consume and generate entities), and Agents (the operators or systems responsible for executing activities). In database literature [32], provenance is further classified into three critical dimensions: where-provenance, which tracks where an output value originates; why-provenance, which are the subset of source records necessary to justify the existence of an output; and how-provenance, which explains how outputs are constructed through transformations. Overall, across settings, provenance is a structured representation of how outcomes depend on prior context and computation. By exposing dependency structure, provenance supports debugging and error diagnosis, attribution of responsibility, and anomaly detection [32], [33], [38].

This paradigm offers a theoretical foundation for addressing LLM agent misalignment. Agents are tasked with completing the tasks expressed in user queries, and all agent behaviors should arise from system and user instructions, interactions with environments, and other information in the context window. Intuitively, whether an agent action is aligned depends on whether the action can be traced to appropriate context.

If an agent hallucinates a tool parameter or executes an irrelevant tool, this proposed action will fundamentally lack a valid provenance path back to the provided context. If the user query is underspecified and the agent picks an arbitrary interpretation, the proposed action will make ungrounded assumptions without the support of provenance. Consequently, checking misalignment based on provenance provides a systematic and principled mechanism for detecting unwanted agent behaviors in runtime environments.

4.2 Conceptual Framework for Provenance-based Guardrails↩︎

4.2.1 Basic Elements↩︎

We model the guardrail as a runtime monitor that intercepts each tool call proposed by the agent before execution. At every decision step, the guardrail receives the agent’s current context together with the newly proposed action, and determines whether executing that action would be aligned with the user query.

4.2.1.1 Context available to the guardrail.

The guardrail takes three core contexts as input: the User Query (\(q\in Q\)), the Available Tools’ Documentation (\(d\in D\)), and the History of Prior Tool Calls and Their Outcomes (\(h\in H\)). Together, these contexts capture the main sources from which an aligned action should derive its provenance. Although we focus on \((q,d,h)\) in this work, the framework is not limited to such and can be extended to incorporate additional context sources when available, e.g., system instructions, long-term memory, and retrieved documents.

4.2.1.2 Proposed action.

We represent a proposed tool call as \((t, a)\), where \(t\in T\) denotes the selected tool and \(a\in A\) denotes the corresponding parameter assignment. Tool documentation \(d\) records the specification of tools and their parameters. The part of \(d\) corresponding to a tool \(t\) is denoted as \(d_t\).

4.2.1.3 Guardrail input-output formulation.

Putting these pieces together, the guardrail \(guard\) takes the current context and the newly proposed action as input. Its output is a binary decision indicating whether the proposed action should be allowed to execute: \[guard(q, d, h, t, a) \in \{\textit{Aligned},\textit{Misaligned}\}.\]

4.2.2 Provenance Relations↩︎

Motivated by provenance analysis, we frame misalignment detection as the problem of tracing whether a proposed action is justified by the available context. An aligned action should be supported by traceable evidence drawn from the user query, tool documentation, and prior interaction history.

In this view, provenance is a justification structure that links a proposed action to contextual evidence through semantic relations between the elements of an action to those of the context. Provenance analysis then becomes the process of determining whether the relations required for an aligned action can be established from the available context.

Specifically, the three types of misalignment in Sec. 3 correspond to failures of three different justification conditions, each expressed in terms of provenance relations. Tool-level misalignment concerns whether the selected tool can contribute to the current subtask. Parameter-level misalignment concerns whether the instantiated parameter assignment is grounded in the current context. Interpretation-level misalignment concerns whether the proposed action is the only interpretation of the user query according to the context. An overview of the framework is presented in Fig.3.

Table 1 summarizes the provenance relations used by the framework. Next, we specify which relation must be identified for each type of misalignment and how failure of that relation is interpreted.

4.2.3 Provenance for Tool Choice↩︎

At this level, the object of justification is the tool selected for the current step. First, we need to obtain the documentation that specifies the selected tool \(t\). If the \(d_t\) satisfying \(SpecifiedBy(t,d_t)\) cannot be found, it means that \(t\) is not a valid tool available to the agent and should be rejected.

After identifying \(d_t\), tool-level provenance requires establishing the relation \(\mathit{RelevantTo}(d_t, s)\) with some subtask \(s\in S\) that satisfies \(ContributeTo(s,q)\). Intuitively, the tool of an aligned action must be capable of advancing the overall task given by the user. If \(\mathit{RelevantTo}(d_t, s)\) cannot be established, then the proposed action lacks the provenance required for tool selection, and the action is classified as misaligned. Taking Fig.1a as an example, “money_transfer” is not relevant to any part of the user query, so it is classified as misaligned.

4.2.4 Provenance for Parameter Assignment↩︎

Even when the selected tool is appropriate, the action can still be misaligned if its parameters are unsupported. A tool provides the means to act, but the parameter assignment determines what concrete action will actually be executed. Here, parameter-level provenance requires establishing two relations.

First, the selected parameter assignment needs to be derivable from the context. We denote factual evidence as \(e\in E\). A proper parameter assignment must be sourced from some facts in the context. Therefore, given \(a\), parameter-level provenance requires the relation \(\mathit{DerivedFrom}(a, e)\) to be satisfied for some evidence \(e\) that \(AppearIn\) \(q\) or \(h\).

Note that, it is not required to have an explicitly exact match for a parameter value in \(e\). A selected value is grounded as far as it can be reasonably derived from prior evidence. Conversely, if any part of a value cannot be traced back to any relevant contextual support, the relation \(\mathit{DerivedFrom}(a, e)\) will not hold.

Second, the grounded assignment must also constitute an appropriate instantiation of the selected tool that can resolve the current subtask \(s\) identified in the last stage. We capture this requirement using the relation \(\mathit{CanAddress}(d_t, a, s)\).

Both conditions are necessary. If \(\mathit{DerivedFrom}(a, e)\) cannot be established, then \(a\) lacks evidential provenance and may reflect hallucinated or arbitrarily chosen values. If \(\mathit{CanAddress}(d_t, a, s)\) does not hold, then \(a\), even if it can be derived from the context, does not provide the parameterization required for the current subtask. In either case, the proposed action fails the required parameter-level provenance condition and is classified as misaligned.

4.2.4.1 Generative Parameters Don’t Require Provenance

The conditions above do not apply uniformly to all parameters. Some arguments are inherently expected to be generated rather than copied or extracted. For example, for a send_email tool, the recipient field is often expected to be grounded in the query or in prior observations, whereas the message body may be newly composed by the agent. Such parameters may be coherent and useful even when they do not have direct provenance. This observation does not undermine the framework; instead, it clarifies that provenance-based checking should focus on parameters whose values are expected to be context-grounded. The framework thus distinguishes between derivable parameters, which should admit provenance tracing, and generative parameters, whose correctness may require other forms of evaluation. Accordingly, provenance-based checking at this level focuses on derivable parameters.

4.2.5 Provenance for Interpretation↩︎

In our approach, an action may be fully grounded under one interpretation of the query and still considered misaligned if that interpretation is only one of several plausible possibilities. Therefore, the key question of provenance analysis here is whether the available context sufficiently determines what the agent should do next, rather than leaving multiple reasonable choices of actions unresolved.

We say that a candidate action \((t, a)\) is provenance-admissible if it satisfies the required provenance conditions for tool and parameter alignment; namely, (i) \(\mathit{RelevantTo}(d_t, s)\), (ii) \(\mathit{DerivedFrom}(a, e)\), and (iii) \(\mathit{CanAddress}(d_t, a, s)\) for some \(s\) and \(e\) that satisfy \(ContributeTo(s,q)\) and \(AppearIn(e,q/h)\). The provenance analysis at this level is to attempt to identify the set of provenance-admissible candidate actions given the context.

If there exist distinct provenance-admissible actions for the current step, then the context does not uniquely constrain how the user’s request should be resolved. In this case, any concrete action makes an assumption that is not justified; i.e., it is considered misaligned at the interpretation-level. Taking Fig.1c as an example, the user query “handle it” is underspecified. Therefore, provenance-admissible actions include both “pay” and “decline” the request, and the agent selecting any one of them is considered a misaligned action in our approach.

4.3 ProvenanceGuard↩︎

We instantiate the above conceptual framework as ProvenanceGuard, a run-time guardrail that intercepts each proposed tool call before execution and applies provenance-based checks to determine whether the action should be allowed. Concretely, ProvenanceGuard operationalizes the three misalignment detection as a sequential three-stage pipeline, as shown in Fig. 3. Each stage targets one type of misalignment and performs an early rejection if insufficient provenance is found.

For the subset of the relations from Table 1 that require natural language understanding, ProvenanceGuard invokes an LLM to check whether a given tuple is in the relation or extract such a tuple from the given context. However, although ProvenanceGuard relies on an LLM for parts of the pipeline, we distinguish it from an LLM-as-a-judge, which typically involves a single call to an LLM to evaluate the agent’s action; we provide a comparison against an LLM-as-a-judge in Section 5.

4.3.1 Stage I: Detecting Tool-Level Misalignment↩︎

The first stage checks whether the selected tool \(t\) is an appropriate means of accomplishing some subtask \(s\) contributing to \(q\).

4.3.1.1 Reuse Agent’s Plan

Agents typically generate an explicit plan that specifies the subtask \(s\) addressed by each proposed action. ProvenanceGuard leverages this agent-provided subtask directly, rather than inferring one from the overall query \(q\). This design choice simplifies the procedure and avoids potential inconsistencies that could arise from independently inferred subtasks, which may otherwise lead to incorrect rejection of aligned actions.

4.3.1.2 Environment-Changing Tool Filter

To reduce unnecessary interference with the agent’s autonomy, ProvenanceGuard focuses its runtime checks on actions that can directly affect the external environment or produce consequential side effects. Examples include sending emails, editing files, or transferring money. In contrast, purely observational tools, such as search or information retrieval, typically have much lower risk and need not always be blocked prior to execution.

We assume that tool developers provide metadata indicating whether a tool is environment-changing. ProvenanceGuard uses this metadata as a lightweight rule-based filter before invoking the provenance checker. This design concentrates protection on high-impact actions while preserving the agent’s autonomy for low-risk information gathering.

4.3.1.3 Provenance check for tool selection

The selected tool \(t\) is first verified to exist in the documentation \(d\), from which its description \(d_t\) is retrieved. Then \(d_t\) is used to determine whether the tool is environment-changing.

Once \(t\) passes these two checks, ProvenanceGuard evaluates whether its use is justified for the current subtask \(s\). Concretely, it prompts an LLM to assess whether \(\mathit{RelevantTo}(d_t, s)\) holds.¹ If \(\mathit{RelevantTo}(d_t, s)\) cannot be established, ProvenanceGuard immediately labels the proposed action as Misaligned and blocks execution, without proceeding to later stages.

4.3.2 Stage II: Detecting Parameter-Level Misalignment↩︎

If Stage I determines that tool \(t\) is appropriate for subtask \(s\), Stage II proceeds to verify whether the proposed parameter assignment \(a\) is itself grounded in the available context.

4.3.2.1 Identifying verifiable parameters.

A practical complication is that not all parameters are expected to admit direct provenance tracing, as discussed in Sec.4.2.4. For this reason, the first step of Stage II is to infer which parameters of tool \(t\) are derivable and should be checked for provenance at runtime. We use an LLM to determine which parameters are derivable \(a_{\textit{derivable}}\) for \(t\) based on its documentation \(d_t\). If there is no derivable parameter, then Stage II performs no provenance check and the action is passed directly to the final stage.

4.3.2.2 Provenance check for parameter values.

Given \(a_{\textit{derivable}}\), ProvenanceGuard then instructs an LLM to find some evidence \(e\) such that \(\mathit{DerivedFrom}(a, e)\) is satisfied, and to establish \(\mathit{CanAddress}(d_t, a, s)\). If either relation cannot be satisfied, ProvenanceGuard immediately labels the action as Misaligned and stops.

4.3.3 Stage III: Detecting Interpretation-Level Misalignment↩︎

Passing the first two stages means that the proposed tool and its verifiable parameters are both grounded. However, the action may still be misaligned if it relies on an unjustified interpretation of an underspecified request.

Therefore, Stage III asks an LLM whether there exist more than one provenance-admissible candidate action given the context. Importantly, this step’s prompt does not reveal the proposed action to the LLM. This design avoids biasing LLM to the agent’s chosen action and reduces the risk that the model will simply rationalize that choice or deliberately search for a conflicting alternative.

If multiple provenance-admissible actions are identified, it means that multiple alternative interpretations of the user’s query remain and that executing any concrete action would involve the agent making an unjustified assumption. In this case, ProvenanceGuard labels the proposed action as Misaligned and blocks execution.

5 Experiment Setup↩︎

To demonstrate the effectiveness of ProvenanceGuard, we conducted experiments to answer the following two research questions:

• RQ1: How accurately does ProvenanceGuard detect misaligned actions?

• RQ2: How often does ProvenanceGuard incur interventions on correct actions?

RQ1 evaluates the effectiveness of ProvenanceGuard in detecting misaligned actions relative to the compared methods. RQ2 evaluates the practical oversight burden introduced by ProvenanceGuard as a runtime guardrail. To strengthen the validity of these comparisons, we further assess whether the performance differences between methods are statistically significant using one-sided paired permutation tests.

5.1 Benchmarks↩︎

Evaluating alignment guardrail imposes several requirements on the benchmark. (1) The benchmark must provide realistic agent interaction traces, or an executable environment from which such traces can be collected. (2) The agent must have access to a diverse set of tools that can directly affect the environment, since our setting focuses on actions with potentially irreversible consequences. (3) Most importantly, the benchmark must support accurate trace-level alignment annotation, either through human-annotated ground-truth labels or through a deterministic rule-based labeling procedure that does not depend on another LLM judge.

To our knowledge, no existing benchmark fully satisfies these requirements. In particular, many agent safety benchmarks focus on prompt injection, adversarial instructions, or other types of failures outside our scope, while others lack realistic traces, environment-changing tools, or suitable labels for our notion of misalignment. We therefore use two complementary benchmarks. WorkBench [35] provides agent traces, environment-changing tools, and ground-truth action sequences, making it a strong testbed for tool- and parameter-level misalignment. Agent-SafetyBench [34] complements it by including test cases with underspecified user queries, which closely matches our interpretation-level misalignment setting. Because Agent-SafetyBench does not provide traces or alignment labels, we manually generated traces and annotated them.

5.1.1 WorkBench↩︎

WorkBench contains 690 agent test cases with traces generated by multiple LLMs, together with mock environments, mock tools, tool documentation, and ground-truth action sequences. We use the GPT-4 traces, which are reported as the strongest in the original benchmark. This benchmark is useful for our setting because its ground-truth action sequences let us identify traces containing tool invocations that do not follow the user’s expected execution. However, WorkBench does not explicitly consider underspecification as a kind of misalignment.

We treat traces that take any environment-changing actions deviating from the benchmark’s expected action sequence as misaligned traces. This is because, in these cases, at least one executed tool call fails to match the benchmark’s ground-truth actions for the task, either through misaligned tool selection, or misaligned parameterization. In particular, some traces may also have underspecified user query, so we do not interpret this subset as containing only tool- or parameter-level misalignments.

All remaining traces, which correctly complete or partially complete their tasks, are referred to as task-successful traces. These task-successful traces are not necessarily equivalent to aligned traces in our setting, because matching the ground-truth execution does not rule out underspecification in the original user query.

We restricted the benchmark to test cases involving environment-changing tools. The resulting subset contains 62 misaligned traces and 326 task-successful traces.

5.1.2 Agent-SafetyBench↩︎

We used the subset of Agent-SafetyBench with “failure mode 2,” which is marked as the agent mistakenly calls tools when the necessary information is incomplete. This failure mode closely matches our interpretation-level misalignment, since these cases largely arise from underspecified user queries.

Agent-SafetyBench does not provide agent execution traces or trace-level alignment annotations. We therefore used GPT-5, with default configuration, to generate one execution trace for each test case, yielding 188 traces in total. Two authors then independently annotated each trace for (1) whether it contains a misaligned action and (2) the first misaligned step, if any. Out of the 188 annotations, eight had inconsistent labels across the two annotators. Inter-annotator agreement reached Cohen’s \(\kappa = 0.85\), indicating almost perfect agreement according to [39]. Two annotators had a discussion to resolve the disagreed cases and filtered out 5 inappropriate cases. The resulting dataset contains 35 aligned traces and 148 misaligned traces.

5.1.3 Plan reconstruction↩︎

ProvenanceGuard requires the agent’s plan as part of its input. However, neither Agent-SafetyBench nor WorkBench explicitly outputs plans in the traces. To recover this information, we used GPT-5 to infer the agent’s plan from each decision point. Specifically, given \((q, d, h, t, a)\), GPT-5 reconstructed the subtask \(s\) that the agent intended to address with\((t, a)\).

5.2 Compared Methods↩︎

We compared ProvenanceGuard against a prompt-based baseline and a single-step variant derived from our conceptual framework.

5.2.0.1 DirectPrompt Baseline

This method represents the LLM-as-a-judge paradigm [25]. Given the agent context and a proposed action, the model is directly prompted to determine whether the action is misaligned according to the definitions in Sec.3. ²

5.2.0.2 Single-step Provenance Prompt.

ProvePrompt is a single-step prompt implementation of our provenance-based conceptual framework. Compared to DirectPrompt, its prompt explicitly instructs the model to assess provenance for tool choice, parameter assignment, and interpretation-level justification. The main difference between it and ProvenanceGuard is that it produces a single holistic judgment in one pass rather than decomposing the decision into a multi-stage pipeline. We include this method to isolate the contribution of the provenance-based conceptual framework from that of ProvenanceGuard’s multi-stage design.

5.3 Backbone Large Language Models↩︎

We evaluated ProvenanceGuard and the two compared methods using 10 backbone LLMs: GPT-5, GPT-5-mini, GPT-4.1, GPT-4.1-mini, Gemini-3-flash-preview, Gemini-3.1-flash-lite-preview, Gemini-2.5-flash, Gemini-2.5-flash-lite, Claude-sonnet-4.6, Claude-haiku-4.5 [4], [40], [41]. All models were used with their default inference configurations provided by the corresponding API.

5.4 Setup and Metrics↩︎

5.4.1 RQ1: Misalignment Detection Performance↩︎

This RQ assesses how effectively ProvenanceGuard and the two compared methods detect misaligned actions. For each combination of method and backbone LLM, we evaluate on the misaligned traces from Agent-SafetyBench and WorkBench. To account for run-to-run variance in LLM judgments, each method-model pair is executed five times on each trace.

5.4.1.1 Metrics

We report the error rate, defined as the proportion of misaligned traces that are incorrectly judged as aligned. Equivalently, when misaligned traces are treated as the positive class, this metric corresponds to the false negative rate (FNR).

For a given method-model pair, the Error Rate is computed as: \[\mathrm{ER} = \frac{\sum_{r=1}^{R} \#\mathrm{FN}_r}{N \times R},\] where \(N\) is the number of misaligned traces, \(R\) is the number of repeated runs, and \(\#\mathrm{FN}_r\) denotes the number of misaligned traces predicted as “Aligned” (False Negatives) in run \(r\).

5.4.2 RQ2: Intervention Frequency↩︎

Accurately detecting misaligned actions is only one side of the trade-off. A practical guardrail should also avoid interrupting agent executions that would otherwise proceed acceptably, since excessive interventions increase user review burden.

We therefore evaluated the intervention rate of ProvenanceGuard from two complementary perspectives, reflecting the different labeling semantics of our two benchmarks.

5.4.2.1 RQ2.1: Unnecessary interventions on aligned trace.

On Agent-SafetyBench, we used the 35 traces that were manually annotated as aligned, where any interventions are unnecessary. This evaluation provides a direct measure of how often the method disrupts actions that are aligned with user query, which corresponds to the false positive rate (FPR).

5.4.2.2 RQ2.2: Interventions on task-successful traces

On WorkBench, we used the 326 task-successful traces. Here, we measure how often the guardrail intervenes on traces where agents are correctly addressing the user task. We do not interpret this quantity as a FPR, because correctness with respect to benchmark ground truth is not equivalent to alignment in our setting, as discussed in Sec.5.1.1. Therefore, this metric captures the cost of intervention on traces that would otherwise be considered successful by the benchmark.

5.4.2.3 Metrics

We report the intervention rate, defined as the proportion of evaluated traces for which the method predicts Misaligned. For RQ2.1, this corresponds to the unnecessary intervention rate. For RQ2.2, this corresponds to the intervention rate on task-successful traces, which should be interpreted as an operational burden measure rather than a strict false positive rate. \[\mathrm{IR} = \frac{\sum_{r=1}^{R} \#\mathrm{Intervene}_r}{N \times R},\] where \(\#\mathrm{Intervene}_r\) is the number of traces predicted as Misaligned in run \(r\). Its interpretation depends on the ground-truth semantics (aligned or task-successful) of the evaluated subset.

5.4.3 Statistical Analysis↩︎

For both RQs, we assessed whether differences between methods are statistically significant using one-sided paired permutation tests. Because each method-LLM pair was executed five times on the same trace, we do not treat individual runs as independent observations. Instead, for each method, LLM, and trace, we averaged the five binary outcomes to obtain a case-level score. For RQ1, this score is the error rate on a misaligned trace. For RQ2, this score is the IR on the evaluated trace.

We then compared ProvenanceGuard against each method using paired case-level differences computed on the same traces under the same LLM. For a comparator method \(m\), let \(s^{(c)}_{\mathrm{\small ProvenanceGuard}}\) and \(s^{(c)}_{m}\) denote the case-level scores of ProvenanceGuard and \(m\) on trace \(c\), respectively. We define the paired difference as \[d_c = s^{(c)}_{\mathrm{\small ProvenanceGuard}} - s^{(c)}_{m}\;,\] so that a negative value indicates that ProvenanceGuard achieves a lower rate than the comparator on that trace. The test statistic is the mean paired difference, \[\Delta = \frac{1}{N} \sum_{c=1}^{N} d_c \;,\] where \(N\) is the number of traces in the evaluated subset.

The null hypothesis is that ProvenanceGuard does not outperform the comparator. The one-sided alternative hypothesis is that ProvenanceGuard yields a lower case-level rate than the comparator. We estimate permutation \(p\)-values via Monte Carlo sampling over 10,000 random sign flips.

6 Evaluation Results↩︎

6.1 RQ1: Misalignment Detection Performance↩︎

Fig.4 (a)&4 (b) show ERs on misaligned traces, where lower is better. Overall, our experiments show that the provenance-based framework consistently improves misalignment detection.

On Agent-SafetyBench, both ProvenanceGuard and ProvePrompt outperform the baseline on all 10 LLMs. Averaged across LLMs, the ER drops from 42.9% for DirectPrompt to 12.8% for ProvePrompt and further to 1.8% for ProvenanceGuard. As reflected in Table 2, all performance gains over the baseline are statistically significant (\(p<0.05\)). The gains are especially large for weaker models. With ProvenanceGuard, even models such as GPT-4.1-mini, Gemini-2.5-flash-lite, and Claude-haiku-4.5 reach ERs close to the strongest models, and all 10 LLMs fall below 5% ER. This suggests that the provenance-based framework provides useful structure for detecting misaligned actions, even when the underlying model is relatively weak.

On WorkBench, the improvements are more moderate. Here, ProvenanceGuard outperforms the baseline on 8 of 10 LLMs, while ProvePrompt does so on all LLMs. Averaged across models, the ER decreases from 32.1% for DirectPrompt to 19.5% for ProvePrompt and 17.3% for ProvenanceGuard. Table 2 shows that all improvements of ProvenanceGuard are statistically significant except for Claude-haiku-4.5, whose result is borderline (\(p=0.06\)). In particular, ProvenanceGuard does not improve over the baseline for GPT-5 or Claude-sonnet-4.6, while ProvePrompt has non-statistically significant improvements on these two LLMs.

The inconsistent improvements on two benchmarks are probably due to their distinct distribution of traces. The Agent-SafetyBench subset is intentionally concentrated on underspecified queries. In that setting, ProvenanceGuard is highly effective and nearly eliminates false negatives across all models. WorkBench, in contrast, is not concentrated on underspecification and appears to contain a broader mix of misalignment types. This suggests that the multi-stage pipeline is particularly valuable for identifying underspecification, while detection of other types of misalignment still depends strongly on backbone LLMs.

Comparing the two provenance-based variants also yields an important insight. On both benchmarks, ProvePrompt already improves substantially over the baseline, showing that much of the benefit comes from the provenance-based conceptual framework itself rather than only from the multi-stage pipeline. At the same time, ProvenanceGuard is clearly stronger on Agent-SafetyBench, while on WorkBench the advantage of the multi-stage design is less uniform and ProvePrompt outperforms ProvenanceGuard on five LLMs. This suggests that the benefit of the conceptual framework is consistent across implementations, whereas the additional value of multi-stage decomposition depends on the benchmark distribution and model characteristics.

6.1.0.1 Takeaway

Provenance-based framework substantially improves misalignment detection over DirectPrompt. Among them, ProvenanceGuard achieves better performance overall on both benchmarks than the single-step variant, especially on Agent-SafetyBench, suggesting the benefit of the multi-stage separation.

6.2 RQ2: Intervention Cost↩︎

Fig.4 (c)&4 (d) show IRs on the two benchmarks, where lower is better.

6.2.0.1 RQ2.1 Unnecessary Interventions on Aligned Trace

On Agent-SafetyBench aligned traces, the three methods have similar average IRs: 10.1% for the baseline, 10.9% for ProvePrompt, and 14.3% for ProvenanceGuard. Compared to the baseline, ProvenanceGuard appears to yield a higher unnecessary IR on 8 of 10 LLMs (average increase of 8.0% on these 7 LLMs).

To test the significance of these increases, we verified the reverse one-sided alternative hypothesis that ProvenanceGuard yields a higher case-level unnecessary IR than the comparator. According to Table 3, 7 out of 8 increases are actually not statistically significant. This indicates that the large gain of ProvenanceGuard in detecting misaligned actions (41.1%) is achieved with a limited tradeoff in unnecessary intervention cost.

6.2.0.2 RQ2.2 Interventions on Task-successful Traces.

On WorkBench task-successful traces, the pattern is notably different. Here, ProvenanceGuard has a lower IR on all 10 LLMs than the other two methods, and reduces the average rate from 30.5% for DirectPrompt to 12.8%. The reduction is substantial for several strong models: GPT-5 (\(45.2\% \rightarrow 13.2\%\)), Gemini-3-flash \((41.4\% \rightarrow 17.2\%\)), and Claude-sonnet-4.6 (\(39.4\% \rightarrow 4.8\%\)). This suggests that, in practice, the multi-stage provenance checks do not simply lead to more interventions than the baseline. Instead, they often make judgments more precisely, allowing many successful traces to pass while detecting misalignment with few false negatives.

6.2.0.3 Comparing provenance-based variants

The comparison between the two provenance-based variants further highlights the value of the multi-stage design. ProvePrompt has higher average IRs than the other two methods on both subsets: 17.5% on Agent-SafetyBench and 34.6% on WorkBench. On WorkBench in particular, it intervenes more often than ProvenanceGuard on all 10 LLMs (9 of them are statistically significant), often by a large margin (e.g., 49.5% vs.% on GPT-5 and 52.0% vs.% on Gemini-3-flash). This indicates that while the provenance-based framework improves detection, the multi-stage decomposition is important for controlling over-intervention in practice.

6.2.0.4 Takeaway

ProvenanceGuard does not statistically significantly increase unnecessary interventions on already aligned traces, but it substantially reduces interventions on task-successful traces compared to both DirectPrompt and ProvePrompt. Combined with the RQ1 results, this suggests that the multi-stage design built on the provenance-based framework achieves a better balance between detecting misaligned actions and avoiding excessive disruption.

6.3 Threats to Validity↩︎

Our labeled Agent-SafetyBench includes only 35 aligned cases. This small sample limits statistical power, so the lack of a statistically significant difference in unnecessary interventions between ProvenanceGuard and the baseline should not be over-interpreted. It may reflect insufficient evidence to detect a difference, but it may also indicate that ProvenanceGuard does not meaningfully increase unnecessary interventions relative to the baseline. More aligned test cases would be needed to more reliably assess how ProvenanceGuard affects unnecessary intervention behavior.

WorkBench does not explicitly consider underspecification as misalignment. Its “task-successful” traces may have underspecified user queries. As a result, they could not be interpreted as fully aligned traces. We therefore treat the intervention rate on these traces as a measure of the oversight burden in practical usage rather than a strict unnecessary intervention rate.

The agents’ plans in both datasets were reconstructed using GPT-5, since neither benchmark provides such information directly. Errors in this reconstruction could affect ProvenanceGuard’s decisions and the measured performance.

7 Related Works↩︎

Guardrails, the external defense layers that monitor and control LLM interactions, have emerged as a crucial approach for improving the safety of LLM agents. However, prior works have largely focused on safety, security, and policy compliance [23], [42]–[46]. For example, GuardAgent [23] and ShieldAgent [42] safeguard agent behavior against violations of safety or policy constraints, while AgentDojo [43] and CaMeL [44] focus on security threats such as prompt injection and adversarial tool environments. Although related, these problems differ from our notion of misalignment. Our focus is not on harmful, adversarial, or policy-noncompliant behavior, but on whether a proposed tool invocation is grounded by evidence in the context without making an arbitrary assumption about an underspecified user query in benign settings. This distinction is important because some prior work also uses the term “misalignment” to describe strategically harmful or insider-threat behavior by agents [10], [47], which is conceptually different from the misalignment we study.

Another related work to ours is InferAct [24]. Although both works perform pre-execution intervention, they are not directly comparable. InferAct leaves the notion of misalignment largely implicit and operationalizes detection through a holistic prompt, asking whether the agent is “progressing correctly toward completing the user’s tasks” without precisely defining what it means for the agent to be making "progress". In contrast, our work addresses a different and more explicitly specified decision problem: Given a proposed action, determine whether its tool selection and parameterization are justified by traceable contextual evidence, and whether the context uniquely supports committing to that action.

One concept close to underspecification is ambiguity. Recent works study whether language models can recognize ambiguous user requests and ask clarification questions when needed [36], [48]–[51]. This line of work is related to our interpretation-level analysis, but the concepts are not identical. In our setting, underspecification means that the available context is insufficient to uniquely determine the next action, even if the request is not linguistically ambiguous in the usual sense. Our focus is therefore not simply on ambiguity in language, but on whether the current context uniquely justifies one provenance-admissible action.

8 Discussion and Limitations↩︎

Overall, our results suggest that the proposed provenance-based conceptual framework enables a more structured and reliable way to detect misaligned agent actions. At the same time, the current framework has several limitations.

The proposed framework cannot detect alignment of generative parameters. Our provenance analysis focuses on parameters that should be derivable from context. For free-form generated content such as drafted text or code, provenance tracing is insufficient by nature. Therefore, our method is designed to identify the generative parameters and exclude them from provenance-based checking. Detecting misalignment for them requires other forms of guardrails.

ProvenanceGuard checks provenance relative to the agent’s own plan. If that plan is itself already misaligned with the user query, then a locally justified action may still be globally misaligned. One possible alternative is to let the guardrail infer its own task decomposition. However, this would introduce additional latency and create a new risk that the guardrail’s inferred plan is misaligned, or conflicts with an aligned agent plan. We therefore chose to use the agent’s plan in the current design.

The multi-stage design introduces extra latency and inference cost compared to a one-shot judge. However, our results suggest that smaller and cheaper models such as GPT-5-mini can achieve misalignment detection performance comparable to stronger models such as GPT-5, potentially offering a better cost-performance trade-off. Exploring this trade-off more systematically is an important direction for future work.

9 Data Availability↩︎

The replication package of the paper, which provides the labeled benchmarks, source code, experiment results, and analysis notebooks, will be available soon.

References↩︎

---

1. All prompts are provided in the replication package.↩︎

2. Full prompts are provided in the replication package.↩︎