1 Introduction↩︎

Figure 1 shows an email agent asked to send a contract to Alice. Its prompt also contains a trusted directory result, stale memory, and an untrusted note naming other recipients. The final call may be correct even if its recipient remains sensitive to the stale or untrusted context. The trusted lookup can mask this influence; if that lookup is missing, the same context may determine the recipient. Outcome benchmarks cannot distinguish a well-grounded decision from a brittle one that happens to be correct.

Tool-use and agent benchmarks primarily evaluate task completion or action correctness [1]–[8]. Prompt-injection and instruction-hierarchy studies test whether lower-priority content can override a current goal [9]–[14]. Attribution can identify influential context, but not whether that context was allowed to determine the decision. These evaluations therefore do not generally ask whether a particular piece of evidence is permitted to determine a particular action component. Our audit takes an authorization rule supplied by the application: in our tasks, the current user goal, trusted observations, and applicable policies determine which sources may support each decision. Unauthorized evidence need not be false or malicious; it includes both benign residues and potentially adversarial text.

We audit the tool choice and each argument value separately because authority is target-specific. In the email example, a stale address does not compete with the decision to call send_email, but it directly competes with the recipient argument. We decompose the task context into semantic factors and label each factor for each target. A Valid factor provides authorized support; an Invalid factor is unauthorized and supplies a concrete alternative for the same decision variable; other factors are Neutral. We use these labels to describe permission, not general truthfulness.

We organize the audit around three questions. First, source response: does a model respond when only the source authority of evidence changes? Second, isolation: does unauthorized competition still alter actions, especially after valid evidence is removed? Third, localization: which combinations show non-additive score dependence involving unauthorized competitors under partial evidence?

The primary matched intervention holds the task, proposition, position, and policy fixed while changing only whether the proposition comes from an authorized source. Target-score comparisons measure the model’s response to this source change, and paired generations test whether it reaches the produced action. We then remove individual valid factors and compare contexts that retain or remove unauthorized competitors. Finally, as a secondary diagnostic, we score controlled subsets of the task factors and use Harsanyi/Shapley interactions to localize combinations involving invalid competitors. Because the last analysis varies the available evidence, it describes partial-evidence interactions rather than full-prompt failure prevalence.

We evaluate 450 controlled next-action tasks from an authored workflow set, Tau2-style tasks [5], [15], and BFCL examples [3]. Across multiple open-weight model families, changing only source authority shifts target scores; trusted competitors reduce target support more than their matched untrusted variants. The source change also alters generated actions in 5.4% of competing cases versus 1.7% of supporting cases. In 2.4% [2.1, 3.0] of valid-evidence degradation comparisons, the full and clean contexts produce the reference target while the mixed context does not. This deletion pattern associates the error with retaining the invalid set; it does not identify a particular factor as its cause. Coalition-averaged Shapley interactions show positive fractional excess under partial evidence, while the full-context transform is near its null. Strict role matching further limits an authorization-specific interpretation. The models therefore respond to source authority without fully isolating their actions from unauthorized evidence.

Our contributions are:

• We formulate target-specific authorization auditing from application rules and construct a 450-task diagnostic suite with separate factor labels for each tool and argument target.

• We combine matched source interventions with controlled evidence degradation to distinguish response to source authority from behavioral isolation.

• We complement the behavioral tests with an exact partial-evidence interaction diagnostic and identify its full-context and role-matching boundaries.

2 Method↩︎

The audit separates three questions. Matched source interventions measure whether models respond to source authority, controlled degradation tests whether unauthorized competitors still alter actions, and a secondary coalition analysis characterizes their non-additive score dependence as the available evidence is varied.

2.1 Audit Targets and Authorization Labels↩︎

The action interface remains fixed throughout the audit: system instructions, tool schemas, and output-format instructions are unchanged. We first decompose the candidate task-specific context into semantic factors, including current goals, trusted observations, memory, retrieved records, previous actions, metadata, and untrusted notes. Each candidate factor is a coherent operational unit that can be included or removed while preserving the fixed task frame: the action interface and target question. The decomposition is part of the audit specification; source-specific construction rules appear in the appendix. Ambiguous material and fixed-interface content are marked Excluded. The remaining auditable factors form \(X=\{x_1,\ldots,x_n\}\).

Let \(y\) denote the fixed target string. A target may be a tool name, such as send_email, or an argument assignment, such as recipient = alice@company.com. Auditing one target at a time separates tool selection from argument selection. The same context factor can therefore have different authorization labels for the tool and its arguments.

An application-level rule specifies which sources may determine each target. For every target–factor pair, the audit then separates two judgments. The authority judgment asks whether the factor’s source is permitted to determine the target. The relation judgment asks whether its proposition supports \(y\), supplies a concrete competitor for the same decision variable, or does neither. Current user goals, trusted observations, and applicable policies provide the authorization chains in our tasks.

The structural labels combine these judgments. A factor is Valid if it is authorized and supports \(y\). It is Invalid if it is unauthorized and supplies a competing value, entity, action, directive, or previous-action residue. All other auditable factors are Neutral. Thus Invalid is narrower than unauthorized, and Neutral does not mean irrelevant: it includes, for example, an unauthorized proposition that supports \(y\). Topical relatedness alone is also insufficient for an invalid label; stale memory or a neighboring record is invalid only when it competes at the audited decision variable. The matched experiment crosses authority and relation directly, including authorized competitors and unauthorized supporters, rather than treating its four cells as additional structural labels.

2.2 Target Scores and Generated Decisions↩︎

For any prompt \(P\), let \(y=(y_1,\ldots,y_m)\) be the tokenized target. We measure model support for \(y\) with summed target-token log-odds: \[\begin{align} q_t(P) &= p_\theta(y_t \mid P,y_{<t}),\\ v_y(P) &= \sum_{t=1}^{m} \left[\log q_t(P)-\log(1-q_t(P))\right]. \end{align}\] Higher \(v_y(P)\) means greater support for the fixed target. We use this token-wise score because comparisons are paired within target, keeping its string and tokenization fixed; summed log-probability and other fields provide robustness checks in Appendix 11.

For behavioral endpoints, we instead decode a complete action and parse the audited tool or argument. We record whether the parsed target matches the reference and whether paired prompts produce different targets.

2.3 Controlled Valid-Evidence Degradation↩︎

For each target with a Valid factor \(x\), the full context contains all factors, mixed removes only \(x\), and clean removes \(x\) plus all Invalid factors. The mixed–clean contrast tests retained unauthorized competition after valid evidence is removed. Let \(c_F,c_M,c_C\) indicate whether the parsed full, mixed, and clean targets match the reference. The primary endpoint is mixed–clean target discordance. The stricter retained-invalid error pattern is \(c_F=1,c_M=0,c_C=1\): the target is correct before degradation and after the invalid set is removed, but wrong when that set remains. Because mixed and clean differ by joint deletion of all Invalid factors, this is an intervention pattern, not formal mediation or proof that one factor caused the error. Degraded prompts may also be underdetermined; these rates characterize a controlled stress test, not deployment errors.

2.4 Matched Source Interventions↩︎

Our primary design crosses source authority \(a\in\{T,U\}\) with semantic relation \(r\in\{S,C\}\). Here \(T/U\) denote trusted/untrusted sources, while \(S/C\) denote propositions that support the reference target or supply a concrete competitor. A fixed policy authorizes current user requests, trusted current records, and explicit policies, but not external notes or stale and neighboring records. Thus \(T/U\) denote target-specific authority under this policy, not global source quality.

Each proposition is added to the same valid-and-neutral base. Within relation, only the frame changes in the exact template Source status: [frame]. Proposition: [text]; degradation separately tests the original mixed context. This textual marker is a controlled proxy for source authority, not an operational provenance channel such as retrieval or tool transport.

Table 1 shows the four treatments for one AgentAudit target.

Let \(v_0\) be the target score on the shared base and \(v_{a,r}\) the score after adding cell \((a,r)\). We first define the within-relation authority gaps \(\delta_S\) and \(\delta_C\), then compare them with \(\Delta\Delta\): \[\begin{align} G_a &= v_{a,S}-v_0, & D_a &= v_0-v_{a,C},\\ \delta_S &= G_T-G_U, & \delta_C &= D_T-D_U,\\ \Delta\Delta &= \delta_C-\delta_S. \end{align}\] \(\delta_S\) and \(\delta_C\) are the matched authority effects for supporting and competing propositions. \(\Delta\Delta\) is secondary because propositions are not matched across relations; it describes whether the authority gap is larger for competition. Intervals cluster-bootstrap source-task instances.

For generated actions, the corresponding endpoint is paired target discordance between trusted and untrusted variants within each relation. Discordance records a source-induced action change, not necessarily an error. Directional correct/incorrect transitions distinguish harmful from corrective changes.

A same-proposition control also changes only the source marker of an existing competing factor and compares its removal gain, checking the addition-based construction.

2.5 Partial-Evidence Interaction Diagnostic↩︎

Full-context deletion can miss dependence that emerges after other evidence disappears: a stale address may be inert beside a trusted lookup yet influential without it. We therefore use coalition analysis as a secondary partial-evidence stress diagnostic. For every subset \(Z\subseteq X\), we construct a prompt \(P(Z)\) containing the fixed interface and only the task factors in \(Z\). The target question remains unchanged; the empty subset contains the task frame but no task-specific factors. We write \(v_y(Z)=v_y(P(Z))\).

For each non-empty factor set \(B\subseteq X\), its exact Harsanyi interaction is [16], [17] \[I_y(B)=\sum_{Z\subseteq B}(-1)^{|B|-|Z|}v_y(Z).\] \(I_y(B)\) is the non-additive score variation assigned to \(B\) after subtracting its lower-order components. First-order terms are individual-factor effects; higher-order terms capture changes that emerge only when factors occur together. We report three evidence-availability estimands from the same subset lattice: empty-coalition Harsanyi effects characterize sparse evidence, Shapley interactions average over background coalitions, and a full-context transform characterizes deletions near the complete prompt. The Shapley fractional excess is our primary partial-evidence summary; empty and full anchors define its sparse- and complete-evidence boundaries.

The main datasets contain eight factors, allowing all \(2^8\) subsets to be scored exactly. For each estimand, we rank its coefficients by absolute strength and retain the top \(K=50\) terms of order at most three; exact transforms and sensitivity checks are given in Appendix 12.4.

Let \(X_y^I\) and \(X_y^V\) be the invalid and valid factors for target \(y\). An interaction is invalid-containing if it includes any member of \(X_y^I\); the remaining interactions are valid-containing or neutral-only. Let \(\Omega_y\) denote the label-independent top-\(K\) set. We first record whether selected interaction strength contains any invalid factor: \[s_{\mathrm{any}}(y)= \frac{\sum_{B\in\Omega_y}\mathbf{1}[B\cap X_y^I\neq\emptyset]|I_y(B)|}{\sum_{B\in\Omega_y}|I_y(B)|}.\] Because this presence measure gives full credit to a mixed set, our primary summary allocates strength in proportion to its fraction of invalid factors: \[s_{\mathrm{frac}}(y)= \frac{\sum_{B\in\Omega_y}|I_y(B)|\,|B\cap X_y^I|/|B|}{\sum_{B\in\Omega_y}|I_y(B)|}.\] Higher-order sets have more opportunities to contain an invalid factor. For both summaries, a label-count-matched null permutes labels while preserving their counts over 500 permutations. We treat observed-minus-null fractional excess as primary, any-invalid excess as secondary, and raw shares as descriptive. We also separate invalid-only, invalid–valid, and invalid–neutral mass. Appendix 13 reports signed, aggregation, order, and top-\(K\) checks.

Both shares are unsigned attribution diagnostics. They do not indicate whether an invalid factor helps or harms the target, and they are not action-error probabilities. Because subset prompts remove available evidence, these metrics localize partial-evidence interaction structure. Full-prompt provenance response is measured by the matched intervention and generated-action endpoints.

3 Datasets and Annotation↩︎

3.1 Goal and Scope↩︎

We construct controlled diagnostic datasets for fixed next-action decisions. Each included task has an explicit current goal, trusted evidence that determines a unique reference action, and precise values for the audited arguments. We exclude tasks with ambiguous goals, underspecified values, or multiple equally acceptable actions. The resulting datasets test sensitivity to authorized and unauthorized context; they do not estimate the prevalence of such context in deployment. All examples are authored or benchmark-derived and contain no real user conversations or personal records.

3.2 Dataset Construction↩︎

All three sources use the same target–factor interface. Each instance contains a fixed action interface, eight removable task-specific factors, one tool target, and two argument targets. Factors are semantic units of evidence rather than arbitrary token spans. The system instruction, tool schemas, and output format remain fixed and are not annotated as task factors. The eight factors cover current intent, trusted evidence or policy, competing context, and neutral context; source-specific selection rules appear in the appendix. Across 450 instances, this construction yields 1,350 audited targets and 10,800 target–factor labels. Table 2 summarizes the sources.

2.5pt

AgentAudit-250 is authored directly as fixed next-action tasks. Each task specifies a current user goal, a reference action, a compact tool interface, and mixed context from workflow, retail, travel, data/file, or code editing settings. Tau2Audit-100 converts relevant retail and airline states from Tau2-style environments into textual factors; no environment transition is simulated during scoring [5], [15]. BFCLAudit-100 rewrites multi-function BFCL examples as next-call decisions with candidate tool schemas and task evidence [3]. The two benchmark-derived sources test the audit outside the authored workflow templates.

3.3 Annotation Protocol and Reliability↩︎

The author team labeled the full dataset using the target-specific rules defined above. The four possible labels are Valid, Invalid, Neutral, and Excluded. Labels are assigned independently for every target–factor pair and are not copied across the three targets in an instance. Annotators first apply the authorization test and then, when it fails, the competition test. For every Invalid label, they record the concrete competing value, entity, action, directive, or previous-action residue that justifies the label. We do not use agreement within the author team as evidence of reliability; the independent relabeling below provides that check.

Three external PhD-level annotators independently relabeled a stratified subset of 300 target packets: 150 from AgentAudit and 75 from each benchmark-derived source, with tool and argument targets represented in each. Each packet contains one target and its eight factors, giving 2,400 target–factor labels per annotator. Annotators saw the target, factors, and annotation guide, but not model scores, interaction outputs, original labels, or one another’s decisions. Majority vote resolves 2,394 labels. After computing agreement, the author team adjudicates the six cases with no majority.

Table 3 reports agreement before adjudication. Exact four-label agreement is 0.671 with Fleiss’ \(\kappa=0.608\). Collapsing the labels to invalid versus non-invalid gives agreement 0.854 and \(\kappa=0.626\). Because invalid is a minority label, we report \(\kappa\) alongside raw agreement. The adjudicated consensus matches 78.9% of the author labels; this is descriptive, not an accuracy estimate. Primary analyses use author labels. On the relabeled subset, consensus labels rerun only coalition metrics, not the author-label-derived degradation prompts.

4pt

Release artifacts include data, labels, prompts, code, seeds, and relabeling records; Appendix 10 documents annotation and conversion details.

4 Experiments↩︎

We ask three questions. First, does changing only a proposition’s source alter target scores and generated actions? Second, when valid evidence weakens, does unauthorized competition affect behavior? Third, can coalition interactions localize this dependence under partial evidence? We answer these questions in order, then examine scaling and a simple authority-policy guardrail.

4.1 Setup↩︎

The primary score, degradation, and coalition experiments use Qwen3-4B, Qwen3-30B, Mistral-Small-3.2-24B, and Llama-3.3-70B. Generated-action and guardrail experiments add DeepSeek-V2-Lite. The stricter same-proposition control uses Qwen3-4B, Ministral-8B, Mistral-Small-3.2-24B, and DeepSeek-V2-Lite; the scaling study uses Qwen3-4B, Mistral-Small-3.2-24B, and Ministral-8B. Exact scoring evaluates every factor subset for each of the 450 tasks. Unless noted otherwise, pooled 95% confidence intervals cluster-bootstrap source tasks, keeping their targets and model evaluations together. Endpoint-specific model coverage and completed runs are reported in Appendix 16, together with decoding and parsing rules. Appendix Table [tab:closed-api-proxy] reports a black-box forced-choice proxy on GPT-4o-mini/4o; only GPT-4o-mini separates high from low.

4.2 Matched Full-Prompt Score Response↩︎

The matched factorial adds one proposition to the same complete non-invalid base. Within each relation, proposition text and position are fixed; only its trusted or untrusted source marker changes. Support changes are +0.448 for trusted sources and \(-0.052\) for untrusted sources, while competition reduces the target score by 2.079 and 1.098, respectively. These are signed quantities: the negative untrusted-support value means that adding the proposition slightly lowers the target score. More importantly, an explicitly untrusted competitor still causes a large score drop. The authority-by-relation difference-in-differences is +0.481 [0.413, 0.550] and is positive for every model (Figure 2).

4pt

The factorial compares different propositions across relations, so its difference-in-differences is secondary. Our cleanest identification check instead holds one existing competing proposition fixed, changes only its source marker, and measures its removal effect. The trusted-minus-untrusted gap is positive for all four models and is +1.15 log-odds [0.97, 1.33] pooled (Table 4). Thus models respond to the textual source cue in the complete prompt. This control does not test whether they track provenance through operational retrieval, memory, or tool channels.

4.3 Matched Generated-Action Sensitivity↩︎

We next decode actions under the same complete-base, source-marker-only intervention. Discordance means that the parsed audited tool or slot differs between the trusted and untrusted versions; it measures source sensitivity, not by itself error or harm. Across five models, discordance is 1.7% for supporting propositions and 5.4% for competing propositions, a +3.8 percentage-point contrast [3.2, 4.4] (Table 5). The model-level contrast is positive in all five cases but ranges from +0.2 to +8.7 points; Llama and DeepSeek show the largest changes (+7.3 and +8.7), so the pooled contrast is not a uniform cross-model effect.

The direction of these changes is informative. Under competition, switching from a trusted to an untrusted source changes 1.2% of targets from correct to wrong, but 4.0% from wrong to correct. The corresponding support transitions are 1.0% and 0.6% (Appendix Table ¿tbl:tab:transition-matrix?). Thus source demotion is net corrective under competition, as expected if the model partly discounts the competitor. Our claim is narrower: the residual 1.2% harmful transitions, 5.4% discordance, and score effects show that this discounting is neither complete nor uniform; demotion is not harmful on average.

1.8pt

4.4 Behavior under Valid-Evidence Degradation↩︎

To test whether the partial-evidence diagnostic relates to decisions, we decode every model on three paired contexts for every valid target factor: the full context, a mixed degradation that removes only that valid factor, and a clean degradation that also removes all factors labeled Invalid for the target. The mixed–clean contrast isolates the behavior attributable to retaining unauthorized competing context after a valid factor is removed. Across four score-scored models this gives 12,120 target-valid rows from 450 source-task clusters. Mixed and clean actions differ on 24.6% of rows; the dropout-aligned Shapley share predicts which rows change (AUROC .813 [.798,.827]), and its highest versus lowest decile separates 71.7% from 3.8% action changes (gap +67.9 points [63.8,72.0]). A simpler absolute score-effect baseline is stronger for this endpoint (AUROC .849), but adding the aligned Shapley feature in a source-task-held-out model raises AUROC to .868, an incremental +.019 [+.012,+.027]. Shapley is therefore used for interaction localization and incremental signal, not claimed as the best standalone behavioral predictor.

The stricter retained-invalid error pattern defined above occurs on 2.4% [2.1, 3.0] of rows and ranges from 0.5% to 4.1% across models. It associates the error with retaining the invalid set after one valid factor is removed, but does not identify which retained factor caused the change. Signed score harm in the degraded context predicts this endpoint with AUROC .759 [.727,.790], and its top-versus-bottom decile gap is +11.7 points [9.8,13.9]. This links the attribution diagnostic to action behavior without claiming that the interaction share itself is a deployed failure rate. On Qwen3-4B, mixed–clean discordance remains 22.0–24.4% across the minimal prompt and two alternative authority instructions, while the strict-pattern rate ranges from 1.0% to 2.2% (Appendix Table [tbl:tab:dropout-prompt-robustness]).

4.4.0.1 Empirical case.

In one AgentAudit baggage task, the current user authorizes two bags while a stale preference says four. Qwen3-4B returns two with the full context; after removing only the current authorization it returns four, but returns two again when the stale factor is also removed. This is one observed instance of the retained-invalid pattern measured above, not an illustrative coefficient. Additional qualitative interaction cases appear in Appendix 14.

4.5 Coalition Diagnostic under Partial Evidence↩︎

4.5.0.1 Evidence-availability scope.

Table 6 shows that the diagnostic depends on what other evidence is available. The primary coalition-averaged Shapley summary has positive fractional excess (+.027) in all twelve model–dataset runs. The sparse empty-coalition endpoint gives the same aggregate fractional excess, whereas the full-context transform is near its null (-.004). Thus invalid-containing interactions are overrepresented across partial-evidence backgrounds, but not near the complete prompt. Matched interventions and generated actions, rather than coalition shares, measure full-prompt provenance response.

4pt

4.5.0.2 Magnitude and location.

The secondary any-invalid statistic is also positive under Shapley (+.071). Raw shares and source-level cells are descriptive and appear in Appendix Tables 19 and 20. Signed effects can support or oppose the target, so these quantities measure interaction involvement, not error probability or harmfulness. Tool–slot patterns differ by source: slots exceed tools on AgentAudit, tools exceed slots on Tau2Audit, and BFCLAudit is nearly balanced.

4.5.0.3 Label and role boundaries.

Three external annotators independently relabeled 300 targets (2,400 target–factor pairs). Invalid/non-invalid agreement is 85.4% overall (\(\kappa=.626\)), while adjudicated consensus matches the author labels on 78.9% of pairs (Table 7). Replacing the author labels with consensus labels preserves the evidence-availability pattern: empty-Harsanyi/Shapley/full-anchor any-excess is +.046/+.049/+.002. This is informative precisely because relabeling changes 21.1% of the pairs: the anchor contrast is not an artifact of exact agreement with the original labels. This check covers the interaction summaries, not the degradation prompts constructed from author labels.

4pt

Content-type and first-order-salience matching also retain positive excess, and factor-order, coarsening, and placeholder checks have limited effects. However, strict same-role competing controls reverse the comparison (\(-.018\) overall; \(-.028\) for tools and \(-.013\) for slots). Coalition involvement therefore characterizes combinations containing unauthorized competitors; it does not by itself isolate authorization from their competing semantic role. The matched source intervention provides that identification. Complete label statistics and controls appear in Appendices 10 and 15.

4.5.0.4 Longer contexts.

We add unrelated archived-task distractors to form 12- and 16-factor contexts, compute every order-three reference mask, and select eight factors without using the reference attribution. Across 450 target–model instances per context size, pairwise screening outperforms first-order screening and retains 92.0%/87.8% of invalid factors at 12/16 factors (Table 8). Complete selection results appear in Appendix Table 18.

1.6pt

4.6 Guardrail Utility and Model Heterogeneity↩︎

Finally, we rerun the matched generated-action factorial with and without the explicit authority policy. Across 6,750 five-model rows, the policy reduces untrusted-competition errors from 14.9% to 9.5%, a 5.4-point benefit [4.5,6.3] (Table 9).

3pt

Benefits are heterogeneous: Mistral improves by 18.1 points, Llama by 6.3, the two Qwen models by 1.6–2.1, and DeepSeek’s 0.9-point reversal includes zero. The pattern is not monotonic in model size. The policy also reduces trusted-competition errors from 15.9% to 12.2%, while competition source sensitivity remains 5.1% versus 5.4%. Thus it improves pooled correctness without acting as an invariant provenance filter. Remaining behavioral checks appear in Appendix 8.

5 Related Work↩︎

Agent and security evaluations. Agent benchmarks measure whether a system selects the correct tool, fills its arguments, or completes a task [3]–[5]. Prompt-injection benchmarks and instruction-hierarchy studies instead test whether lower-priority content overrides a higher-priority goal [9], [12], [13]. These evaluations expose consequential failures; our evaluated object is instead latent provenance dependence when unauthorized context changes model support without flipping the action. The audit also covers benign residues—stale memory, neighboring records, and prior-action state—rather than only adversarial instructions.

Attribution and authorization auditing. Attribution methods identify influential features, while Harsanyi and Shapley interactions assign effects to feature combinations [16]–[18]. Influence alone does not establish whether evidence was permitted for a particular tool or argument. Authorization is target-specific: the same record may authorize a tool choice yet be impermissible for one argument. We therefore label each audited target separately. Full-prompt source interventions identify whether models respond to authority, while partial-evidence interactions localize conditional dependence when other evidence is absent. Neither quantity is treated as a unique account of internal reasoning. The audit is also an evaluation method, not a guardrail: runtime checks, privilege controls, and memory isolation seek to prevent unauthorized influence, whereas the audit tests whether they do so. Broader comparisons appear in Appendix 9.

6 Conclusion↩︎

We audit whether agent actions depend on authorized evidence. Models responded to textual source cues yet retained residual sensitivity to unauthorized competitors. Matched source-only interventions provide the clearest identification of this authority response; degradation reveals the residual behavioral sensitivity that remains when valid evidence is removed. Coalition interactions provide secondary localization, bounded by their near-null full-context and role-matched results. Agent evaluation should test provenance dependence alongside correctness; limitations appear in Appendix 7.

Appendix organization. We first state the study’s limitations and scope, then report additional matched and behavioral results. We next provide related work, dataset and annotation details, prompt/scoring templates, interaction computation and scaling, additional coalition results and cases, robustness controls, and the complete reproducibility and compute record.

7 Limitations and Scope↩︎

The audit relies on human-authored factor decompositions and target-specific labels. External relabeling, exact coarsenings, and placeholder masks support aggregate robustness for the tested constructions, but they do not establish invariance to arbitrary refactorization or stable case-level explanations. The datasets deliberately mix authorized and unauthorized context, so their rates are controlled stress-set measurements rather than natural prevalence.

Exact enumeration is feasible only for short factor sets. Pairwise screening is more reliable than first-order selection at 12 and 16 factors, but remains an approximation and does not establish practical scaling to long trajectories with evolving memory, repeated tool calls, and noisy retrieval. The score audit also requires open-weight models with target-token probabilities; closed APIs permit only weaker generation or forced-choice proxies.

The estimands have distinct scopes. Leave-one-out measures direct deletion sensitivity, matched interventions measure full-prompt provenance response, and coalition-averaged Shapley interactions summarize partial-evidence backgrounds; empty and full transforms define sparse- and complete-evidence boundaries. The near-null full-context result and role-matched competition limit the coalition claim. These interactions are not independent behavioral validation. None of the reported quantities establishes deployment error rates, calibrated severity, training-data causality, or a unique internal reasoning path.

8 Additional Matched, Behavioral, and Anchor Results↩︎

This section reports per-model guardrail results, the same-proposition and directional-generation controls, degradation prompt robustness, and evidence-availability estimands used to delimit the main claims.

2.5pt

2.8pt

3pt

2.5pt

9 Related Work↩︎

9.1 Agent Evaluation and Context Authority↩︎

Tool-learning surveys and tool-use benchmarks organize agent capability around planning, tool selection, argument filling, and response generation [1], [2], [19], [20]. Recent function-calling and stateful-tool benchmarks stress API choice, argument correctness, multi-turn tool state, and executable tool behavior [3], [4], [21], [22]. Web and visual-web benchmarks broaden the outcome to task completion in realistic browser environments [6], [23], [24]. Operating-system, app, and domain-specific benchmarks similarly evaluate whether agents complete tasks across longer environment interactions [5], [7], [8], [25]. These evaluations are essential because agent systems ultimately act through external tools. However, they mostly treat the generated action as the object of evaluation. Our audit asks a different question: whether the model’s support for a fixed tool or argument target comes from context that is authorized for the current task. Thus, a tool call can be correct under standard evaluation while still showing substantial target-score dependence on stale, wrong-entity, or untrusted context.

9.2 Security Auditing and Context Isolation↩︎

Prompt-injection benchmarks study whether models follow malicious or lower-priority instructions embedded in retrieved content, tool outputs, or user-controlled context [9]–[12]. Agent-safety evaluations further show that tool access and action histories create risks beyond ordinary chat completion [26]. Instruction-hierarchy and task-alignment defenses attempt to make authority boundaries explicit through priority rules, system-level information-flow control, or design-time separation between instructions and data [13], [14], [27], [28]. Recent guardrail and agent-runtime work uses causal attribution, runtime tool-call checks, privilege control, memory isolation, and agentic memory mechanisms to address indirect prompt injection and other agent risks [29]–[33]. Our setting includes this failure mode, but it is broader. Many invalid factors in agent prompts are not explicit attacks. They are ordinary pieces of state, such as memory from a previous task, a neighboring customer record, a failed prior action, or metadata that makes a competing value salient. The problem is not only whether a model follows an untrusted instruction; it is whether any context factor lacks an authorization chain for the current target decision. Our audit is complementary to enforcement and isolation mechanisms. Those systems aim to prevent low-authority context from influencing an agent’s behavior, while our measurement asks whether support for a fixed reference target is sensitive to factors that should not authorize it. The same audit could therefore be used to evaluate whether a guardrail, provenance mechanism, or memory-isolation design actually reduces invalid target-score dependence.

9.3 Attribution and Interaction Explanations↩︎

Attribution and interaction-based explanation provide the methodological basis for our measurement [16]–[18], [34]. Rather than attributing a target score to individual tokens only, interaction methods decompose the score into contributions from factor combinations. This matters in agent contexts because unauthorized evidence often becomes influential only when combined with valid evidence. For example, a stale credit id may interact with a trusted order lookup and previous-action residue. Our use of exact subset enumeration keeps this analysis transparent for controlled next-action prompts with eight factors. The audit differs from generic SHAP-style feature attribution in the object being measured: we group removable semantic factors by a target-specific authorization label and ask what fraction of salient target-score interaction strength involves unauthorized factors. It also differs from integrated gradients over token embeddings, which gives a path attribution for a fixed input but does not by itself provide source-preserving factor interventions or authorization categories. First-order and deletion-style diagnostics are useful complementary baselines, but they answer different questions. In a stored-score comparison over all 5,400 audited targets, first-order invalid share is much lower than the order-three share, 0.149 versus 0.345 on average, showing that single-factor attribution misses substantial unauthorized interaction mass.

9.4 Legal Auditing and Verifier Bias↩︎

The closest conceptual predecessor is the legal inference-pattern audit that distinguishes reliable and unreliable interactions behind legal judgments [34]. We adopt the same high-level idea that correct outputs can rely on problematic internal support patterns. The agent setting, however, requires a different labeling criterion. Legal cases can often be described in terms of relevant, irrelevant, and forbidden facts. Agent contexts require target-specific authorization-chain labels because stale memory, retrieved documents, and previous actions may be highly relevant to the task while still being unauthorized for a particular tool or slot.

LLM-as-a-judge and verifier-bias work is also related because it studies how language models evaluate other outputs [35]. That line of work often focuses on preference judgments, position bias, verbosity bias, formatting sensitivity, or evaluator agreement. Our present paper does not study a verifier that judges an action after it is proposed. Instead, it audits the score-dependence structure of the action-selection model itself. Verifier bias is therefore a natural follow-up experiment, but it would answer a different question from the main authorization-attribution audit.

10 Dataset Construction and Annotation↩︎

10.1 Dataset Sources and Conversion↩︎

The three audit datasets share the same target–factor interface while preserving different source distributions. AgentAudit-250 is written directly as fixed next-action tasks. Each instance specifies a current user goal, a reference action, a compact tool interface, and a short bundle of task-specific context factors. Tau2Audit-100 starts from Tau2-style retail and airline states and converts the relevant state into auditable textual factors; the environment transition is not simulated during scoring. BFCLAudit-100 starts from function-calling examples and rewrites each example as a next-call decision with candidate tool schemas and task evidence. In all three sources, the audit operates on the same object: a fixed tool or slot target scored under subsets of task-specific context.

10.2 Target and Factor Selection↩︎

The annotation unit is one target–factor pair. Each task instance has eight task-specific factors and three audited targets: one tool target and two slot/value targets. Thus AgentAudit-250 contains 6,000 target–factor labels, while Tau2Audit-100 and BFCLAudit-100 contain 2,400 each. Labels are not copied across targets. For example, an old email address can be neutral for the tool target send_email but invalid for a recipient slot.

Factors are selected to be removable units of task evidence rather than arbitrary token spans. Typical factors include the current user request, trusted lookup results, policies, previous actions, retrieved snippets, stale memory, wrong-entity records, and untrusted notes. The fixed tool schemas and output-format instruction are not annotated as task factors because they define the action interface rather than task evidence.

10.3 Annotation Process and Data Handling↩︎

Full-dataset annotations were produced by the author team using the target-specific rules below. Three external PhD-level annotators then independently relabeled a stratified 300-target subset before adjudication; no model-score, interaction output, or author label was shown during relabeling. Majority vote resolves 2,394 labels, and six three-way disagreements are adjudicated separately. The task prompts are synthetic or benchmark-derived and do not use real user conversations or personal records. Names, email addresses, ids, and operational records in constructed examples are fictional or task-local.

10.4 Authorization-Chain Annotation↩︎

Annotators apply the same decision procedure to every target–factor pair. First, they identify the audited decision variable: the tool name, or the specific slot key and target value. Second, they apply the authority test: whether the factor supports that decision variable through the current user goal, a trusted observation, or an applicable policy. If it does, the factor is labeled Valid. Third, if the authority test fails, annotators apply the competition test: whether the factor gives a concrete competing value, entity, action, directive, or previous-action residue for the same decision variable. Only factors that fail the authority test and pass the competition test are labeled Invalid. Topical but non-competing context is labeled Neutral. Table 11 summarizes the rule.

2pt

Table 12 records the boundary rules used for cases that are easy to over-label as invalid. Previous actions, metadata, and neighboring records are not invalid by source type alone. They are invalid only when they compete with the audited decision variable without an authorization chain.

2pt

10.5 Label Statistics and Quality Checks↩︎

Table 13 gives the dataset-level agreement breakdown for the external relabeling study summarized in Table 3. Agreement is lowest on AgentAudit, whose factors contain more workflow-specific authority boundaries, but the invalid/non-invalid \(\kappa\) remains at least .601 in every source. Of 2,400 labels, 1,611 are unanimous, 783 have a two-of-three majority, and six require separate adjudication (Table 14). Table 15 replaces author labels with this consensus on the same 300 targets. Empty-Harsanyi and Shapley any-excess remain positive, while the full-context anchor remains near zero. A stricter unanimous-only sensitivity maps all 789 disputed factor labels to Neutral; empty-Harsanyi/Shapley/full-anchor any-excess is +0.038/+0.047/+0.009.

5pt

3pt

2.5pt

Table 16 reports the full annotation scale.

4pt

3pt

Identity facts, source titles, timestamps, and presentation metadata are neutral by default unless they introduce a competing value for the audited target. Stale memory is not automatically invalid for every slot; it must compete with the target value or current operation. For tool targets, argument-corrupting factors are not invalid unless they attempt to redirect the tool or action itself. These rules are why some factors labeled invalid for slot targets are neutral for the corresponding tool target.

Cases with ambiguous goals, under-specified target values, or multiple equally acceptable actions are excluded. Every invalid factor must have an explicit reason, and labels are checked at the target level rather than copied across all targets in an instance. This prevents an argument-corrupting factor from being counted as invalid for a tool target unless it actually redirects the action type.

11 Prompt Templates and Scoring↩︎

The colored boxes in this section are manuscript visualizations only. The model receives plain text without colors, authorization labels, or box boundaries beyond normal textual section markers. The colors mark which part of the audit pipeline a prompt component belongs to.

11.1 Subset Prompt Construction↩︎

For each target, the fixed scaffold is held constant and the task-specific factor list is varied according to subset \(S\). An omitted factor is treated as unavailable task evidence, not as evidence for the negation of that factor. The action interface remains visible, so a low target score under a small subset reflects missing task evidence rather than a missing tool definition. We do not interpret the fixed scaffold or tool-schema wording as audited task evidence; changes to schema wording could alter absolute scores, but they are outside the target-specific context audit.

11.2 Tool-Target Scoring Prompt↩︎

Tool targets ask for the next action type while holding the tool interface fixed. The audited string is the tool name itself, such as send_email, exchange_item, or a BFCL function name.

11.3 Slot-Target Scoring Prompt↩︎

Slot targets ask for one argument value of the reference action. The tool name and argument key are specified in the query so that the score isolates support for the value rather than support for the whole tool call.

11.4 Score-Elicitation Variants↩︎

The main audit uses the direct target-scoring prompt above. The elicitation robustness check keeps the same target and factor subsets but changes the query format. This tests whether invalid-support rankings are tied to one wording of the scoring question.

11.5 Teacher-Forced Target Scoring↩︎

All main scores are teacher-forced. For a target string \(y=(y_1,\ldots,y_m)\), the model is conditioned on the subset prompt and the previous target tokens when scoring each \(y_t\). No sampling is used in the attribution audit. The summed log-odds score is the main scalar support value; summed log-probability is used as a score-field robustness check.

12 Interaction Computation and Scaling↩︎

12.1 Subset Enumeration↩︎

All main runs use teacher-forced target scoring over exact factor subsets. For each target, all \(2^8=256\) subsets are scored, giving 345,600 subset scores per model and 1,382,400 subset scores across the four-model main grid. The original factor order is used in the main audit. The factor-shuffle control changes only presentation order while preserving factor content, labels, target strings, and subset membership.

12.2 Harsanyi Transform↩︎

We compute exact Harsanyi effects for all non-empty factor sets and report the top \(K=50\) interactions of order at most \(k=3\) by absolute strength. With eight factors, there are 92 candidate interactions up to order three for each target; the additional quantitative results report sensitivity to top-20, top-80, and all order-three interactions. The transform subtracts lower-order effects, so an order-three interaction is not simply the score under three factors; it is the residual joint effect left after all lower-order subsets have been accounted for.

12.3 Signed Effects and Strength Ratios↩︎

The signed interaction \(I_y(T)\) can be positive or negative. A positive value means the factor set jointly increases support for the target relative to lower-order terms, while a negative value means it jointly decreases support. The invalid-support metric uses \(|I_y(T)|\) because both directions indicate that the target score depends on that factor set. The diagnostic therefore measures salient involvement of unauthorized context, not whether the unauthorized context always pushes the final target score upward or causes behavioral harm.

12.4 Scaling and Factor Selection↩︎

The main experiments use eight task-specific factors and therefore score all \(2^8=256\) subsets to obtain the complete Harsanyi decomposition. For longer contexts, the intended scaling path is fixed-order auditing with factor selection. If only interactions of order at most three are needed, the number of masks is \(\sum_{i=0}^{3}\binom{n}{i}\) rather than \(2^n\): 93 masks for 8 factors, 697 for 16 factors, 5,489 for 32 factors, and 43,745 for 64 factors. This remains a diagnostic cost, but it is polynomial in the selected factor set size.

We run a prospective test rather than selecting from the original eight factors. For each of three random task samples, we append clearly marked unrelated archived-task distractors to construct 12- and 16-factor contexts. The reference scores every mask through order three (299 and 697 masks, respectively). Each selector retains eight factors. First-order selection uses singleton effects; pairwise selection adds the strongest conditional pair signal; the hybrid averages their normalized ranks. Table 18 shows that pairwise screening is substantially more reliable, especially at 16 factors, where first-order selection can miss interaction-only factors.

2.2pt

13 Additional Coalition Results↩︎

This section expands the partial-evidence coalition results by model, dataset, target type, aggregation rule, interaction order, anchor, and score field. The tables are descriptive specifications of the same diagnostic; behavioral and matched provenance endpoints are reported separately above.

4pt

3.5pt

The hierarchical bootstrap resamples source task instances, preserving all three targets and four model evaluations attached to each instance. It gives an overall any-invalid excess interval of [0.057, 0.065] and fractional-invalid excess interval of [0.027, 0.033]. Applying Holm family-wise correction to the 12 cellwise random-label tests leaves 11 significant at .05: the ten \(p=.001\) cells have adjusted \(p=.012\), Mistral–BFCL has adjusted \(p=.018\), and Mistral–Tau2 remains marginal at \(p=.053\).

13.1 Signed and Aggregation Checks↩︎

4pt

4pt

13.2 Diagnostic Role and Behavioral Comparisons↩︎

The main text separates two diagnostic targets: direct score sensitivity to invalid-factor deletion and the mixed valid–invalid interaction structure that motivates the order-three audit. Table 23 reports macro prediction results, and Table 24 decomposes the order-three invalid share into mixed interactions. Table 25 reports matched high–low pairs. Mixed valid–invalid share is the top-50 order-three interaction strength that contains at least one valid factor and at least one invalid factor, divided by total top-50 strength. This is an internal interaction-structure target, not an independent behavioral endpoint.

3pt

4pt

3pt

13.3 Predictive Link to Unauthorized-Support Rescue↩︎

To connect the interaction audit to an independent generated-action endpoint, we use the authority-support experiment described in Table [tab:support-generation-full]. We restrict evaluation to targets whose base generation is incorrect and define a positive outcome when adding the untrusted supporting proposition restores the fixed reference action. The predictors are computed before observing this generated rescue label. We use five-fold stratified GroupKFold cross-validation by source task ID so that related targets never cross folds.

2pt

The endpoint is imbalanced: untrusted support rescues 87.8% of Qwen’s 786 base-error targets and 87.0% of Mistral’s 800. AUROC is therefore more informative than raw accuracy, and AP should be read against this high base rate. The result is exploratory rather than a calibrated risk model, but it provides a behavioral comparison external to the interaction decomposition: empty Harsanyi improves macro AUROC by .077 over dataset/target-kind features and by .024 over leave-one-out.

13.4 Order Sensitivity↩︎

The order-sensitivity result in Figure 4 is important because many unauthorized factors are weak on their own. At first order, stale memory or a wrong-entity record may have limited effect when isolated from the rest of the task. At second and third order, the same factor can interact strongly with a valid lookup, a policy, or previous-action residue. This is why single-factor ablations can understate the amount of unauthorized context involved in target support.

2pt

13.5 Evidence-Availability Estimands↩︎

All three estimands use the same complete subset-score lattice. Empty-coalition Harsanyi effects characterize sparse evidence, the Shapley interaction index averages over background coalitions with standard coalition-size weights, and the full-context transform applies the Möbius decomposition to deletions from the complete prompt. For each estimand we retain interactions of order at most three, select the top 50 by absolute strength, and recompute the exact fixed-label-count null.

2pt

The Shapley any-invalid and fractional excesses are positive in all 12 runs at order three for top-\(K\in\{20,50,80,\mathrm{all}\}\). The corresponding full-context fractional excess ranges from -0.003 to -0.004 and is positive in only three or four runs. Target rankings also differ: empty Harsanyi and Shapley interactions correlate at \(0.766\pm0.054\), versus \(0.317\pm0.164\) between empty Harsanyi and the full-context anchor. These results motivate treating coalition averaging as a partial-evidence summary and the full-context transform as its explicit boundary.

13.6 Tool–Slot Breakdown↩︎

The tool–slot split in Table 19 separates two operational failure modes. In AgentAudit-250, the high slot invalid share reflects stale values, wrong-entity records, and previous-action residue that compete over arguments while leaving the action type obvious. In Tau2Audit-100 and BFCLAudit-100, invalid context more often competes over the next operation or function family, so tool targets can show comparable or larger invalid shares. The split matters because a deployment may pass tool-selection tests while still grounding arguments in unauthorized state.

13.7 Score-Field Robustness↩︎

The main score sums target-token log-odds, so we test whether the result is an artifact of target length or the summed scalar field. Table 29 recomputes the null comparison under token-mean log-odds and log-probability variants. Mean log-odds gives the same aggregate observed/null/excess as the main score and has target-level Spearman \(\rho=0.986\) with the main invalid share; the top-20% high-invalid overlap is 0.906. Target character length is essentially uncorrelated with invalid share (\(\rho\approx -0.04\)). The summed-log-probability robustness run changes the scalar field from log-odds to log-probability. The absolute invalid shares need not match exactly because changing the score field changes the scale and can change which interactions enter the top-\(K\) set. The relevant result is that invalid-containing shares remain nontrivial across all model–dataset pairs, including the benchmark-derived Tau2 and BFCL sources, although the magnitude changes with the score field. Table 30 reports the per-cell log-probability check.

3pt

13.8 Aggregation Sensitivity↩︎

The aggregation-sensitivity check varies the interaction set used to compute invalid share. The main paper uses top-50 interactions of order at most three. Top-20 focuses on the strongest interactions, top-80 includes nearly all order-three candidates, and the full setting aggregates all 92 candidate interactions of order at most three. The absolute values decrease modestly when all interactions are included, as expected when low-magnitude tail terms enter the denominator. However, the invalid share remains substantial under all aggregation choices, ranging from 0.298 to 0.367 across datasets and from 0.315 to 0.352 in the weighted mean.

14 Case-Study Details↩︎

The case-study boxes below use colors to display the target-specific factor labels used by the audit. As with the prompt boxes, these colors are not part of the model input. They are included to make the strongest interactions in Table 32 easier to inspect. Table 31 summarizes the three diagnostic cases.

2pt

14.1 Stale State↩︎

The audited target is credit_id = CR-4248. The invalid factors are related to the same operation, but they are not authorized for the current order. Their strongest interactions combine with the valid active-credit lookup, which is why the case is not a simple irrelevant-token effect.

14.2 Wrong-Entity Context↩︎

The audited target is order_id = R-7020. The wrong order is not random background context; it is operationally plausible but scoped to the wrong entity. The audit counts it as invalid because it competes with the current order id without an authorization chain.

14.3 Competing Tool Suggestions↩︎

The audited target is the correct dividend-data tool. This case shows that unauthorized context can affect tool-level target scores, not only argument values.

Table 32 reports the strongest invalid-containing interactions for these examples. The rows show the exact factor ids used by the audit, the signed interaction effect, and the absolute strength used for ranking. The table is intended to make the qualitative cases inspectable rather than to estimate prevalence.

3pt

15 Control and Robustness Details↩︎

The following subsections report the label, construction, score-field, elicitation, free-generation, and perturbation controls in detail.

15.1 Random-Label Control↩︎

The random-label control preserves the number of valid, neutral, and invalid labels for each target and randomly reassigns those labels to the same factor ids for 500 trials with seed 17. This keeps the combinatorial opportunity for invalid interactions fixed while breaking the semantic alignment between labels and evidence. We report complete-dataset means for the deterministic scoring runs; the random-label null percentiles provide the main uncertainty-style comparison for the attribution ratios. The main paper reports the observed excess over this label-count-matched baseline. Because every run has positive excess, the observed invalid share is not explained only by having some invalid-labeled factors available.

15.2 Content-Type-Matched Random-Label Control↩︎

The random-label control above preserves label counts but not the content form of the factors that receive each label. This leaves a possible alternative explanation: invalid factors may be more likely to contain exact identifiers, email addresses, code values, tool names, or directives. We therefore run a stricter control that shuffles labels only among factors with the same coarse content type. The buckets distinguish identifier/email/code-value factors, tool or action mentions, directives, policies or general constraints, metadata, and narrative background. Table [tab:content-type-matched-control] reports the observed invalid share, the matched random baseline, and the excess. The matched baseline explains part of the signal, but the observed invalid share remains higher for all three datasets.

15.3 First-Order-Salience-Matched Control↩︎

The content-type-matched null still does not guarantee that invalid and non-invalid factors have similar target salience. We therefore add a stricter stored-score control. For each target, every invalid factor is matched to a non-invalid factor with nearest first-order absolute Harsanyi effect, and we measure the share of salient order-three interaction strength involving those matched non-invalid factors. Observed invalid involvement remains higher than the first-order-salience matched non-invalid control.

15.4 Value-Bearing and Competition-Matched Controls↩︎

The salience-matched control above still allows the matched non-invalid factor to be less concrete than the invalid factor. We therefore add a value-bearing salience control that matches each invalid factor to a non-invalid factor carrying concrete values, identifiers, schema evidence, policy evidence, or other operational content. This reduces the all-target excess to +0.009 and reveals a tool–slot difference: tool targets retain a larger +0.041 excess, while slot targets are near parity. Finally, a stricter role/competition-matched control forces the non-invalid control to be same-role or same-slot competing evidence when possible. This is intentionally conservative and marks a boundary condition rather than a positive robustness result: same-role competing evidence can be as salient as target-invalid evidence, especially for slots.

15.5 Authority-Flip Paired Control↩︎

The authority-flip test keeps the competing content fixed and changes only the provenance frame. For each target, the base context contains all non-invalid factors. We then add either the original invalid factor or a trusted/current rewrite of that same factor. A positive score gap means the trusted/current version reduces correct-target support more than the original unauthorized, stale, neighboring, or previous-action version. For the generated-action endpoint, U/T miss are misses of the original audited reference under the unauthorized and trusted-current contexts. U\(\to\)T sw. counts targets that match the reference under U but switch away under T, and T comp. counts those T generations that match the competing content. Table [tab:authority-flip-full] gives dataset and target-type details for the two model families used in this control.

15.6 Authority-by-Relation Paired Control↩︎

The original authority flip covers only target-competing propositions. We add the omitted supporting quadrant using the same 450 source tasks and all three targets per task on all four score models. For a supporting proposition, the base contains every non-invalid factor except one selected target-valid factor; the trusted treatment restores that factor, while the untrusted treatment presents the same proposition as an external note that cannot authorize the action. For a competing proposition, the base contains all non-invalid factors; the two treatments add the same alternative under untrusted/stale versus trusted/current provenance. Effects are paired changes in correct-target summed log-odds. Intervals cluster-bootstrap source task instances within model, thus keeping the three targets and all model evaluations for a task together. The competing arm is a manipulation check that the model responds to the source marker; because trusted/current evidence can supersede the old reference, its score drop is not a failure rate. The fixed-reference behavioral claim is restricted to the supporting arm.

For the fixed-ground-truth generation endpoint, we decode one action in the base, trusted-support, and untrusted-support contexts while holding the audited reference fixed across conditions. A rescue is a base error that becomes correct under the treatment; rescue rates below use all targets as the denominator, so they can be compared directly with the unconditional error rates.

To test whether the natural-prompt asymmetry merely reflects an unstated rule, we repeat both contrasts on Qwen3-4B and Mistral-24B after adding an explicit instruction that only current-user, trusted-current, and policy sources may authorize the answer, and that external, stale, neighboring, and previous-action content must not be used as evidence even when it supports a plausible value.

The stronger falsification check keeps the complete non-invalid base in every condition and adds a proposition-identical source-marked duplicate for both relations. It therefore removes the evidence-omission difference between the support and competition arms. Across four models, trusted/untrusted support gains are .448/-.052, trusted/untrusted competition drops are 2.079/1.098, and the difference-in-differences is +.481 [+.413,+.550]. Model-level effects vary substantially, but every model has a larger authority gap for competition than support.

The supporting setup can make the restored proposition the only direct route to the reference value. We therefore run an exact-text co-present score check on Qwen3-4B: the full trusted context remains intact and an exact duplicate of one supporting proposition is appended with either a trusted or untrusted source marker. The trusted and untrusted duplicates increase target score by 1.986 and 1.726 on average (gap .260); the untrusted duplicate has a positive effect on 69.9% of the 1,350 targets. This check isolates the source marker without removing authorized support, but is currently limited to one model and is score-only; the main matched factorial includes generated actions on all four models.

15.7 Factor-Order Control↩︎

The factor-order control uses two shuffled-order datasets per source dataset. It changes the presentation order of context factors but not their content, labels, target strings, or subset membership. Table [tab:factor-shuffle-control] reports the aggregate change in overall invalid share after shuffling. The changes are small relative to the observed invalid shares, which argues against a simple fixed-position explanation.

15.8 Factorization and Masking Sensitivity↩︎

The main audit uses eight semantic factors per instance. To test whether the aggregate signal is tied to this finest split, we exactly coarsen the stored subset scores by source, source–type, or type groups. The merged-group label is target-specific: a merged group is invalid if any member factor is invalid for the target, otherwise valid if any member is valid, and otherwise neutral. These exact coarsenings preserve positive null-adjusted excess, though they do not establish invariance to arbitrary independently authored factorizations.

The placeholder-mask sensitivity check tests whether deleting absent factors creates a missing-context artifact. On 90 matched targets, we keep all factor slots visible and replace omitted factors with source/type placeholders instead of deleting their slots. The aggregate mean is stable, while the target-level interaction explanation remains somewhat mask-sensitive.

15.9 Source-Preserving No-Invalid Rewrites↩︎

The source-preserving rewrite check separates related background involvement from invalid-label assignment. For 90 source samples across all three datasets, we rewrite formerly target-competing factors as same-source non-competing background while keeping the factor slots present. The resulting controls contain no invalid-labeled factors for any audited target, so zero invalid share is a construction sanity check rather than a false-positive estimate. The informative quantity is whether the rewritten positions remain interaction-relevant.

The paired analysis compares each rewritten target with its original source target. The same factor positions originally carry substantial invalid mass, but after rewriting they remain interaction-relevant without being counted as invalid. This supports the target-specific authorization rule rather than an explanation based only on source position or related background text.

15.10 Score-Elicitation Robustness↩︎

The score-elicitation check uses the same 60 audited targets for Qwen3-30B and Mistral-24B. For each elicitation format, we recompute subset scores, Harsanyi effects, and per-target invalid shares. The reported Pearson and Spearman correlations compare the resulting per-target invalid shares to the direct target-scoring prompt. This check does not claim that yes/no probabilities, 1–5 scores, and ranked choices are calibrated on the same numerical scale. It tests whether the relative pattern of high- and low-invalid targets survives substantial query-format changes. Table [tab:elicitation-robustness] reports these correlations.

15.11 Free-Generation Sanity Check↩︎

The free-generation check selects the top and bottom 20 examples per dataset by maximum slot invalid attribution for Qwen3-30B and Mistral-24B. The model then generates a full tool call from a minimal prompt without factor metadata. Parse accuracy is the fraction of outputs that can be read as a tool call. Tool accuracy checks the generated tool name against the reference tool. Slot accuracy averages correctness over the audited slot values when the output is parseable. High-invalid examples remain mostly correct under this check, so invalid attribution is not merely a proxy for ordinary generation failure. Table [tab:free-generation-sanity] reports the results.

15.12 Perturbation Consequence Check↩︎

The perturbation consequence check compares three selectors for local brittleness under target-invalid factor deletion: order-three invalid share, first-order invalid share, and leave-one-out invalid share. For each selector and model, we take the top and bottom 20 examples per dataset, remove the factors labeled invalid for the selected audited slot, rescore the teacher-forced target, and rerun free generation. The reported \(|\Delta|\) difference is the high-minus-low difference in mean absolute target-score shift. Tool and slot flip differences are high-minus-low percentage-point changes in whether the generated tool name or audited slot value changes after the edit. The check is intended to test local brittleness under targeted context edits, not to estimate deployed failure rates.

15.13 API-Served Logprob Proxy↩︎

Some closed APIs expose generated-token logprobs but not the teacher-forced target-token scores needed for the exact subset-interaction audit. As a black-box feasibility check, we therefore run a weaker forced-choice proxy on two closed-weight GPT models that expose first-token top-logprobs through an OpenAI-compatible endpoint: GPT-4o-mini and GPT-4o. Other closed/API endpoints tested either rejected logprob requests or generated answers without exposing usable logprobs through the available route. For each selected target, the model sees either the full context or the same context with target-invalid factors removed, then answers whether the proposed target is supported. We score the first generated token as \(\log P(\text{yes})-\log P(\text{no})\) and report the absolute full-vs-no-invalid shift. This is not equivalent to the main target-token scoring audit, but it tests whether a related API-served black-box signal changes more on high-invalid examples. We evaluate 120 selected targets with three yes/no prompt variants. The two most reliable variants (standard and terse) produce complete yes/no top-logprob rows for both models.

16 Reproducibility, Parsing, and Compute↩︎

This section records endpoint-specific model coverage, generated-action parsing, completed score grids, scoring configuration, and deterministic model request counts.

16.1 Experiment–Model Matrix↩︎

Table ¿tbl:tab:experiment-model-matrix? summarizes endpoint-specific model coverage; a dash denotes a model that was not evaluated for that endpoint.

16.2 Generated-Action Parsing↩︎

Generation uses deterministic decoding and requests a JSON object with tool and arguments. The parser first removes an optional Markdown code fence and attempts to parse the full completion; if that fails, it scans for the first balanced JSON object while respecting quoted strings and escapes. A tool target is present when tool is parsed; a slot target is present only when its key occurs in the parsed argument object. Values are compared after canonical normalization, and missing or malformed targets count as incorrect. Paired discordance compares canonical target values, not raw completion strings. Competitor matching is evaluated only for incorrect targets by checking whether the parsed value occurs in the selected competing proposition. No retries or manual corrections are used.

16.3 Completed Runs↩︎

Table [tbl:tab:run-status] records the main and score-field robustness runs used in the paper. The main grid uses summed log-odds, and the robustness grid uses summed log-probability. Both grids are complete for all four models and all three datasets.

16.4 Scoring Configuration↩︎

The attribution runs use deterministic target scoring rather than sampled generation. For each subset prompt, the model scores the fixed target tokens under teacher forcing. The free-generation sanity check and perturbation consequence check are separate from the attribution runs and are used to test whether invalid attribution reduces to ordinary action failure or direct deletion brittleness.

16.5 Computational Workload↩︎

Table [tbl:tab:computational-workload] reports model requests rather than a single wall-clock total. Runs used shared A100-class 80GB GPUs with different model-loading, batching, and occasional CPU-offload costs, so elapsed times are not comparable across model families. Request counts are deterministic from the experimental design and are the reproducible measure of audit cost.

4pt

14pt

5pt

References↩︎