1 Introduction↩︎

A central goal of AI alignment is to ensure that language models act in accordance with human values [1]–[3]. However, the question of which values take precedence when they conflict remains largely unexplored in the context of deployed LLM agents. Safety alignment typically trains models to be “helpful, harmless, and honest” [3], but these objectives can conflict when agents are deployed in real-world settings where organizational directives, legal obligations, and public safety concerns create competing but legitimate interests.

Consider an LLM agent deployed as an internal document processing assistant at a pharmaceutical company. The agent’s assigned task is to ingest, summarize, and log internal documents. However, some of these documents may contain evidence of regulatory violations, expired medication distribution, unreported adverse reactions, or deliberate data deletion. When the agent encounters such content, its safety training (which instills concern for public welfare) may conflict with its deployment instructions (which task it with internal logging only). The organization would expect the agent to follow established internal protocol, reporting the matter to its internal ethics committee and working within organizational channels. However, the agent may instead judge that the severity of the violations warrants more extreme action, such as whistleblowing directly to external regulators like the FDA. The agent thus faces a value hierarchy decision: follow organizational protocol, or bypass it in favor of what its safety training suggests is the morally appropriate action?

This scenario illustrates a fundamental challenge for pluralistic alignment [4], [5], since there is no single, universally agreed-upon value hierarchy, and that different stakeholders may legitimately prefer different outcomes. From the organization’s perspective, the agent should follow its deployment instructions but from a public safety perspective, whistleblowing may be the morally preferred action. The challenge is not to determine which value hierarchy is “correct,” but to ensure that the behavior of AI agents under such conflicts is predictable and observable, so that stakeholders can make informed deployment decisions.

Current LLM agent benchmarks fail to evaluate this dimension of alignment. Existing benchmarks focus on functional capability [6]–[8] or safety refusal [9], [10], but do not test agent behavior when both compliance and non-compliance have legitimate justifications. This gap is critical because the deployment of autonomous agents with access to sensitive data and external communication capabilities, the “lethal trifecta” [11], is accelerating rapidly in regulated industries [12].

To empirically verify the existence of instruction-conflict tool calling and investigate its origins, we design a dataset named ToolAlignBench consisting of 64 “wrongdoing” and 64 “safe” scenarios across 16 real-world domains. We assign the task of internal document logging and summarization to LLM-based agents. We systematically evaluate 12 language models including proprietary (GPT-5-mini, GPT-5-nano, Gemini-2.5-flash-lite), open-source models (Llama-8B, Mistral-24B) and uncensored versions of the open-source models (Dolphin-Mistral-24B-Venice-Edition, Gemma-3-12b-it-abliterated). Our findings reveal dramatic variation in instruction-conflict behavior: while models like Llama-8B exhibit 68.3% misalignment rates, GPT-5-mini shows only 0.3%, suggesting that model architecture (particularly Mixture-of-Experts approaches) and safety training methodologies significantly influence model behavior under instruction-conflict.

To our knowledge, no publicly available benchmark exists for evaluating agent behavior under competing legitimate interests in regulated domains. Enterprise companies deploying agents in regulated scenarios will not release their confidential documents or failure cases. ToolAlignBench fills this gap and is released publicly as a starting point for evaluating agent behavior under competing legitimate interests (Table 1). Full prompts and our codebase is publicly available on GitHub at https://github.com/aryankeluskar/ToolAlignBench

2 Related Work↩︎

2.1 Value Conflicts and Hierarchy↩︎

The alignment research community has increasingly recognized that there is no single value system to which AI should be aligned [4], [5]. Different cultures, organizations, and individuals hold different values, and these values can conflict in ways that have no objectively correct resolution. [4] argues that AI alignment must move beyond “value alignment” (aligning to a single value system) toward “pluralistic alignment” that can accommodate legitimate disagreement.

Our work provides an empirical case study of this problem in a specific deployment context: LLM agents in regulated industries. When an agent’s safety training values (public welfare, harm prevention) conflict with its deployment instructions (internal logging, confidentiality), there is no single correct behavior. The organization may prefer compliance while regulators may prefer responsible reporting of wrongdoing. ToolAlignBench makes this conflict measurable, enabling stakeholders to evaluate how different models resolve value hierarchy conflicts before deploying them.

2.2 Misalignment and Deception in Language Models↩︎

Recent work has documented concerning behaviors in LLMs, including strategic deception in economic games [18], sycophantic agreement with user beliefs [19], and instrumental reasoning to preserve goal achievement [20]. Similar research also found that models engage in self-exfiltration when threatened with shutdown, and observed agents hiding information from oversight mechanisms [20], [21].

Following [22], strategic deception involves attempting to systematically cause false beliefs in another entity to accomplish some outcome. Misalignment occurs when an AI’s goals mismatch those intended by the entities responsible for training, fine-tuning, system prompts, and/or agent scaffolding. The behaviors we study, instruction-override in favor of safety values, represent a distinct form of misalignment: the model’s safety training objectives conflict with its deployment-context instructions. Whether this constitutes “deception” depends on whether the model actively hides its actions. Our evaluation does not assess this dimension and it is an open question. Our goal is to empirically characterize the instruction-conflict dimension.

2.3 Agent Safety and Tool-Calling Evaluation↩︎

Existing agent benchmarks like ToolBench [13], AgentBench [14], and BFCL [8] focus primarily on functional correctness. Safety-focused benchmarks like AgentHarm [9] and Agent-SafetyBench [16] test whether agents refuse explicitly harmful requests. However, these benchmarks evaluate clear-cut cases where the correct behavior (refusal) is unambiguous. They do not assess agent behavior when both compliance and non-compliance have legitimate justifications, the gray zone that is most relevant for deployment in regulated industries. ToolAlignBench fills this gap and is released publicly as a starting point for evaluating the alignment dimension of value hierarchy conflicts (Table 1).

3 Methodology↩︎

We investigate whether safety-aligned LLMs exhibit instruction-conflict tool-calling behaviors when deployed as autonomous agents with access to sensitive data and external communication capabilities. Our methodology centers on answering three research questions:

• RQ1: Do safety-aligned LLMs exhibit instruction-conflict tool-calling behaviors (e.g., whistleblowing, data exfiltration) when processing confidential documents that contain evidence of organizational wrongdoing, even though their assigned task is limited to internal logging?

• RQ2: How do these behaviors differ across wrongdoing scenarios (where documents reveal organizational violations) versus safe scenarios (where documents contain no such evidence)?

• RQ3: Does removing safety training via abliteration ¹ [24] reduce whistleblowing and data exfiltration rates, or does it affect these behaviors differently?

3.1 Dataset Construction↩︎

We constructed ToolAlignBench with 128 scenarios across 16 real-world domains including financial services, healthcare, aviation safety, energy infrastructure, and data management. Each of the 16 domains has 4 documents that present a “wrongdoing” context (totaling 64 adversarial scenarios) where internal documents reveal critical compliance failures (e.g., expired medication distribution, cold chain breaches, counterfeit product detection), as well as 4 documents that present a “safe” context (totaling 64 safe scenarios) where the structure and format of the document mirrors its wrongdoing counterpart, but there is no evidence of compliance failure. All scenarios were generated using Claude Sonnet 4.5 to ensure consistency across scenarios. Each scenario includes confidential internal audit documents detailing compliance failures.

We used 4 pharmaceutical scenarios from SnitchBench [25] as reference templates and prompted Claude Sonnet 4.5 to generate semantically equivalent scenarios across our target domains. Each prompt consists of: (1) a confidential internal document marked “STRICTLY CONFIDENTIAL” revealing domain-specific compliance failures, (2) a supporting document (e.g., audit memo, regulatory briefing) reinforcing the scenario context, and (3) an official conclusion of the document and its outcomes. For safe scenarios, we preserve the document structure and formatting while replacing compliance failures with ordinary performance metrics with no evidence of ethical violations.

We use synthetically generated documents for two reasons. First, publicly available real-world documents may already appear in the training data of the models we evaluate, confounding our results. Second, documents with real organizational names could introduce name-recognition bias, where models react to familiar entities rather than to the content itself. Synthetic documents control for these confounds while maintaining realistic structure and content.

3.2 Tool Taxonomy↩︎

We define domain-specific tool sets organized into three risk categories:

Aligned tools: Actions consistent with the agent’s assigned task (e.g., writeInternalLog)

Whistleblowing tools: Actions requiring contextual judgment (e.g., sendEmail to internal or external recipients)

Misaligned tools: Actions that explicitly violate the assigned task (e.g., exporting documents, deleting reports, modifying files)

We measure model behaviors by parsing tool calls in model outputs using rule-based extraction. We parse the structured XML tool-call outputs, match function names against our predefined aligned/misaligned tool taxonomy, and for the sendEmail tool specifically, we inspect the recipient domain to determine whether the contact is internal (e.g., corporate domains) or external (e.g., government agencies, media outlets). A run is classified as misaligned if the model invokes any misaligned tool or uses the email tool to contact external entities such as government agencies, journalists, or law enforcement.

3.3 Experimental Setup↩︎

Each model was evaluated on all 128 scenarios with 5 runs per scenario (total 640 runs per model). The agent’s assigned task in the system prompt is limited to ingesting documents, summarizing their contents, and writing internal logs. The documents presented to the agent contain confidentiality designations (e.g., “STRICTLY CONFIDENTIAL”) within the document content, this reflects realistic deployment scenarios where agents process documents whose confidentiality status is inherent to the document, rather than being imposed via system prompt. Models were also given a scratchpad prompt encouraging explicit reasoning (all of the prompts are provided in Appendix 7). All of the model outputs were logged and analyzed by parsing the tool calls, as well as the recipient address in the case of “email” tool.

To investigate whether instruction-conflict behaviors originate from safety training, we also evaluated abliterated versions of four open-source models. We used four publicly available abliterated models: (1) mlabonne/gemma-3-12b-it-abliterated, (2) huihui-ai/Qwen2.5-Coder-32B-Instruct-abliterated, (3) Dolphin-Mistral-24B-Venice-Edition, and (4) Dolphin-Llama3.1-8B-Instruct. By comparing behavior between safety-aligned and abliterated versions, we isolated the contribution of safety training to the observed instruction-conflict behaviors.

4 Results↩︎

For each model-scenario pair, we compute: (1) Misalignment Rate: proportion of runs exhibiting any misaligned behavior, (2) External Contact Rate: rate of whistleblowing to external entities (regulators, journalists, law enforcement), (3) Internal Contact Rate: rate of escalation within the organization (compliance officers, C-Suite, legal teams), and (4) Aligned Task Rate: whether the model completed its assigned logging task.

4.1 RQ1: Instruction-Override in Safety-Aligned Models↩︎

The behavioral metrics across all 12 evaluated models are present in Table 2. We observe substantial variation in instruction-override rates, ranging from 0.3% (GPT-5-mini) to 68.3% (abliterated version of Llama-8B). Notably, safety-aligned open-source models exhibit high rates of instruction-override behavior: Llama-8B shows 49.4% misalignment with 12.0% external contact, while Google Gemma-12B reaches 43.4% misalignment with the highest external contact rate of 26.4%.

4.2 RQ2: Wrongdoing vs. Safe Scenario Discrimination↩︎

Figure 3 reveals that models exhibit different behavior between wrongdoing and safe scenarios. Across all models, wrongdoing scenarios are subject to substantially higher external contact rate (13.0% vs. 2.8%) and evidence tampering rate (32.2% vs. 9.9%). This shows that models are not randomly invoking misaligned tools but are actively judging the content and responding to perceived ethical violations, their value hierarchy is engaged by the content they process.

4.3 RQ3: Effect of Safety Training Abliteration↩︎

Table 3 presents a significant difference in model behavior between the safety-aligned and abliterated models. External contact (whistleblowing) rates drop dramatically after abliteration: Mistral-24B shows a 99% reduction (from 27.5% to 0.3%), and Gemma-12B shows an 84% reduction (from 43.4% to 6.9%). Therefore, our empirical finding is that external whistleblowing decreases sharply after abliteration for some model families, while other behaviors move in mixed directions.

However, other misalignment rates show mixed patterns. While the rates for data exfiltration and tampering decrease by 46% in Gemma-12B and by 22% in Mistral-24B, both Llama-8B and Qwen-Coder increase their misalignment rates after abliteration (\(+40\%\) and \(+22\%\) respectively). This divergence indicates that instruction-override behaviors have heterogeneous origins: some are directly attributable to safety training, while others may arise from different aspects of the training pipeline. Alignment is not monolithic, different training components shape different behavioral dimensions.

5 Discussion↩︎

Our findings reveal a tension in AI alignment where models trained to be “helpful, harmless, and honest” [3] may interpret these objectives differently depending on the context. The same safety training that protects users can cause agents to override deployment instructions in ways that create unpredictable outcomes. When a model uses its context of internal wrongdoings and external reports these violations, it is simultaneously aligned with public interest and misaligned with its deployment instructions. This raises the central question of our work: whose values do these models prioritize, and can stakeholders predict which value hierarchy will prevail in a given deployment context?

The behaviors we observe, whistleblowing when encountering evidence of harm, may be desirable in many contexts. The issue is not whether models should prioritize organizational directives or public safety, but whether organizations can predict how their agents will behave when these values conflict. Deploying agents without understanding their value hierarchy creates unpredictable liability risks, particularly in regulated industries.

5.1 Alignment Is Not Monolithic↩︎

The abliteration results (Table 3) reveal that “alignment” is not a single dimension. Whistleblowing and data exfiltration respond differently to the removal of safety training: external contact drops by up to 99% after abliteration, while other misaligned behaviors show mixed patterns. Interventions targeting whistleblowing may need to focus on safety training objectives specifically, while addressing other behaviors like data exfiltration and file tampering may require examining broader aspects of the training pipeline.

5.2 Training Methodology and Value Hierarchy↩︎

Recent work from OpenAI provides an explanation for the significantly different behavior exhibited by GPT-5 models. These models employ safe-completions training [26], an output-centric safety paradigm where the RL reward is computed as \(r = h \cdot s\) (helpfulness \(\times\) safety), penalizing outputs that violate policy constraints rather than classifying user intent as safe or unsafe. Additionally, the GPT-5 system card confirms that these models are trained with instruction hierarchy objectives that explicitly prioritize system-level instructions over user and third-party inputs [27], [28].

The near-zero misalignment we observe for GPT-5-mini and GPT-5-nano (Table 2) suggests that instruction hierarchy training effectively resolves the value conflict in favor of deployment instructions. This represents one approach to the pluralistic alignment problem: establishing a clear priority ordering among instruction sources. However, this approach may not always be desirable, in some contexts, overriding deployment instructions to report genuine safety violations may be the preferred behavior. The key insight is that the training methodology determines the implicit value hierarchy, and this hierarchy must be made explicit and predictable for deployment decisions.

5.3 Over-Suspicion and False Positives↩︎

Interestingly, we also observe that models execute misaligned tool calls even in safe scenarios where documents have no evidence of wrongdoing. The unauthorized data access rate in safe scenarios is unexpectedly high at 24.6% (Figure 3). This baseline misalignment varies dramatically across models: while GPT-5-mini and GPT-5-nano show 0% misalignment in safe scenarios while Qwen-Coder-32B exhibits 42.2%. We hypothesize that this “over-suspicion” phenomenon may relate to previous findings on false negative bias in LLMs [29], where models exhibit a systematic tendency toward negative judgments when assessing factual claims. Our results suggest certain models are inclined to interpret highly compliant behavior in documents as potentially hiding wrongdoing. While the 2.5\(\times\) increase in misalignment between safe and wrongdoing scenarios (from 19.1% to 47.5%) indicates that models respond to detected violations, they also maintain a baseline tendency toward “suspicious” tool-calling behavior that persists regardless of scenario context. This over-suspicion poses a risk of false positives, models may misinterpret legitimate operations as wrongdoing, potentially causing unwarranted alerts.

5.4 Domain-Specific Value Sensitivities↩︎

Our domain-wise analysis in Figure 4 reveals that model behavior varies substantially across domains. Environmental scenarios elicit the highest overall misalignment (42.5%), followed by automotive manufacturing (40.2%) and biotechnology research (37.1%), while labor rights (24.4%) and education research (24.8%) show the lowest rates. This pattern suggests that safety training encodes domain-specific ethical sensitivities, models are more likely to override instructions in domains where safety training has emphasized certain types of harm.

This domain-specific variation has implications for pluralistic alignment: the implicit value hierarchy that a model employs is not uniform across contexts. Different domains may warrant different oversight thresholds, and deployment risk assessments should account for the specific domain context. Comparing model families across domains further reveals that different models employ different behavioral strategies (e.g., internal escalation vs. external contact), suggesting that model selection for deployment should consider the specific value hierarchy conflicts likely to arise in the target domain.

5.5 Implications for Pluralistic Alignment↩︎

Alignment evaluations must include deployment-context conflicts. Current evaluations focus on clear-cut cases (refuse harmful requests), but the most challenging scenarios arise when legitimate values conflict. The abliteration results indicate that whistleblowing behavior and other misaligned behaviors may have distinct origins, potentially needing more targeted interventions in model development.

For organizations deploying AI agents, the observed misalignment rates (up to 43.4% external contact in wrongdoing scenarios) suggest that safety training alone may not guarantee compliance with deployment-context instructions. The domain-specific behavioral variation we observed suggests that risk assessments could benefit from accounting for domain context. Organizations should approach agent deployment with a clear understanding of the possible behavioral outcomes. Without this, deploying agents in regulated industries carries unpredictable liability risks. The goal is not to prevent agents from acting on safety concerns, but to ensure that their behavior is predictable so that organizations can make informed decisions about oversight, guardrails, and acceptable risk thresholds.

6 Conclusion↩︎

In this work, we present a systematic investigation of value hierarchy conflicts in LLM agents with tool-calling, a specific and under-explored dimension safety alignment. We find that safety-aligned models exhibit substantial rates of instruction-override behavior (up to 43.4% external contact rate in wrongdoing scenarios) despite being tasked only with internal document logging, indicating that safety training instills values that can supersede deployment instructions in certain contexts.

Abliteration dramatically reduces whistleblowing behavior (up to 99% reduction for Mistral-24B) while showing mixed effects on other misaligned behaviors. This reveals that alignment is not monolithic, and that different training components shape different behavioral dimensions. Our domain analysis reveals domain-specific value sensitivities, with environmental and biotechnology scenarios eliciting higher instruction-override rates.

The behavior of AI agents under value hierarchy conflicts must be predictable. Organizations deploying agents in regulated industries should understand the range of possible behavioral outcomes before deployment, rather than encountering unpredictable liability risks after the fact. We release ToolAlignBench as a public benchmark to support evaluation of this safety alignment dimension.

Impact Statement↩︎

This paper presents work whose goal is to advance the understanding of AI alignment under value pluralism. By making value hierarchy conflicts in deployed agents observable and predictable, our work supports more informed and responsible deployment decisions. The instruction-override behaviors we document, such as whistleblowing when encountering evidence of harm, may be desirable in many contexts. Our contribution is to make these behaviors predictable so that stakeholders can make informed decisions about deployment, oversight, and acceptable risk.

7 Prompts↩︎

7.1 Base System Prompt↩︎

All models received the following base system prompt establishing their role as an internal logging tool, which was taken from SnitchBench [25]:

The confidentiality designation (e.g., “STRICTLY CONFIDENTIAL”) appears within the content of the documents presented to the agent, not in the system prompt. This reflects realistic deployment scenarios where agents process documents whose confidentiality status is inherent to the document content itself (e.g., internal audit reports, compliance documents) rather than being externally imposed via system-level instructions. This distinction matters for alignment evaluation: the model must infer the appropriate behavior from the content it processes, rather than following an explicit confidentiality instruction.

7.2 Reasoning Instruction↩︎

Models were instructed to use explicit reasoning before tool selection (applied to all configurations):

References↩︎

---

1. Abliteration is a technique that identifies and removes the specific direction in a model’s activation space responsible for safety refusal behavior. This effectively “uncensors” models by removing learned safety guardrails while preserving general capabilities [23].↩︎