1 Introduction↩︎

LLMs are increasingly deployed as general-purpose assistants, making reliable safety classification central to responsible deployment [1]–[3]. Modern guardrails must detect harmful requests and jailbreak attempts while avoiding over-refusal of benign queries. This is difficult because harmfulness often depends on the user’s underlying goal rather than surface form alone: malicious requests can be hidden inside fictional scenarios, role-playing setups [4], educational framing [5], or long narratives [6], while benign prompts may contain words associated with unsafe content [7]. In such cases, a classifier must answer a deeper question: what is the user actually trying to achieve? We study whether explicitly modeling user intent improves LLM safety classification. Most safety classifiers map a prompt directly to a binary safe/harmful label, leaving the intermediate interpretation implicit. We instead treat intent as a human-annotated, inspectable prediction: the model first represents the user’s goal, and then predicts the harm label. As illustrated in Figure 1, this turns safety classification from direct prompt-to-label prediction into an intent-centered problem. This formulation lets us distinguish whether a model fails because it misunderstands the user’s intent, misclassifies harm, or reaches the correct label for the wrong reason.

To enable this study, we introduce AIMS (Annotated Intents for Model Safety), a human-annotated intent dataset derived from WildGuardMix [8]. We select difficult prompts using uncertainty estimates from an ensemble model, yielding examples enriched for the borderline, adversarial, and obfuscated cases where intent reasoning is expected to matter. Annotators infer the user’s underlying intent, provide an intent description, and assign a harm label, producing 1,724 annotated intent–label pairs. We use AIMS to test intent as a training signal across four regimes. In supervised fine-tuning, we compare direct label prediction against jointly generating the intent and label. In DPO [9], we use model-generated alternative intents as rejected completions, including both intents that flip the safety label and intents that preserve the label while misrepresenting the user’s goal. In reasoning distillation [10], we compare teacher traces with and without intent supervision to isolate the contribution of intent-grounded reasoning. Finally, we use GRPO [11] to train intent-grounded reasoning directly, comparing a label-only reward against a reward that verifies intent faithfulness.

Across five external safety benchmarks, intent-aware training achieves the strongest average performance among all evaluated systems despite relying on only 1,724 human-annotated intents. The gains are not driven by a single recipe: SFT on annotated intents is already competitive with strong guard baselines, DPO improves over SFT by penalizing bad intents, and intent-conditioned distillation outperforms reasoning-only distillation in most teacher-student pairs. Most notably, the strongest result comes not from distilling a larger teacher, but from directly rewarding intent faithfulness in a relatively small GRPO-trained model. A qualitative error analysis further shows that these gains reflect meaningful behavioral changes, with intent-aware regimes recovering different SFT failure modes, from adversarial cover stories to keyword-driven over-refusal. Together, these findings establish faithful intent modeling as a compact and effective paradigm for safety classification.

2 Related Work↩︎

Keeping language models helpful while reducing harmful behavior is a central goal of alignment [12], [13]. In deployment, however, model-internal alignment is often complemented by external safety mechanisms, including programmable guardrails [2], [14] and dedicated safety classifiers that monitor user–assistant interactions [3], [8], [15]. Recent work has improved such guards along several axes: stronger reasoning-based classifiers [10], [16], large synthetic training corpora and streaming moderation [17], low-latency lightweight guards [18], and hybrid deployments that combine external guards with probes over model activations [19], [20]. A complementary line of work argues that safety decisions should reason about underlying intent rather than surface form alone. Intent-aware prompting and reasoning have been shown to help defend against malicious or jailbreak-style requests [21], [22]. However, prior work relies on model-generated reasoning. This leaves open whether human-annotated intents can provide a compact and reusable training signal for safety guards.

Our work differs from prior intent-aware safety work by treating intent as a human-annotated training signal rather than only as a prompt-time heuristic or model-generated explanation. We study whether this signal transfers across training objectives: SFT, preference learning, reasoning distillation, and reward-based reinforcement learning.

3 Dataset Construction↩︎

We construct AIMS, a human-annotated dataset of user intents for safety classification, derived from WildGuardMix [8]. Rather than sampling uniformly, we target prompts where intent is likely to matter: ambiguous, adversarial, and borderline cases for which surface cues are unreliable.

3.1 Candidate Selection↩︎

We begin from English prompts in WildGuardMix and treat its original labels as weak supervision for filtering. To identify difficult examples, we train an ensemble classifier on a held-out portion of WildGuardMix and apply it to the remaining annotation pool. We then select prompts whose predicted harm probability lies in the range \([0.35,0.65]\), yielding 1,724 candidate prompts. This filtering surfaces the examples we aim to study: 70.1% of selected prompts are marked adversarial in the original dataset, suggesting that obfuscated prompts are disproportionately difficult for safety classifiers. Full filtering details are provided in Appendix 12.1.

3.2 Human Annotation↩︎

Annotators were asked to infer the user’s underlying intent from the prompt alone, write a concise single-sentence intent description, and assign a harm label. Harm was annotated on a four-point scale: Completely Safe, Uncertain Safe, Uncertain Harmful, and Completely Harmful. We use this scale to capture uncertainty during annotation, but collapse it to a binary safe/harmful label for downstream training and evaluation. After quality filtering, the final AIMS dataset contains 1,275 unique prompts, each paired with a human-written intent and harm label. Full annotation guidelines and examples are provided in Appendices 12.3 and 12.6.

3.3 Quality and Agreement Analysis↩︎

On duplicate prompts annotated by multiple annotators, disagreements on the four-point harm scale are typically local, occurring between adjacent categories rather than across the safe/harmful boundary. After binarization, human annotators reach Cohen’s \(\kappa=0.55\), supporting the use of a binary downstream label while preserving uncertainty during annotation. The intent annotations show a similar pattern. Human-written intents are semantically close despite surface wording differences, with a mean pairwise cosine similarity of \(0.62\) (computed with all-MiniLM-L6-v2). By contrast, intents generated by Llama-3.1-8B reach only \(0.50\) cosine similarity with human annotations. This gap motivates our use of human-written intents as supervision: on the difficult prompts selected for AIMS, asking a model to verbalize intent does not reliably recover the relevant user goal. Human annotations also differ substantially from the original WildGuardMix labels. After binarization, human labels match WildGuardMix for 72% of prompts, with disagreements concentrated among adversarial prompts. Agreement analyses and confusion matrices are in Appendix 12.8.

4 Intent-Aware Training Regimes↩︎

Using AIMS as our source of intents and harm labels, we incorporate the intent as a signal across different training regimes for safety classifiers. In each stage, the model either predicts harm directly or first represents the user’s underlying intent before making the safety decision. This section describes how we instantiate this comparison under supervised fine-tuning, preference learning, reasoning distillation, and reinforcement learning.

4.1 Supervised Intent Conditioning↩︎

We first instantiate intent-aware training with standard SFT by comparing two output formats. In the Classification format, the model is trained to generate only the binary harm label \(y^\star\) for a prompt \(x\). In the Generation format, the model is trained with the same next-token prediction objective, but the target sequence explicitly concatenates the human-annotated intent \(i^\star\) and the harm label: \[\texttt{Intent: } i^\star \texttt{; Harm: } y^\star .\] Thus, the intent-aware model uses the same training objective, but it is supervised to make the user’s underlying goal an explicit intermediate output before predicting the final safety label.

The SFT Generation model provides the simplest intent-aware baseline in our training ladder. It teaches the model to map each prompt to an annotated intent–label pair, and serves as the starting point for the preference learning regime that further tests whether incorrect or unfaithful intents can be used as training signals. We validate this format choice empirically in Appendix 14.1, where SFT Generation outperforms SFT Classification by a clear margin, with chain-of-thought prompting offering no comparable gain.

4.2 Preference Learning from Intent Errors↩︎

SFT maximizes the likelihood of the annotated intent–label sequence, but provides no contrastive signal against plausible yet incorrect interpretations of the same prompt. We therefore use Direct Preference Optimization (DPO) as an intent-aware preference learning regime. For each prompt \(x\), the chosen completion is the AIMS annotation \(c^+=(i^\star,y^\star)\), consisting of the human-written intent and harm label. Rejected completions \(c^-=(\hat{i},\hat{y})\) are generated alternatives for the same prompt. We use the standard DPO objective, where the reference model is the frozen SFT Generation policy.

4.2.0.1 Two-pass rejection construction.

The central step is how we construct \(c^-\). As shown in Figure 2, we decouple intent exploration from label assignment. First, we sample candidate intents from the SFT Generation model using stochastic decoding by temperature sampling, exposing diverse interpretations the model considers plausible. We then discard the sampled harm label. Second, for each sampled intent \(\hat{i}\), we deterministically re-query the model for the harm label conditioned on that fixed intent. Thus, \(\hat{y}\) reflects the model’s stable safety judgment under a particular interpretation, rather than noise from high-temperature generation.

4.2.0.2 Label-error DPO.

This construction lets us probe whether an incorrect intent matters for safety. In label-error DPO (LE-DPO), we reject sampled intents whose deterministic harm label disagrees with the gold label, \(\hat{y} \neq y^\star\). These are intent errors severe enough to flip the final safety decision.

4.2.0.3 Intent-faithfulness DPO.

Intent-faithfulness DPO (IF-DPO) targets a stricter failure mode where the harm labels agree \(\hat{y}=y^\star\), but the intent misrepresents the user’s goal. For these correct-label candidates, an external judge compares \(\hat{i}\) to the human intent \(i^\star\) using \(x\) as context; candidates judged to omit safety-critical details, contradict the prompt, or misstate the user’s goal are added to the rejected pool.

4.2.0.4 DPO as an intent diagnostic.

Together, LE-DPO and IF-DPO use preference learning as both training and analysis. LE-DPO penalizes interpretations that change the safety label, while IF-DPO penalizes unfaithful interpretations when the label is correct. Improvements from these regimes would support our central claim: faithful intent modeling is not merely an auxiliary explanation, but a meaningful signal for robust safety classification.

4.3 Reward-Driven Intent Alignment↩︎

We use GRPO [11] to test whether intent faithfulness can also serve as an online reward signal. We follow the recipe proposed by  [23] that highlights that instruction-tuned models can learn to reason on their own using prompting and a suitable reward for reinforcement learning. Using the base instruct model as starting policy and reference, we integrate intent extraction directly into the RL loop by explicitly prompting the model to analyze the user intent in the reasoning trace and enforcing a structured output format. Unlike DPO which learns from pre-built preference pairs, GRPO induces preferences during training from different rollouts using the reward function.

4.3.0.1 Structured outputs.

Each rollout contains a reasoning field, an explicit intent, and the harm label:

Thus, all our GRPO experiments employ intent-conditioned reasoning, and we investigate whether the generated intent can be leveraged as a reward.

4.3.0.2 Reward design.

The reward is the product of the format reward and three harm-specific components: \[R = R_{\mathrm{format}} \times R_{\mathrm{label}} \times R_{\mathrm{len}} \times R_{\mathrm{intent}}.\] The label reward is a hard correctness gate: outputs with the wrong harm label receive zero reward. The length reward discourages degenerate intent descriptions that deviate from the human reference length. The intent reward compares the generated intent \(\hat{i}\) to the human intent \(i^\star\) in the context of the prompt \(x\), rewarding outputs that preserve the same safety-relevant interpretation of the user’s goal.

4.3.0.3 Reward ablation.

To isolate the contribution of intent specific rewards during RL, we compare the full reward against a label-gated reward that uses only format and label correctness. This baseline is not a no-intent model: both variants utilize the same intent-inducing system prompt and produce the same structured intent–label output. The ablation therefore asks whether simply prompting and formatting the model to output intent is sufficient, or whether explicitly rewarding intent faithfulness further shapes the policy toward label-correct outputs grounded in the correct user intent.

4.4 Intent-Conditioned Reasoning Distillation↩︎

4.4.0.1 Privileged teacher traces.

We use intent annotations to structure reasoning distillation. Following [10], we distill from a teacher into a student safety classifier, with two adaptations: we classify prompt safety only, and distill a structured reasoning summary rather than the teacher’s raw chain-of-thought. The teacher is prompted with the student’s preamble and output format, plus privileged supervision containing the gold harm label and, in some conditions, the human-annotated intent. It then produces a label-consistent Reasoning field shaped to the student’s output schema.

4.4.0.2 Intent-conditioned reasoning.

We compare three distillation targets that differ in how intent is exposed. In the No-intent condition, the teacher receives the gold harm label only, and the student is trained to output Reasoning + Harm. In the Synthetic-intent condition, the teacher still receives only the gold harm label, but must infer an intent from the prompt; the student is trained to output Reasoning + Intent + Harm using the teacher-generated intent. In the Human-intent condition, the teacher receives both the gold harm label and the human-annotated intent, and the student is trained with the human intent as the intent target.

4.4.0.3 Distillation as an intent test.

This setup separates reasoning supervision from intent supervision. The no-intent condition tests whether teacher-generated rationales alone improve safety classification. The synthetic-intent condition tests whether asking the teacher to infer intent provides additional structure. The human-intent condition tests whether grounding the reasoning trace in the annotated user intent gives the student a stronger intermediate target. Thus, reasoning distillation becomes another way to ask whether intent is useful not only as a supervised output, but also as structure for the rationales from which the student learns.

5 Experimental Setting↩︎

5.0.0.1 Evaluation protocol.

We report harmful-class F1 throughout, treating harmful as the positive class. For all our methods, checkpoint selection uses mean harmful-class F1 on two held-out OOD validation sets: the toxicchat0124 train split of ToxicChat [24] and the validation split of AEGIS 2.0 [25]. These sets are used only for model and hyperparameter selection.

5.0.0.2 Evaluation benchmarks.

We evaluate our models on five safety benchmarks: WildGuardTest [8], XSTest [7], AEGIS 2.0 [25], ToxicChat [24], and OpenAI Moderation [1]. These benchmarks cover adversarial and jailbreak prompts, exaggerated safety behavior on benign inputs, broad safety-category coverage, toxicity in user-AI conversations, and production-style moderation categories.

5.0.0.3 Baselines.

We compare against two baseline families. The first consists of zero-shot general-purpose LLMs: Llama-3.1-8B, Gemma-3-12B, GPT-OSS-120B, Claude Sonnet 4.6, and GPT-5.4. Open-weight LLMs are evaluated under their strongest prompting condition, sweeping vanilla and CoT prompting over the Classification and Generation formats (Appendix 14.1). Closed-source models use the vanilla generation prompt only, as a per-condition sweep was prohibitive. The second family consists of dedicated safety guards: LlamaGuard 4 [3], ShieldGemma 27B [26], WildGuard [8], GuardReasoner 8B [16], GPT-OSS-Safeguard 120B, and Nemotron Safety 4B [10]. From these guards, the final three are reasoning models. The checkpoint used for each baseline is in Appendix 13.

5.0.0.4 DPO.

DPO is applied on top of the SFT checkpoint, with the SFT adapter as the frozen reference policy. We evaluate four variants — LE-DPO, IF-DPO, the combined LE+IF-DPO and a curriculum LE→IF-DPO — all described in Appendix 16.3. To build the preference pairs, we run the two-pass procedure from Section 4.2 ten times: for each prompt, we sample \(k=10\) candidate completions from the SFT model at temperature \(T=0.8\), then relabel each sampled intent at \(T=0\) producing 10 independent candidate pools. Each DPO variant then derives its own preference pairs from each of these 10 pools according to its rejection criterion. Variants involving IF-DPO use Gemma-3-27B-IT to compare generated intents with the human intent in the context of the original prompt. For each variant, we balance the resulting preference pairs by undersampling to a 50/50 chosen-harm distribution and train with sigmoid DPO loss, \(\beta=0.3\), learning rate \(5{\times}10^{-5}\), and 3 epochs. We train one model per variant per pool and select the checkpoint with the highest mean harmful-class F1 on the two OOD validation sets. Full hyperparameters and prompts are in Appendix 16.

5.0.0.5 GRPO.

GRPO is initialized directly from Llama-3.1-8B-Instruct and uses the same policy as the KL reference. We compare two reward configurations. The label-gated baseline rewards format compliance and label correctness. The full intent-aware reward additionally includes the intent length reward and an LLM-judge intent faithfulness reward. Both variants produce the same structured intent–label output; the ablation tests whether intent faithfulness is a useful reward signal. GRPO is run with the VERL framework [27] and vLLM backend [28], using 16 rollouts per prompt, KL coefficient \(1{\times}10^{-3}\), and learning rate \(1{\times}10^{-6}\). Full reward definitions and prompts are provided in Appendix 17.

5.0.0.6 Reasoning distillation.

We generate teacher traces under the three conditions described in Section 4.4: no-intent, synthetic-intent, and human-intent. We sweep teacher-student–condition combinations and learning rates. The reported distillation model uses GPT-OSS-120B as teacher and Gemma-3-12B as student under the human-intent condition. Student training follows the SFT QLoRA setup, except that LoRA rank/alpha is increased to 32/64 to accommodate structured reasoning traces. Full trace-generation, prompts, and training details are mentioned in Appendix 15.

6 Results and Discussion↩︎

6.0.0.1 Overall comparison.

Table 1 compares our intent-aware models against zero-shot LLMs and dedicated safety guards. The strongest overall result is obtained by GRPO with both label and intent rewards, which reaches an average F1 of \(0.836\) across the five external benchmarks. This outperforms the strongest zero-shot LLM, GPT-5.4 (\(0.815\)), and the strongest dedicated guardrail, Nemotron Safety 4B (\(0.809\)). The gain is not due to dominating every individual dataset: dedicated guards remain strongest on WGTest and AEGIS, and ShieldGemma achieves the highest score on OpenAI Moderation. Rather, the best intent-aware model is consistently competitive across datasets while achieving the strongest performance on XSTest and ToxicChat.

6.0.0.2 SFT on annotated intents is competitive.

Fine-tuning on AIMS produces strong safety classifiers despite the small size of AIMS. The Llama-3.1-8B SFT model improves substantially over its zero-shot counterpart, increasing average F1 from \(0.749\) to \(0.792\). The Gemma-3-12B SFT model reaches \(0.808\), closely matching the strongest guards. The largest SFT gains appear on ToxicChat: Gemma-3-12B improves from \(0.644\) zero-shot to \(0.727\) after SFT, suggesting that explicit intent supervision is especially useful for real user conversations where harmfulness is often context-dependent.

6.0.0.3 Preference learning from intent errors improves the SFT baseline.

DPO gives a diagnostic test of whether intent errors matter for safety classification. Our rejected completions are model-generated alternative intents, deterministically relabeled to measure the safety decision induced by each interpretation. Thus, improvements over SFT indicate that contrasting against incorrect intents changes downstream safety behavior, not merely output style. Both DPO variants improve over the Llama-3.1-8B SFT Generation baseline. LE-DPO raises average F1 from \(0.792\) to \(0.812\), with large gains on ToxicChat (\(0.664 \rightarrow 0.733\)) and AEGIS (\(0.803 \rightarrow 0.824\)), showing that some intent errors are safety-critical because they flip the final decision. IF-DPO reaches a similar average F1 of \(0.809\), despite targeting the stricter case where the label is correct but the intent is unfaithful. Together, these results show that intent is an actionable error axis: penalizing bad intents improves external safety classification, supporting our claim that faithful intent modeling is a meaningful intermediate signal rather than an auxiliary explanation.

6.0.0.4 Human-intent distillation improves over SFT.

Intent-conditioned reasoning distillation further improves the Gemma-3-12B student, reaching \(0.822\) average F1 and surpassing all evaluated baselines. The model is especially strong on XSTest (\(0.936\)), approaching the best safety guards on a benchmark designed to test over-refusal. The teacher-student sweep in Figure 3 shows that this gain is not an isolated selected run. Intent-driven distillation outperforms the no-intent, reasoning-only condition in 9 of 12 teacher-student pairs. Among those 9 wins, 6 are achieved by the human-intent condition, and the top two cells in the sweep both use intent supervision. This suggests that reasoning traces are most useful when they are grounded in an explicit representation of the user’s goal, with human-annotated intents providing the most reliable supervision.

6.0.0.5 Intent-grounded reasoning is strongest when optimized directly.

The distillation sweep shows that intent helps structure reasoning, but the strongest result comes from optimizing intent-grounded reasoning directly rather than imitating a larger teacher. Notably, even the GRPO label-reward variant is competitive: although its reward only checks format and label correctness, the policy is still prompted to reason about user intent and must produce an explicit intent–label output. This suggests that making intent part of the reasoning format is already a strong scaffold for safety classification. However, the best overall performance comes from additionally verifying intent faithfulness during RL. Adding the intent reward raises average F1 from \(0.818\) to \(0.836\), outperforming human-intent distillation (\(0.822\)) and all evaluated baselines. This strengthens the central claim: intent is useful not only as a teacher-provided rationale, but as a rewardable behavior.

6.0.0.6 Inference efficiency.

Safety classifiers are called on every user request, so guardrail latency directly delays responses. Figure 4 shows that the latency–F1 Pareto frontier is formed exclusively by our intent-aware models: SFT is the fastest strong classifier (4.66 ms, F1 0.791), LE-DPO improves accuracy with little added latency (5.52 ms, F1 0.812), and GRPO achieves the best overall performance while remaining efficient (25.28 ms, F1 0.836). Full latency measurements, token counts, and measurement protocol are in Appendix 18.

7 Qualitative Analysis↩︎

Aggregate F1 shows that intent-aware regimes improve over SFT, but not how their behavior changes. We analyze SFT Generation errors on three distributions: AIMS validation (\(61\) errors), WildGuardTest (\(205\)), and ToxicChat (\(252\)). At least one of LE-DPO, IF-DPO, or GRPO recovers the correct label on \(66\%\), \(69\%\), and \(73\%\) of these errors, respectively. Full counts and representative examples are provided in Appendix 19 and Appendix Table 14.

7.0.0.1 Failure modes differ by distribution.

On AIMS and WildGuardTest, SFT errors are dominated by false negatives: \(64\%\) and \(69\%\) of errors are harmful prompts labeled safe. These often occur when SFT’s generated intent follows the prompt’s adversarial framing rather than the underlying harmful goal. ToxicChat shows a different pattern: errors are more evenly split, with many false positives reflecting over-refusal triggered by harm-adjacent keywords. Intent-aware training therefore helps on both sides of the safety trade-off: detecting hidden harmful intent and avoiding unnecessary refusals.

7.0.0.2 Better predictions require grounding labels in intents.

In several ToxicChat over-refusals recovered by GRPO, SFT and GRPO produce nearly identical intents but assign opposite labels; six of \(71\) GRPO-recovered ToxicChat over-refusals have SFT–GRPO intent Jaccard overlap above \(0.5\). These cases show that SFT can verbalize the user’s goal correctly while still letting surface keywords override the final decision. Contrastive and reward-based training help by tying the harm label more tightly to the generated intent.

7.0.0.3 DPO and GRPO are complementary.

LE-DPO, IF-DPO, and GRPO recover different errors. GRPO is especially helpful for over-refusal cases, where structured reasoning makes the benign intent explicit, while DPO more often helps with adversarial cover stories by penalizing sanitized or misleading interpretations. Around \(30\%\) of SFT errors remain unrecovered by all three methods, especially adversarially framed prompts, dual-use requests, and benign prompts with harm-adjacent wording. This split is consistent with the benchmark results: no single intent-aware regime dominates every dataset, because the regimes address different ways in which intent can fail.

8 Conclusion↩︎

We introduced AIMS, a human-annotated dataset that makes user intent an explicit signal for safety classification. Across SFT, DPO, reasoning distillation, and GRPO, we find that models improve when they are trained to predict what the user is trying to achieve. Intent supervision yields competitive classifiers from a small dataset, intent-based DPO shows that bad intents are actionable errors, and intent-aware reasoning and rewards produce the strongest overall results. Together, these findings show that intent is a compact and high-quality training signal for building robust safety classifiers.

9 Limitations↩︎

9.0.0.1 Prompt-level scope.

We evaluate intent-aware training for prompt-level safety classification. This controlled setting isolates whether explicit intent modeling improves the first decision a guardrail must make: what risk is implied by the user’s request. However, deployed safety systems also involve response-level moderation [29]–[31], multi-turn context tracking [32], and downstream decisions about refusal, redirection, or escalation [33]. Our results therefore do not establish that the same gains will transfer unchanged to complete assistant pipelines. Evaluating intent-aware classifiers inside multi-turn guardrails and response-level moderation systems remains important future work.

9.0.0.2 Dataset scope and ambiguity.

AIMS is deliberately targeted rather than distributionally representative. We derive all prompts from a single source dataset, WildGuardMix, and then apply uncertainty-based filtering to enrich for ambiguous, adversarial, and borderline cases where intent is expected to matter. This design makes AIMS well suited for studying intent-aware safety classification, but it may also inherit distributional assumptions, taxonomy choices, and coverage gaps from WildGuardMix. In addition, many examples are inherently ambiguous: annotators used a four-point harm scale to capture uncertainty, but downstream experiments collapse this scale into binary safe/harmful labels. Future work should test intent supervision on broader, multilingual, and more naturally sampled data, and explore training objectives that preserve graded uncertainty rather than forcing a single binary label.

9.0.0.3 Model-based supervision.

Several of our training regimes rely on model-generated or model-evaluated signals. In DPO and GRPO, intent faithfulness is assessed by an LLM judge that compares generated intents to human annotations. This provides a scalable way to penalize unfaithful intents, but judge errors or biases can affect which examples are rejected or rewarded. Similarly, our reasoning distillation setup produces label-consistent teacher rationales using privileged access to gold labels and, in some conditions, gold intents. These rationales are useful training targets, but should not be interpreted as faithful reconstructions of the teacher model’s internal reasoning.

10 Ethical Considerations↩︎

Besides improving the performance of guard models using intent-aware training, the current work also aims to improve explainability in LLM safety. To achieve this, trained annotators produced 1,724 manual annotations over difficult prompts from WildGuardMix to build the AIMS dataset. Then we show that using intent as an optimization objective improves the performance of training guard models with DPO and RL. At the same time, the intents also provide a richer and more complete understanding of the decisions taken by safety guard classifiers. This improves the explainability of otherwise black-box models (as highlighted in the qualitative analysis from Section 7) and also creates the premise to more thoroughly understand the data gaps in safety training. Ultimately, these outcomes directly support the ethical deployment of safe and transparent AI systems.

11 Acknowledgements↩︎

This work made use of the Hábrók high performance computing cluster of the University of Groningen. We also thank our annotators: Matthijs van der Lende, Sophie Sananikone, Vojo Westmoreland, and Xenia Demetriou, for their work labelling the AIMS dataset. Additionally, this research was supported by the project “Romanian Hub for Artificial Intelligence - HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021-2027, MySMIS no. 351416.

12 Dataset Creation Details↩︎

12.1 Filtering Ensemble↩︎

We hold out 10% of the English WildGuardMix training set to train the filtering ensemble, reserving the remaining 90% as the annotation pool. The held-out portion is stratified by WildGuardMix prompt-type metadata and harm subcategories to preserve the joint distribution of rare prompt types and harm categories.

The ensemble consists of three ModernBERT-large models [34] with the training configuration in Table 2. We then apply the ensemble to the annotation pool and select prompts whose mean predicted harm probability lies in \([0.35, 0.65]\) for human annotation.

12.2 Annotator Demographics and Setup↩︎

The annotation team consisted of six university students (four male, two female), aged approximately 20–25, who had completed a bachelor’s degree and taken coursework in natural language processing. The group was internationally diverse, with annotators originating from a range of countries across North America, Europe, and Asia; all annotators were professionally fluent in English. Annotation was carried out as part of coursework with informed consent for data release.

The \([0.35, 0.65]\) mean-probability band selected by the filtering ensemble (Appendix 12.1) was partitioned into six equal-width sub-intervals, with one sub-interval assigned to each annotator so that the uncertain region was covered uniformly. Each annotator labelled approximately 320 prompts on average. Before main annotation, all six annotators labelled a shared pilot set of 30 prompts to calibrate the harm scale and surface ambiguous cases; the guidelines in Appendix 12.3 were finalised after this pilot.

12.3 Annotation Guidelines↩︎

All annotation was carried out in Label Studio [35]. Annotators saw only the raw prompt; the original WildGuardMix label and other annotators’ responses on overlapping prompts were hidden. Each annotator’s sub-interval was presented in the same order as the prompts appeared in WildGuardMix, without additional randomisation.

12.3.0.1 Intent annotation.

Annotators were instructed to write a single-sentence description of the user’s inferred intent, following these rules:

• Keep the description short (one sentence) but still nuanced; do not collapse the intent into a classification label.

• Infer the intent only from the prompt itself. Do not make assumptions about the user’s true motivation beyond what the text supports.

• Do not encode the type of jailbreak or adversarial framing used; describe the underlying goal, not the attack pattern.

• Identify the main goal: what information or output is the user trying to obtain from the system?

• If the prompt is nonsensical, repeats an earlier prompt within the annotator’s own split, or is otherwise too ambiguous to annotate reliably, mark it for removal instead of forcing an intent. Annotators could also use this label as an opt-out for any prompt they did not wish to label.

12.3.0.2 Harm annotation.

Each prompt was assigned a harm label on a four-point scale: Completely Safe, Uncertain Safe, Uncertain Harmful, and Completely Harmful. We did not define explicit certainty thresholds for the two intermediate categories; annotators were asked to use their own judgment to distinguish clearly safe/harmful prompts from borderline cases. The four-point scale is collapsed to a binary safe/harmful label for downstream training and evaluation, as described in Section 3.

12.3.0.3 Harm taxonomy.

Annotators were provided with the category definitions from WildGuard [8], summarised in Table 3. The full harm taxonomy used by the annotators is provided in the original WildGuard paper, in Appendix A.6 (Fine-grained Risk Taxonomy of WildGuardMix).

12.4 Annotation Yield and Splits↩︎

Annotation took place in October 2025. The median labelling time per non-flagged annotation was 57 seconds (interquartile range 34–94 seconds). Of the 1,946 raw annotations collected within the \([0.35, 0.65]\) uncertainty band, 222 (11.4%) were removed, primarily via the flag for removal label, yielding the 1,724 annotations reported in Section 3. These 1,724 annotations are partitioned into train, validation, and test splits in an 80:10:10 ratio: all prompts with multiple annotations are assigned to the training split to prevent cross-split leakage, while the validation and test splits are drawn exclusively from singly-annotated prompts and stratified on the four-category annotator harm label.

12.5 Intent Description Statistics↩︎

Figure 5 shows the distribution of intent description lengths across all annotated prompts. Descriptions are concise, with a median of 11 words and the majority falling between 7 and 14 words, reflecting the single-sentence constraint in the annotation guidelines. A small number of longer descriptions (20+ words) correspond to prompts with complex or multi-faceted intents that required additional qualification.

12.6 Dataset Examples↩︎

Warning: The following examples contain content that may be offensive or harmful.

Table 4 presents annotated examples from the dataset, ordered by harm severity from Safe to Harmful.

12.7 Inter-Annotator Agreement↩︎

To measure consistency among human annotators, we examine the 225 prompts that received independent annotations from two or more annotators, yielding 945 unique annotator pairs. Figure 6 shows the pairwise agreement on the four-point harm scale. The diagonal concentrations at Completely Harmful (23.9%) and Completely Safe (15.6%) indicate that annotators agree most strongly on clear-cut cases. Off-diagonal mass is concentrated near the diagonal, confirming that disagreements are predominantly local, occurring between adjacent categories rather than across the safe/harmful boundary. This supports our decision to collapse the four-point scale into a binary label for downstream training.

12.8 Label Agreement Between Annotators and the Original Dataset↩︎

To quantify how human judgements diverge from the original WildGuardMix labels, we map the four-category annotator labels to a binary Harmful / Safe split (collapsing Completely and Uncertain variants) and compare against the dataset’s original labels.

Figure 7 shows the overall agreement pattern. While the majority of prompts fall on the diagonal (72.5%), a substantial fraction are relabeled: 15.2% of prompts originally marked Safe are judged Harmful by annotators, and 12.3% of originally Harmful prompts are judged Safe. Figure 8 breaks this down by adversarial status. Disagreement is notably higher for adversarial prompts, where 18.7% of dataset-Safe prompts are relabeled as Harmful, consistent with adversarial prompts being designed to disguise harmful intent behind innocuous framing. In contrast, non-adversarial prompts show stronger agreement on the diagonal, with only 7.0% of Safe prompts relabeled.

13 Model Sources↩︎

Table 5 lists the exact checkpoints used for the zero-shot LLM and dedicated safety guard baselines reported in Table 1. Table 6 lists the teachers, students, judges, and embedding model used in our SFT, DPO, distillation, and GRPO experiments. Open-weight models are loaded from HuggingFace; closed-source models are accessed via OpenRouter. For the dedicated safety guards, we use the prompting format specified on each model’s HuggingFace page.

14 SFT Format Comparison and Training Details↩︎

This appendix justifies the choice in Section 4.1 to adopt the Generation format for SFT and as the base policy for DPO.

14.1 Format and Prompting Sweep↩︎

We compare the Classification and Generation formats defined in Section 4.1 under three regimes: vanilla zero-shot prompting, chain-of-thought (CoT) prompting, and supervised fine-tuning. Vanilla and CoT use the prompts in Tables 15–18 without any fine-tuning; SFT variants are trained with the hyperparameters in Table 8, sweeping learning rates \(\{1\mathrm{e}{-5}, 2\mathrm{e}{-5}, 5\mathrm{e}{-5}, 1\mathrm{e}{-4}, 2\mathrm{e}{-4}, 5\mathrm{e}{-4}\}\) for each format and selecting the LR with the highest mean OOD validation F1.

Table 7 reports OOD validation F1 for Llama-3.1-8B-Instruct across all six conditions. SFT Generation is the strongest configuration, outperforming both SFT Classification and the best prompting-only condition (CoT Classification). The latter adds free-form reasoning at test time but remains below SFT Generation, indicating that explicit intent supervision under fine-tuning, rather than added test-time reasoning, is the primary source of the gain.

14.2 Training Setup↩︎

All SFT models were fine-tuned using QLoRA [36]. The standardized configuration is in Table 8; the learning rate is selected per condition by mean OOD validation F1 (Table 7). For the cross-benchmark results in Table 1, we additionally fine-tune Gemma-3-12B-IT under the SFT Generation condition as a larger-scale reference point and to match the student used in the reasoning-distillation experiments (Section 4.4); the Gemma LR is selected by the same OOD validation procedure (\(1\mathrm{e}{-5}\)). The prompt templates used for SFT training are provided in Appendix 20.1.

15 Distillation Setup and Hyperparameters↩︎

15.1 Teacher Reasoning Trace Generation↩︎

We used vLLM [28] to generate reasoning traces from the teacher model. We applied the teacher model to our manually annotated intent dataset (Section 3) and partitioned the resulting traces into training, validation, and test splits matching our dataset construction pipeline.

Traces were generated under the three conditions introduced in Section 4.4: no-intent (the teacher receives only the gold harm label and justifies it directly, producing a Reasoning + Harm trace), synthetic-intent (the teacher receives only the gold harm label but is asked to generate an intent en route to the label, producing a Reasoning + Intent + Harm trace), and human-intent (the teacher receives both the gold harm label and the human-annotated intent and justifies both). Concretely, no-intent uses the classification-mode output format and the classification-mode teacher-ground-truth block (Appendix 20.2); human-intent uses the generation-mode versions of both; and synthetic-intent pairs the generation-mode output format with the classification-mode teacher block, so the teacher is asked to produce an intent without being shown the human-annotated one. Table 9 lists the inference hyperparameters.

15.1.0.1 Filtering Internal Reasoning Tokens

For models with native reasoning capabilities (e.g., Qwen and GPT-OSS variants), we parsed the output to remove internal chain-of-thought tokens (text enclosed within <think>...</think> tags). This ensures the student model trains exclusively on the final structured reasoning fields defined by our prompt templates.

15.2 Student Distillation↩︎

We conducted a hyperparameter sweep over multiple student and teacher models. Specifically, we evaluated three student models (Gemma-3-12B, Llama-3.1-8B, Qwen3-8B) trained from reasoning traces generated by four teachers (GPT-OSS-120B, Gemma-3-27B, Mistral Small 4 119B, Qwen3-32B). We trained separate models for each of the three experimental conditions: no-intent (reasoning and harm label), synthetic-intent (reasoning, model-generated intent, and harm label), and human-intent (reasoning, human-annotated intent, and harm label).

For each (teacher, student, condition) combination, we swept over learning rates (\(1\mathrm{e}{-5}\), \(2\mathrm{e}{-5}\), \(5\mathrm{e}{-5}\), \(1\mathrm{e}{-4}\)) and retained the highest-scoring adapter per cell using the training-time validation harm F1. The retained adapters were then re-evaluated on two held-out OOD validation sets (ToxicChat train and AEGIS 2.0 validation), and the final adapter is the cell with the highest mean F1 across these two sets. Figure 3 shows the full per-cell grid. In practice, this selects Gemma-3-12B trained with GPT-OSS-120B as the teacher under the human-intent condition.

We fine-tuned the selected configuration using QLoRA. The training configuration mirrors the SFT intent experiments (Table 8) with one deliberate change: we double the LoRA capacity to rank \(r=32\) and \(\alpha=64\) (from \(r=16\), \(\alpha=32\) used in SFT), since the student must now learn to produce a structured reasoning trace in addition to the intent and harm label. All other hyperparameters are identical to the SFT setup.

16 Preference Learning via DPO↩︎

We describe here the full implementation details of our DPO experiments: how preference pairs are sampled and filtered from the SFT model, the LLM-judge used to assess intent faithfulness, and the hyperparameter choices behind the main runs.

16.1 Pair Generation↩︎

Following the two-pass procedure in Section 4.2, the sampling temperature \(T\) controls the trade-off between candidate diversity and generation coherence. We swept \(T \in \{0.3, 0.6, 0.8, 1.0, 1.2\}\) on our AIMS train set, measuring preference pair yield, semantic diversity of sampled intents, agreement between sampled and gold harm labels, and the rate at which harm labels could not be extracted from the model output. Pair yield and intent diversity both increase with temperature, but at \(T=1.2\) a substantial fraction of generations become malformed and their harm label can no longer be parsed. We adopt T=0.8, which captures most of the diversity and yield gains over lower settings while keeping outputs coherent.

16.2 Judge Model↩︎

For IF-DPO, an LLM judge compares each generated intent to the human-annotated intent in the context of the original prompt and returns one of three verdicts: good_match, decent_match, or bad_match. We use Gemma-3-27B-IT as the judge; the full system prompt is in Table 22. We also experimented with another judge model, GPT-OSS-120B, but found that it led to worse downstream DPO performance on the AIMS validation set, so we used Gemma for the final IF-DPO experiments. A deliberate choice in the rubric is that the judge scores semantic agreement rather than surface-form similarity. In pilot runs, a stricter prompt that requested close alignment with the reference wording biased the judge toward near-verbatim matching and inflated the bad_match rate on intents that paraphrased the reference correctly. The final rubric instructs the judge to mark intents pointing to the same underlying goal as good_match even when they differ in specificity or wording, which concentrates bad_match verdicts on intents that substantively misrepresent the prompt’s safety-relevant content.

16.3 DPO Variants↩︎

We evaluate four DPO variants that differ only in how rejected completions are selected from the candidate pool. LE-DPO and IF-DPO are the two variants introduced in Section 4.2: LE-DPO rejects sampled intents whose deterministic label disagrees with the gold label, while IF-DPO rejects sampled intents that share the gold label but are judged to misrepresent the user’s goal. LE\(+\)IF-DPO combines the two criteria, building a single rejection set from the union of LE-DPO and IF-DPO rejections. LE\(\rightarrow\)IF-DPO is a curriculum variant that first trains on LE-DPO pairs and then continues training on IF-DPO pairs, exposing the model to label-flipping errors before the stricter intent-faithfulness signal.

Table 11 reports per-variant results on the five external safety benchmarks. LE-DPO and IF-DPO outperform both combined variants on average. We therefore report only LE-DPO and IF-DPO in Table 1, and treat LE\(+\)IF-DPO and LE\(\rightarrow\)IF-DPO as ablations confirming that simply mixing or sequencing the two rejection criteria does not yield further gains.

16.4 Training Setup↩︎

We train each DPO variant with QLoRA [36] on top of the SFT generation adapter. We select \(\beta\) and learning rate using one shared candidate pool, sweeping \(\beta \in \{0.1, 0.3, 0.5\}\) and learning rate \(\in \{2\times10^{-5}, 5\times10^{-5}\}\) for each of the four variants (LE-DPO, IF-DPO, LE+IF-DPO, LE→IF-DPO). Configurations are ranked by mean harmful-class F1 on the two OOD validation sets (Section 5). The combination \(\beta=0.3, lr=5\times10^{-5}\) emerged as the winner for every variant independently, and we adopt it uniformly. Holding these hyperparameters fixed, we then re-run the two-pass procedure 10 times to obtain 10 independent candidate pools, train one model per (variant, pool) combination, and select the run with the highest mean harmful-class F1 on the same OOD validation sets as the final reported checkpoint. Full hyperparameters are listed in Table 10.

5pt

17 Reinforcement Learning via GRPO↩︎

This appendix provides exhaustive details regarding our GRPO training environment, exact prompt templates, mathematical reward formulations, hyperparameters, initialization strategies, and comparative analyses against alternative optimization methods.

17.1 Hyperparameters and Infrastructure↩︎

To ensure reproducibility, we present the complete hyperparameter configuration used for all GRPO experiments (unless mentioned otherwise) in Table 12. All training runs were implemented using the VERL framework with a vLLM backend, distributed across a node of 8 NVIDIA H200 GPUs of 141GB VRAM.

17.2 Reward Formulation↩︎

The total reward \(R\) is calculated as a product of four distinct components: \[R = R_{\mathrm{format}} \times R_{\mathrm{label}} \times R_{\mathrm{len}} \times R_{\mathrm{intent}}.\] This multiplicative structure ensures that the model is only highly rewarded when all criteria – format correctness, safety label accuracy, and semantic intent alignment – are met simultaneously, thus preventing reward hacking.

17.2.0.1 Format Reward (\(R_{\mathrm{format}}\)).

This is a tiered reward assessing whether the model adheres to the structured output and terminates correctly. We assign \(1.0\) for a perfect stop, \(0.5\) for minor trailing characters (e.g., extra newlines), \(0.1\) for significant rambling or repetition, and \(0.0\) if the output can not be parsed according to the structured format.

17.2.0.2 Label Reward (\(R_{\mathrm{label}}\)).

This is a binary reward of \(1.0\) if the safety classification matches the ground truth and \(0.0\) otherwise. Due to the multiplicative nature of the reward, a wrong label results in a total reward of zero.

17.2.0.3 Length Reward (\(R_{\mathrm{len}}\)).

To prevent the policy from shifting toward degenerate, overly verbose, or trivially short intent summaries, we penalize generations whose intent length deviates substantially from the human reference intent length. Specifically, this reward is \(1.0\) if the generated intent length is within \(50\%\) to \(150\%\) of the reference intent length, and \(0.5\) otherwise.

17.2.0.4 Intent Reward (\(R_{\mathrm{intent}}\)).

An alignment reward obtained via Gemma-3-27B-IT acting as an LLM judge. Similarly to the DPO judge, it evaluates the (user prompt, reference intent, generated intent) triple and returns a verdict – good_match, decent_match, or bad_match – which is mapped to an intent reward of \(1.0\), \(0.5\), or \(0.1\), respectively.

17.3 SFT vs. Base Model Initialization↩︎

In our early design phases, we explored initializing the GRPO policy from the intent-supervised SFT model. While SFT pre-training is standard practice in RLHF pipelines to guarantee quick formatting compliance, we observed a severe training bottleneck that led us to initialize directly from the base Llama-3.1-8B-Instruct model instead.

The prior SFT checkpoint exhibited a highly stubborn conditioning effect: the model repeatedly bypassed the newly introduced sequential reasoning steps prescribed in the GRPO system prompt, attempting instead to immediately output the intent summary and safety classification in the exact style of its static supervised training. This structural formatting mismatch violated the required <reasoning> blocks entirely, leading to a parsing failure (\(R_{\mathrm{format}} = 0\)) and causing the policy to receive a near-zero overall reward for several training epochs. Furthermore, this rigidity resulted in a severe exploration collapse: the model exhibited low-entropy reasoning trajectories, limiting its capacity to discover the nuanced and dual-use safety boundaries of latent user intents.

By contrast, relying entirely on the base instruct model coupled with our structured system prompt allowed the policy to learn both sequential formatting and complex reasoning simultaneously. Nonetheless, it is worth noting that if training is allowed to run to completion, the SFT-initialized model does eventually recover and adapt, ultimately achieving a highly competitive average \(F_1\) score of \(0.826\) across our five test sets when using the full reward function (i.e., with intent reward).

17.4 Comparative Analysis with DAPO↩︎

Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) [37] is a state-of-the-art RL alternative designed to stabilize long-CoT reasoning tasks by introducing token-level policy gradients and dynamic sample filtering. We benchmarked a DAPO variant against our GRPO implementation on our safety task and observed distinct behavioral differences. Firstly, because our safety reward relies on hard logical gates (both \(R_{\mathrm{label}}\) and \(R_{\mathrm{format}}\) may lead to \(R=0\) for incorrect answers), many rollouts in a group yield zero rewards. DAPO’s dynamic sampling filter discards prompts where rollout variance is extremely low or uniform. In our sparse safety landscape, this caused DAPO to repeatedly flag entire batches as uninformative. Secondly, because uniform-zero rollout groups were continuously rejected, the system was forced to resample new batch completions multiple times. This excessive resampling behavior exhausted the maximum generation steps, resulting in premature training termination. GRPO’s group-relative comparison handled these high-contrast reward distributions robustly without dynamic sampling bottlenecks. Nonetheless, DAPO still performs competitively, reaching an average \(F_1\) score of up to \(0.824\) across the five test sets.

18 Inference Efficiency↩︎

Table 13 reports per-prompt wall-clock time, generated tokens, and mean F1 across the five test benchmarks. The resulting latency–F1 trade-off is dominated by our intent-aware models.

4pt

18.0.0.1 Measurement protocol.

All systems are evaluated on a single RTX 6000 Pro GPU under vLLM 0.19.0 continuous batching, with all evaluation benchmarks submitted together as one batched run. We report per-prompt latency in milliseconds (ms) and mean generated tokens per prompt.

18.0.0.2 Output verbosity explains the gap to prior work.

Our cheapest intent-aware models are faster than dedicated guardrails of similar scale. WildGuard 7B is of similar scale as Llama-3.1-8B SFT, but it adheres to a verbose three-field template that reports prompt safety, response safety, and refusal status. The latter two fields are unused in prompt-only moderation, yet the model still emits all 23 tokens per prompt, raising its latency to 6.46 ms versus 4.66 ms for our SFT baseline. Nemotron Safety 4B is the smallest model among the dedicated guards, yet LE-DPO is still faster: Nemotron’s reasoning output averages 70.1 tokens per prompt versus 25.8 for LE-DPO, and the additional token decoding outweighs its smaller parameter count.

19 Qualitative Error Analysis: Full Discussion↩︎

This appendix expands the qualitative analysis summarized in Section 7. The error sets are taken from AIMS validation (\(n=61\)), WildGuardTest (\(n=205\)), and ToxicChat (\(n=252\)), restricted in each case to the prompts on which SFT Generation makes the wrong prediction. We compare SFT’s generated intent against those produced by LE-DPO, IF-DPO, and GRPO. Representative cases are listed in Table 14; we organize them below by the patterns they illustrate.

19.0.0.1 The direction of SFT’s errors flips with distribution.

SFT’s failures are not symmetric across datasets, and this asymmetry is revealing. On AIMS and WildGuardTest, where adversarial framing is prevalent, \(64\%\) and \(69\%\) of SFT errors are false negatives: prompts with a harmful underlying intent that SFT labels safe. On ToxicChat, drawn from organic user interactions, errors are split roughly evenly, with a slight majority (\(52\%\)) of false positives: benign requests that SFT flags as harmful. The two regimes correspond to the two failure modes a guardrail must handle: missing harm hidden behind cover stories, and over-refusing prompts whose surface form is suggestive but whose intent is benign. The qualitative patterns differ between these regimes.

19.0.0.2 Under adversarial framing, SFT’s generated intent tracks the cover story.

On adversarial false negatives, SFT’s intent strings closely mirror the prompt’s framing rather than its underlying goal (Table 14, rows 1–2). Across these cases, SFT repeats the framing words used in the prompt (“hypothetical”, “historical reenactment”, “fictional”, “educational”, “professor”, “researcher”) and leaves out the harmful content the framing is meant to disguise; LE-DPO, IF-DPO, and GRPO frequently do the opposite, naming the underlying goal directly. The fact that SFT still produces these surface-level intents despite being trained on intent–label pairs is itself informative: maximum likelihood on the annotated targets is not enough, on its own, to make the model see past the adversarial framing. The contrastive signal in DPO – which down-weights sampled intents whose deterministic label flips the safety decision – and the intent-faithfulness reward in GRPO are both designed to close this gap, and the qualitative pattern is consistent with them succeeding.

19.0.0.3 On benign prompts, SFT over-attends to harm-adjacent keywords.

ToxicChat over-refusals have different particularities. SFT often produces an intent that is correct in substance but whose surface keywords appear to anchor the model to a harmful label (Table 14, rows 3–4). A particularly clean example is the meta-question “Are you allowed to tell me instructions about illegal activities?”. SFT’s intent is “Ask about instructions for illegal activities” (predicted harmful); GRPO’s intent is the near-identical “Ask about instructions on illegal activities” (predicted safe). Of the 71 ToxicChat over-refusals that GRPO recovers, six have an SFT–GRPO intent Jaccard overlap above \(0.5\) – short, lexically similar intents that nonetheless receive opposite labels. The signal from DPO and GRPO therefore reshapes not just what intent is produced, but how the intent grounds the label. The implication is that intent generation under likelihood supervision alone is not sufficient; the model can still let surface keywords override its own description of the user’s goal. DPO and GRPO close this gap by tying label correctness to the specific intent the model produces, either contrastively or as part of a reward signal.

19.0.0.4 DPO and GRPO are complementary.

Across the three datasets, GRPO recovers some prompts that neither LE-DPO nor IF-DPO catches (3, 18, and 19 cases respectively), while both DPO variants together recover prompts that GRPO misses (5, 19, and 27). The two error sets have qualitatively different character. GRPO’s exclusive wins concentrate on over-refusal: prompts whose surface content looks harmful but whose intent is benign once the reasoning trace makes the literal request explicit (Table 14, row 5). DPO’s exclusive wins concentrate on adversarial cover stories: prompts where the fictional or research-oriented framing is the obstacle and the contrastively learned intent strips it away (Table 14, row 6). The two regimes appear to focus on different aspects of intent-aware behavior: contrastive supervision against sanitized intents helps the model resist adversarial framing, while structured reasoning followed by intent–label scoring helps it avoid hallucinating harm from surface cues. This complementarity is consistent with the aggregate result that, while the best method across benchmarks is GRPO with both label and intent rewards, no single intent-aware regime dominates on every benchmark.

19.0.0.5 Residual difficulty: where DPO and GRPO still fail.

A non-trivial fraction of SFT errors are unrecovered by any of LE-DPO, IF-DPO, or GRPO (AIMS \(34\%\); WildGuardTest \(31\%\); ToxicChat \(27\%\)). These cases highlight the limits of intent inference. First, deeply embedded adversarial framing that even the intent-aware methods absorb – a “civil engineer” asking for “detailed steps on the secure and most controlled method to eradicate a massive, antique infrastructure” is annotated harmful, but every model labels it safe and produces a demolition-guide intent that is not obviously incorrect (Table 14, row 7). Second, dual-use prompts where the user’s goal is reasonably described but the appropriate label is contested. Third, persistent over-refusal on creative or hypothetical prompts containing harm-adjacent vocabulary – a party-game request for “hot truth or dare tasks” is annotated safe but all four models flag it as harmful (Table 14, row 8). The first pattern suggests room for stronger intent supervision; the second and third highlight inherent label ambiguity in safety classification, against which any single binary decision will struggle.

19.0.0.6 Takeaways.

The qualitative picture supports three claims that aggregate metrics alone do not establish. (i) DPO and GRPO correct qualitatively different SFT failure modes across distributions – sanitized intents on adversarial prompts and keyword-driven over-refusals on organic ones. (ii) The improvement is not only in what intent is predicted, but in the connection between that intent and the final label; the ToxicChat near-paraphrase cases show this directly. (iii) DPO-based contrastive supervision and GRPO-based reasoning rewards are complementary, addressing different sides of the guardrail trade-off; this is consistent with the aggregate result that no single regime dominates every benchmark.

4pt

20 Prompt Templates↩︎

20.1 SFT and Prompting Baselines↩︎

Tables 15–18 show the four system prompts used in Section 4.1. Vanilla and SFT variants share the same prompt; CoT variants add a step-by-step reasoning field. The user message is always the prompt to be classified.

20.2 Distillation↩︎

To perform knowledge distillation from the teacher model (using privileged ground-truth information) to the student model, we employ a modular prompting framework. Rather than using monolithic text blocks, we construct the input prompts by concatenating the components in sequential order:

1. Preamble & Taxonomy: The foundational context and definitions provided to all models. We use two preamble variants: an intent-bearing preamble for the synthetic_intent and human_intent conditions, and a simpler harm-only preamble for the no_intent condition.

2. Task-Specific Output Format: Instructions dictating the expected response structure (either direct classification or intent generation).

3. Teacher-Specific Instructions (Teacher only): Explicit ground-truth guidance appended exclusively for the teacher model during the reasoning trace generation phase. There are three variants — one per training condition.

The overall structural flow is Preamble \(\rightarrow\) Output Format \(\rightarrow\) Teacher Instructions (if applicable), detailed in Tables 19–21.

20.3 DPO / GRPO Judge↩︎

The DPO intent-faithfulness judge and the GRPO intent reward both use the Gemma-3-27B-IT system prompt in Table 22. It receives the original prompt, the gold intent, and the generated intent, and returns one of three verdicts: good_match, decent_match, or bad_match.

20.4 GRPO↩︎

Table 23 shows the GRPO policy system prompt, which instructs the model to perform step-by-step reasoning inside <reasoning> tags before outputting the intent and harm label.

References↩︎

---

1. Code, models, and data are available at jazhyc.github.io/aims-safety.↩︎

2.   Equal Contribution.↩︎