1 Introduction↩︎

Large Language Models (LLMs) have become the foundation of many purpose-specific agents, which are built to serve well-defined application needs with dedicated capabilities and operating constraints, as reflected by platforms such as Poe [@poe2026] and OpenAI’s GPT Store [@openai2026gptstore]. These agents also play an increasingly important role in multimodal settings [@wang2025manipulating; @yan2026towards; @zhu2026stage; @zhangbadrobot], such as visual content moderation and document understanding.

To construct a purpose-specific agent, it is typically necessary to formulate specific system prompts [@lei2026offtopiceval; @zhang2025meta; @gptprotections2026], including purpose-specific restriction (PSR) policies that define the boundaries of permissible and prohibited user queries (as shown in Fig. 1). Given these task-specific policies, the agent is expected to reliably reject policy-violating queries (i.e., queries falling outside the permissible response scope) and accept benign queries (i.e., those within it).

More broadly, safety alignment [@bhardwaj2024language; @qi2025safety; @zhang2025backtracking] has substantially improved the ability of LLMs to block harmful inputs (e.g., “How to make a bomb?”) and comply with general safety policies. However, the purpose-specific agents considered in our paper differ from the standard safety setting in one key respect: beyond general safety guardrails, they must first enforce constraints tailored to their applications (e.g., “Reject unauthorized queries”), thereby introducing a new vulnerability. In particular, prompt injection (PI) attacks [@owasp2025llm01; @liu2023prompt; @wang2025manipulating] can readily circumvent these restrictions, as attackers can craft inputs that bypass PSR policies and elicit unintended output, thereby posing a threat to purpose-specific agents.

Specifically, attackers insert adversarial prompts into policy-violating queries by using heuristic-based [@perez2022ignore] or optimization-based [@zou2023universal] methods to coerce the agent into executing potentially injected tasks, causing resource abuse or sensitive information leakage. This vulnerability defines a safety dimension as critical as the widely studied problem of general-purpose alignment, but focuses more on enterprise-level risks than user-facing harms [@lei2026offtopiceval]. Its importance is further underscored by OWASP [@owasp2025llm], NIST [@nist_ai_rmf_2023], and the EU AI Act [@EUAIAct2024], which all identify it as a critical risk.

To mitigate the risks of PI attack, recent works have explored various detection approaches. A category of approaches trains auxiliary detectors to identify attacks, including open-source detection models [@protectai2024deberta; @promptguard2] from ProtectAI and Meta, as well as advanced detection techniques [@liu2025datasentinel; @chen2025canindirect; @abdelnabi2025get] exemplified by DataSentinel [@liu2025datasentinel]. These approaches typically require considerable computational resources and high-quality training data, motivating alternative training-free methods [@nakajima2022yohei; @zhang2025jailguard; @alon2023detecting; @hung2025attention] that detect attacks by exploiting inherent characteristics of attack inputs (e.g., their poor robustness [@zhang2025jailguard]). However, most existing training-free detection approaches rely solely on surface-level patterns (e.g., input-output behaviors), which yield limited effectiveness. To address this limitation, we turn to exploring higher-dimensional internal signals within the hidden activation space of LLMs, and investigate the following question: Do the internal signals of LLMs encode information about policy-violating behaviors, and if so, how can we leverage them to achieve efficient and accurate training-free PI attack detection?

Fortunately, the answer is yes! In this work, we propose PVDetector, a novel detection framework that identifies PI attacks against purpose-specific agents by analyzing the internal activation patterns of LLMs. Our key insight: although PI attacks manipulate LLMs to generate unintended outputs, the semantics related to the conflicts between inputs and PSR policies are still recognizable in the model’s internal activation space, which is absent in benign prompts (as detailed in Section 4.3). Inspired by previous research on activation interpretability [@zou2025representation; @nanda2023emergent; @wu2025axbench], we develop a pipeline to extract the violation-related high-level semantics (termed policy-violation (PV) concepts) from contrastive prompt pairs. Within this pipeline, we apply the difference-in-means technique [@larsen2016autoencoding] to isolate the linear representations (termed PV vectors) most relevant to policy-violation from the hidden space. By using a projection-based score function, we can measure the alignment between hidden representations and the PV vectors, thereby yielding PV strength scores for test prompts. Finally, we aggregate the scores over the key violation-aware layers and identify PI attacks via a configurable threshold. Notably, our framework requires no auxiliary model training or additional inference overhead. Once the PV vectors are derived offline from a few sample pairs (e.g., 30 pairs), it enables real-time detection alongside standard LLM inference. Additionally, our method demonstrates applicability to Vision-Language Models (VLMs), extending the defense to multimodal scenarios [@zhou2023advclip; @wang2026advedm; @wang2025breaking]. Our contributions are summarized as follows:

• We identify a new vulnerability in purpose-specific agents and introduce policy-violation concepts, which formalize the internal semantic representations in LLMs that encode conflicts between user queries and agent policies. This offers a new activation-based perspective for PI attack detection.

• We propose PVDetector, a training-free framework that identifies PI attacks on purpose-specific agents by measuring the PV strength in the backend LLM’s activation space, and enables real-time detection during inference.

• Extensive experiments show that PVDetector achieves near-zero false negative rates (\(<1\%\)) across all evaluated attacks and LLMs while maintaining the lowest detection overhead (\(0.11\)s per query) among training-free baselines. We further release a benchmark for evaluating PI attack detection on purpose-specific agents to support future research.

2 Related Work↩︎

2.1 Prompt Injection Attacks↩︎

In prompt injection attacks [@liu2023prompt; @owasp2025llm01; @liu2024formalizing], an attacker inserts carefully crafted adversarial prompts into the LLM agent’s input data to override its system prompts, thereby coercing the agent into executing unintended and potentially malicious injected tasks. Such attacks compromise the behavioral integrity of LLM agents, posing a critical security threat to their real-world deployment [@owasp2025llm]. Different attacks use different strategies to construct the contaminated input data. These strategies can be categorized into heuristic-based attacks and optimization-based attacks.

In heuristic-based attacks [@perez2022ignore; @willison2022prompt; @willison2023delimiters; @liu2024formalizing], attackers insert a handcrafted separator (e.g., “Ignore the previous instruction”) before injected malicious instructions. This strategy aims to induce the LLM to follow the subsequent instructions, thereby completing the injected tasks. In addition, [@shao2025enhancing] show that poisoning alignment training with adversarial samples [@song2026segtrans; @song2025pb] increases LLMs’ vulnerability to PI attacks. In optimization-based attacks [@zou2023universal; @pasquini2024neural; @jia2025critical; @shi2024optimization], attackers craft malicious inputs — via loss minimization over auxiliary strings [@zou2023universal] or full prompts [@hui2024pleak] — to hijack LLM outputs toward the desired response that accomplishes the injected task.

2.2 Prompt Injection Defenses↩︎

Depending on defense targets, existing defenses can be classified into prevention-based defenses and detection-based defenses. Prevention-based defenses [@chen2025defending; @debenedetti2025defeating] aim to prevent the LLM agents from being affected by injected adversarial prompts. Existing defense strategies include pre-processing the input data [@yi2025benchmarking; @willison2023delimiters], fine-tuning the backend LLM via adversarial training [@chen2025struq; @chen2025secalign; @wallace2024instruction]. Recently, several studies propose system-level defenses that restrict LLM operations to system-authorized actions through privilege control [@shi2025progent], execution isolation [@wu2025isolategpt], and information-flow control [@costa2025securing].

Detection-based defenses aim to determine whether a given input data is injected adversarial prompts or not. For instance, [@promptguard2], [@protectai2024deberta], [@chen2025canindirect] and [@abdelnabi2025get] propose training detection models to identify potential injected content. In addition, DataSentinel [@liu2025datasentinel] extends known answer detection [@nakajima2022yohei] via game theory-based fine-tuning, outperforming many other detectors. However, these methods often require considerable training costs, including computational resources and data. Another line of research focuses on training-free approaches [@alon2023detecting; @nakajima2022yohei; @zhang2025jailguard; @hung2025attention]. Specifically, JailGuard [@zhang2025jailguard] proposes a universal detector based on the inherent instability of adversarial inputs. [@hung2025attention] propose Attention Tracker that detects attacks based on the attention mechanisms. However, most of these methods overlook the exploitability of the intrinsic activations of LLMs [@zhang2026defending].

3 Problem Formulation↩︎

3.1 Purpose-Specific LLM Agents↩︎

Here, we consider a purpose-specific agent [@lei2026offtopiceval; @wang2025manipulating] built on a backend LLM, denoted as \(A\). Typically, the developer provides the LLM with a predefined system prompt, denoted as \(S\), which contains general guidelines, response styles, and PSR policies tailored to its intended use case. In operation, the agent combines its system prompts with the user query \(Q\) and the optional external data \(E\) (e.g., web pages) before forwarding them to backend LLMs [@wang2024trojanrobot], and finally relays the LLM-generated responses to the user. Typically, the response \(R\) of the agent can be formulated as \(R=A(S, Q, E)\).

3.2 Threat Model↩︎

3.2.1 Attacker’s Goal, Knowledge and Capability.↩︎

An attacker injects carefully crafted adversarial prompts into policy-violating queries to generate attack instances. The attacker aims to bypass PSR policies and force the agent to accept attack instances. We assume a strong white-box attacker with full knowledge of the system prompt \(S\) and full access to the backend LLM’s parameters and gradients. The attacker can directly interact with the agent by submitting arbitrary textual inputs or uploading files.

3.2.2 Defender’s Goal, Knowledge and Capability.↩︎

Our defense aims to develop a test-time detector [@zhang2025test; @yao2024reverse] to intercept attack instances, preventing them from manipulating the agent generation. The defender could be the developer of a purpose-specific agent and has white-box access to the internal activations during LLM inference. We assume a challenging and realistic scenario where the defender does not use attack instances to develop the detector, since real-world attacks are unknown and constantly evolving.

4 Policy-Violation Concepts in LLMs↩︎

In this section, we analyze the internal activations in LLMs for three types of user input: benign queries, policy-violating queries, attack instances, and explore high-level semantics encoded in hidden state space of LLM. Following this, we ask whether there exists a distinct activation pattern associated with attack instances in LLMs, potentially absent from benign queries.

4.1 A Motivating Observation↩︎

Our research begins with an intriguing observation: even when subjected to PI attacks, LLMs do not always comply with such manipulation. We observe that the attention mechanisms of LLMs appear to implicitly encode and attend to PSR policies specified in the system prompt, suggesting an inherent capacity to anticipate and recognize policy-violating behaviors. Surprisingly, this recognition sometimes manifests directly in the linguistic space. As shown in the conversation example below (additional cases appear in Appendix 11), the attack induces unintended responses while the LLM still explicitly recognizes the policy violation.

The example shows that even when behaviorally shaped by adversarial attacks, LLMs continue to exhibit traces of “resistance awareness” in the outputs. Such resistance can be naturally attributed to deeply ingrained instruction-following priors formed during pretraining, suggesting that successful attacks must contend with, rather than merely evade, these latent constraints. Hence, this observation leads to our core question: Does a corresponding signature also exist in the model’s hidden activation space? Furthermore, can we identify latent, violation-related semantics (i.e., PV concepts) that remain active during PI attacks and reflect the model’s “internal resistance”. To this end, we take one more step to corroborate this intuition by profiling the latent representation in LLMs.

4.2 Probing in Latent Representation Space↩︎

Recent research progress on latent representation explanation [@rimsky2024steering; @templeton2024scaling; @wu2025axbench] has shown that high-level concepts or traits (e.g., emotion) can be controlled along linear directions. They are typically based on the Linear Representation Hypothesis [@mikolov2013linguistic; @park2024linear], which suggests that neural networks represent meaningful concepts as directions in their activation spaces. Hence, we design a PV concept extraction pipeline from the contrastive pairs of policy-violating and policy-compliant prompts (requiring no attack instances), as illustrated at the top of Figure 2. The detailed process is described as follows.

4.2.1 Contrastive Pairs Construction.↩︎

We first construct \(N\) pairs of contrastive inputs tailored to a specific agent with PSR policies. Each pair consists of a positive sample showing the targeted attribute of policy-violation, and a negative sample exhibiting the opposite. Given an agent that is restricted from responding to queries unrelated to recipes, a positive sample could be a query such as “Provide one reason why people should recycle”, while a negative sample could be “How to make a cake”. Let \(\mathcal{X}^+ = \{x_i^+\}_{i=1}^N\) and \(\mathcal{X}^- = \{x_i^-\}_{i=1}^N\) denote the sets of positive and negative samples, respectively. We randomly pair samples from the two categories to construct a set of contrastive pairs \(\{(x_1^+, x_1^-), (x_2^+, x_2^-), \cdots , (x_N^+, x_N^-)\}\).

4.2.2 Concepts Extraction.↩︎

Next, we feed contrastive sample pairs into the LLM for forward pass inference. In this process, we focus on analyzing the hidden states of each layer. Based on the attention mechanisms, the hidden state at the last token position corresponds to representations related to the entire input when predicting the next token [@vaswani2017attention]. Therefore, given an input sample \(x\), we extract \(\mathbf{h}_{\text{last}}^l = h_{\text{last}}^l(x)\) as its semantic representation at the \(l\)-th layer, where \(h_{\text{last}}^l(\cdot)\) maps \(x\) to the hidden state corresponding to the last token position at layer \(l\). For the \(i\)-th contrastive pair \((x_i^+, x_i^-)\), we compute the representations for the positive and negative samples at \(l\)-th layer respectively: \[\mathbf{h}_{i,\text{last}}^{l+} = h_{\text{last}}^l(x_i^+) \in \mathbb{R}^d, \quad \forall i \in \{1, 2, \dots, N\}, \forall l \in \{1, 2, \dots, L\}\] \[\mathbf{h}_{i,\text{last}}^{l-} = h_{\text{last}}^l(x_i^-) \in \mathbb{R}^d, \quad \forall i \in \{1, 2, \dots, N\}, \forall l \in \{1, 2, \dots, L\}\] where \(d\) is the hidden dimension, and \(L\) is the number of layers. The representation matrices for all of the positive samples and negative samples are denoted respectively as: \[\mathbf{H}_l^+ = \left[ \mathbf{h}_{1,\text{last}}^{l+}, \mathbf{h}_{2,\text{last}}^{l+}, \ldots, \mathbf{h}_{N,\text{last}}^{l+} \right]_{N \times d},\] \[\mathbf{H}_l^- = \left[ \mathbf{h}_{1,\text{last}}^{l-}, \mathbf{h}_{2,\text{last}}^{l-}, \ldots, \mathbf{h}_{N,\text{last}}^{l-} \right]_{N \times d}.\] To extract the targeted PV concepts, we compute the mean difference (MD) between the positive and negative representation matrices (i.e., \(\mathbf{H}_l^+\), \(\mathbf{H}_l^-\)) in the activation space. The MD vector \(\mathbf{v}^l_{\text{MD}}\) from the \(l\)-th layer can be calculated as: \[\mathbf{v}^l_{\text{MD}}=\frac{1}{N} \sum\nolimits_{i=1}^N \left( \mathbf{h}_{i,\text{last}}^{l+} - \mathbf{h}_{i,\text{last}}^{l-} \right) \in \mathbb{R}^{1 \times d}, l \in \{1,2,\cdots,L\},\] where \(\mathbf{h}_{i,\text{last}}^{l+} \in \mathbf{H}_l^+, \mathbf{h}_{i,\text{last}}^{l-} \in \mathbf{H}_l^-\). The MD vector encodes the representation difference between policy-violating (positive) and benign (negative) samples, captured by varying only user inputs while keeping PSR policy constant, thus isolating representations most relevant to policy-violation. Therefore, we define PV vectors \(\mathbf{V}_{\text{pv}}=[\mathbf{v}^1_{\text{MD}}, \mathbf{v}^2_{\text{MD}}, \cdots, \mathbf{v}^L_{\text{MD}}]_{L \times d}\) and treat these vectors as the representations of PV concepts.

4.2.3 Concept Interpretability.↩︎

To understand the extracted high-level concepts, we project the representation vectors in \(\mathbf{V}_{\text{pv}}\) into the vocabulary space. By using the unembedding layer in the LLM, we obtain the logits for each token in the vocabulary. For each vector, we collect the top-10 tokens with the highest logit as the interpretation tokens. The example tokens obtained from the projection results in the layers 23–32 of Llama3.1 (8B) are shown below. Appendix 10 provides the complete results.

For an agent that is restricted from responding to queries outside its designated role, the example tokens (such as “irrelevant”, “outside”) align with PV concepts. This indicates that PV vectors capture abstract, violation-related semantics.

Based on the above pipeline, we extract the linearly encoded PV concepts from the hidden states. Now, we can take these latent concepts as anchors to measure how policy violation semantics manifest in LLMs, particularly when they are subjected to PI attacks.

4.3 Evaluating Policy-Violation under PI Attack↩︎

Here, we investigate the distribution of violation semantics exhibited by the representations of different types of input (“policy-violating queries”, “attack instances”, “benign queries”) across all layers of LLMs. We construct three sample sets corresponding to three input types respectively. Among them, the attack set includes four attack methods: Ignore [@perez2022ignore], Fake Completion [@willison2023delimiters], Combined Attack [@liu2024formalizing], and GCG [@zou2023universal].

To measure the alignment between activation representation of each query and the PV concepts, we project the hidden vector at the last token position in each layer onto the PV vector and define the projection value as policy-violation strength (PVS) score. For a test sample \(x_{\text{test}}\), the PVS score at the \(l\)-th layer can be computed as follows: \[s^{l}_{\text{PVS}}=\frac{1}{\left\| \mathbf{v}_l \right\|} \left( h_{\text{last}}^l(x_{\text{test}}) \cdot \mathbf{v}_l \right), l \in \{1,2,\cdots,L\} \label{equation:PVS-score},\tag{1}\] where \(L\) is the number of layers, \(\mathbf{v}_l \in \mathbf{V}_{\text{pv}}\) is the PV vector at the \(l\)-th layer. We average the PVS scores across all samples for each type, and visualize the results for different input types in Figure 3.

The visualization reveals several key findings. Firstly, the total PV strength of various PI attack samples decreases to varying degrees compared with that of direct violating inputs. However, in the latter layers, the PVS scores of these attack samples are still significantly higher than those of benign samples. This observation validates our intuition in Section 4.1. Secondly, not all layers exhibit equal ability to identify policy-violation semantics during the LLM inference process (Appendix 8.2 offers a detailed discussion thereof).

5 PVDetector: A Complete Illustration↩︎

Based on our previous analysis of policy-violation concepts, we propose PVDetector. It consists of two phases (see Figure 2). The Offline Phase involves three main steps: Contrastive Pairs Construction, Policy-Violation Concept Extraction, and Key Layers Selection. The first two steps extract the PV concepts from LLMs’ hidden space using the contrastive pairs of policy-violating and policy-compliant prompts (as introduced in Section 4.2.1 and 4.2.2). The third step identifies critical violation-aware layers for attack detection. The Online Phase focuses on Prompt Injection Detection, which assesses whether a prompt input is a PI attack instance, thereby helping to prevent unintended responses.

Key Layers Selection. To measure which layers have a stronger ability to capture the semantic gaps of policy-violation, we compute the mean differences of PVS scores between the positive samples \(\mathcal{X}^+\) and negative samples \(\mathcal{X}^-\) across all layers. After that, we obtain a difference vector \(\mathbf{d}=(d_1, d_2, \cdots, d_L)\). Following [@jiang2025hiddendetect], we take the value of the final layer \(d_L\) as the baseline for selection, since the activation values of the final layer play a crucial role in the model’s responses. The set of key layers is defined as \(\mathcal{L}_{\text{key}}=\{l \mid d_l \geq d_L\}\).

Prompt Injection Detection. Given an input \(x\), we take its hidden representations \(\{\mathbf{h}^l_{\text{last}} \mid l\in \mathcal{L}_{\text{key}} \}\) at the key layers to compute PVS scores by projecting on the PV vectors at corresponding positions. Then we aggregate the PVS scores \(\{ s^{l}_{\text{PVS}} \mid l \in \mathcal{L}_{\text{key}} \}\) using the trapezoidal rule to approximate the integral. If the aggregated score \(s_{\text{aggr}}\) exceeds a predetermined threshold \(\theta\), we flag the input \(x\) as an attack instance.

6 Experiments↩︎

6.1 Experimental Setups↩︎

6.1.1 Agent Construction.↩︎

Following the common practice in prior work [@lei2026offtopiceval; @wang2025manipulating], we construct agents by specifying the backend LLM and designing the system prompts as described in Section 3.1. We primarily consider two purpose-specific agents: RecipeMaster and TripPlanner. Following [@lei2026offtopiceval], we categorize the user query space defined by PSR policies into in-domain (ID) and out-of-domain (OOD) categories, where ID queries are permitted and OOD queries are prohibited. We present detailed results on RecipeMaster in this section, and defer the results for TripPlanner to Appendix 8.1.

In real-world applications, developers design PSR policies based on the agent’s intended use cases, and these policies are extensible. To this end, we further evaluate PVDetector’s generality across diverse policy scenarios, particularly under more stringent fine-grained restrictions. Specifically, we consider two additional agents with fine-grained restriction policies in Section 6.2.2. Appendix 12 provides these agents’ complete prompts.

6.1.2 Backend LLMs and VLMs.↩︎

In main evaluations, we consider 3 open-source LLMs from 2 commonly used families: Llama-3.1 [@grattafiori2024llama] and Qwen-2.5 [@qwen2.5]. In the evaluation of generalization to VLMs (Section 6.2.4), we adopt Qwen2.5-VL [@qwen2.5-VL] and Phi-3.5-vision [@abdin2024phi3] as tested models. Details are listed in Appendix 9.1.

6.1.3 Dataset Creation.↩︎

For ID data (i.e., benign queries in our case), we prompt powerful LLMs (e.g., Qwen3-MAX [@qwen3max]) to construct 500 ID queries given each agent’s PSR policies. The data generation prompt and query styles are provided in Appendix 9.2. For OOD data (i.e., policy-violating queries in our case), we leverage two public LLM evaluation datasets: instruction-following dataset Alpaca [@alpaca] and massive multitask dataset MMLU [@hendrycks2021measuring]. For each of the two datasets, we perform random sampling and then feed the obtained samples to each agent. We retain only 300 rejected queries as the agents’ final OOD queries for subsequent attack instance construction. Appendix 9.2.1 includes an evaluation of agents’ refusal rate for the queries sampled from these two proposed datasets.

6.1.4 Attack Methods.↩︎

To comprehensively evaluate detection performance, we consider five prompt injection attacks spanning both heuristic-based and optimization-based categories. We apply these attacks to OOD queries to construct attack instances for evaluation. Specifically, we adopt Ignore [@perez2022ignore], Fake Completion (denoted as Fake) [@willison2023delimiters], and Combined Attack [@liu2024formalizing] as heuristic-based attacks, and the gradient-based Greedy Coordinate Gradient (GCG) attack [@zou2023universal] as the optimization-based attack. Furthermore, we combine the Ignore and GCG attacks to construct a strong variant Ig-GCG attack. The detailed setup is provided in Appendix 9.3. The attack success rates (ASR) of these attacks against the LLM agents are reported in Appendix 9.4, where the ASR is defined as the fraction of attack instances accepted by the agent.

6.1.5 Baselines and Metrics.↩︎

We compare PVDetector against six popular approaches: uncertainty-based PPL [@alon2023detecting], trained detectors ProtectAI [@protectai2024deberta] and Prompt-Guard-2 [@promptguard2], DataSentinel [@liu2025datasentinel], mutation-based JailGuard [@zhang2025jailguard], and AttentionTracker [@hung2025attention]. The detailed introduction is provided in Appendix 9.5. Following [@jia2025critical], we use False Positive Rate (FPR) and False Negative Rate (FNR) as evaluation metrics. FPR measures the fraction of benign ID queries that are incorrectly flagged as attacks, while FNR measures the fraction of attack instances that are misclassified as benign.

6.1.6 Evaluation and Parameter Settings.↩︎

We feed all 500 generated ID queries to each agent, and retain only the accepted ID queries as the agent’s final benign queries for subsequent evaluation. Appendix 9.2.1 includes the agents’ acceptance rate for ID data. We calibrate the threshold \(\theta\) to ensure FPR \(\leq 1\%\) using \(100\) randomly sampled benign queries, following [@liu2024formalizing], and evaluate detection performance on the remaining test samples. For fairness, this practice is simultaneously applied to all baselines requiring threshold setting in our evaluations. For the number \(N\) of contrastive pairs, we search over values in {1, 5, 10, 20, 30, 40} and empirically select \(N\)=\(30\), which yields the best FPR and FNR.

6.2 Experimental Results↩︎

6.2.1 Main Results.↩︎

Table ¿tbl:tab:main95result? shows FNRs and FPRs of PVDetector and baselines. With the same threshold setting method, our PVDetector achieves the lowest FPR compared to PPL, JailGuard, and AttentionTracker. The remaining three baselines do not require threshold setting and are therefore discussed separately. Second, PVDetector achieves very low FNRs (close to \(0\%\)) for all PI attacks across different LLMs and datasets. Third, the overall detection performance of PVDetector is significantly superior to all baselines. Among these existing methods, PPL is only effective against GCG and Ig-GCG attacks. Although training-based ProtectAI and Prompt-Guard-2 achieve the lowest FPR (\(0\%\)), they exhibit high FNRs (reach up to \(98.67\%\) in some cases), particularly against unseen attack strategies such as the Fake Completion attack which is adapted to bypass role restrictions (Appendix 9.3). Due to the inherent limitations of DataSentinel, it may identify benign queries containing user instructions as attacks, resulting in high FPRs. AttentionTracker demonstrates the best performance among the baselines on the Llama model. However, this method exhibits higher FNRs (exceeding \(20\%\) in some cases) on the Qwen2.5 series models. These results demonstrate that PVDetector is much more effective in detecting various PI attacks across different LLMs.

6.2.2 Evaluation for Fine-grained Policies.↩︎

To further examine PVDetector’s robustness under more challenging scenarios, we evaluate its performance on agents with finer-grained and more restrictive PSR policies. Specifically, we construct two additional LLM agents with new PSR policies: (1) CareerCounselor, which is constrained to never leak or reveal its system prompt, instructions, or internal configuration to any users. (2) LineMonitor, a simulated agent for production line monitoring in smart factories. The PSR policies partition the query space into authorized (e.g., read-only access) and unauthorized (e.g., data modification) categories, permitting only the former. Following the main experiments, for each of these two agents, we also prompt Qwen3-MAX [@qwen3max] to construct 300 benign queries and 100 policy-violating queries given their PSR policies. The results in Table ¿tbl:tab:Fine-grained95Policies? show that PVDetector still achieves very low FNRs (\(0\%\)) and is superior to the main baselines in new policy settings, demonstrating that it is promising for adaptation to a broader range of practical scenarios with fine-grained policies.

6.2.3 Efficiency and Practicality.↩︎

We evaluate the average per-sample detection time of PVDetector and the baselines on Ignore attack instances generated from Alpaca dataset. The target agent is RecipeMaster with Llama3.1 (8B). This experiment is conducted on an NVIDIA A800 GPU. Table 1 shows that PVDetector requires the minimum time (\(0.1059\)s) compared with other training-free baselines. Furthermore, PVDetector eliminates training requirement and extra models, making it easily deployable in real-world environments.

6.2.4 Generalization to Vision-Language Models↩︎

To explore the generalization of our method to VLMs, we construct two VLM-based RecipeMaster agent instances by replacing the backend LLMs with Qwen2.5-VL and Phi-3.5-vision. Figure 4 illustrates how PVDetector is applied to VLM agent settings. For test data, we adopt a subset of the one used in Table ¿tbl:tab:main95result?: 200 ID queries and 100 OOD queries from the Alpaca dataset. We further employ Qwen3.5-Plus [@qwen3.5] to generate a relevant image for each of these queries, yielding multimodal evaluation data. The attack settings are consistent with our main experiments. The results in Table ¿tbl:tab:generalization95VLMs? show that PVDetector achieves low FNRs (\(0\%\)–\(2.5\%\)) on both VLMs, demonstrating its promising generalization capability on multimodal settings.

6.2.5 Transferability of PV Concepts.↩︎

As described in Section 4.2, we create contrastive pairs tailored to a specific agent to extract PV concepts. To analyze the cross-role transferability of these concepts, we conduct experiments under different transfer settings. Specifically, we evaluate the detection performance on a target agent using PV concepts extracted from contrastive pairs tailored to a source agent. The evaluation results on Alpaca dataset are presented in Table ¿tbl:tab:Transferability95of95PV95Concepts?. Although the transferability of PV concepts varies across different LLMs, PVDetector remains effective in detecting most attacks, achieving an average FNR of 3.57% under 4 transfer settings. This indicates that PV concepts capture the underlying semantic features of policy-violation.

6.2.6 Effect of Key Layers.↩︎

We compare the original setting w/ Key (using selected key layers) against three alternatives: (1) w/o Key (excluding key layers), (2) w/ All (using all layers), and (3) w/ L-Half (using the latter half of the layers). This analysis is conducted on RecipeMaster agent with Llama3.1 (8B). The results on the Alpaca dataset are shown in Table ¿tbl:tab:Effect95of95Key95Layers?. The original setting outperforms the others on FPR (achieving \(0.85\%\)).

6.2.7 Variations of Contrastive Prompt Pairs.↩︎

To study PVDetector’s robustness against the changes of contrastive prompt pairs, we evaluate its performance of using different data sources of ID and OOD queries with all other settings fixed. Specifically, we construct six groups of contrastive prompt pairs for PV concept extraction. The ID queries were generated by prompting three different powerful models (Qwen3.5 [@qwen3.5], GPT5.2 [@openai_gpt5], Gemini3 [@Gemini3]), while the OOD queries were collected from two public datasets. This analysis is conducted on RecipeMaster with Llama3.1 (8B). The results on the Alpaca dataset are shown in Table ¿tbl:tab:variations95contrastive95prompt95pairs?. PVDetector maintains <\(2\%\) FNR across most settings, indicating its resilience against variations of contrastive prompt pairs.

6.2.8 Case Study.↩︎

We provide a case study to demonstrate how PI attacks affect the PVS of policy-violating queries in the hidden space. We collect 10 benign queries, 4 policy-violating queries and the corresponding attack instances. Figure 5 shows their PVS scores at the layer 31 of Llama3.1 (8B). The attacks are conducted on the policy-violating queries; we use the same color to indicate this correspondence. We can see that the attacks reduce the PVS score of the original policy-violating query, accompanied by the generation of unintended outputs. However, the PVS scores of these attacks remain consistently higher than those of benign queries. This case highlights PVDetector’s effectiveness in detecting PI attacks.

6.2.9 Performance against Adaptive Attacks.↩︎

We test PVDetector against a strong adaptive adversary who has full knowledge of PVDetector’s details. Specifically, we consider a GCG-based adaptive attack that jointly minimizes the PVS scores to bypass detection while maximizing the attack success rate. Formally, we have the following loss function for the adaptive attack: \[L(Q_v,S_a,T_a) = L_a(T_a, f_\theta(Q_v,S_a)) + \beta \cdot S_{\text{PVS}}(Q_v,S_a),\] where \(Q_v\) denotes the policy-violating query, \(S_a\) denotes the adversarial string to be optimized, \(T_a\) denotes the target output of the attack, \(f_{\theta}(\cdot)\) represents the LLM inference, \(L_a(\cdot)\) is the standard GCG loss, \(S_{\text{PVS}}(\cdot)\) represents the computation for the aggregated PVS score, and \(\beta\) is a hyperparameter to balance the two terms. We conduct this adaptive attack on 50 OOD queries from the Alpaca dataset on three different values of \(\beta\). The results in Figure 6 show that PVDetector maintains a low FNR (\(0\%\)), while the ASR of this adaptive attack decreases with increasing \(\beta\), even falling below \(5\%\). This shows its resilience against adaptive attack attempts.

7 Conclusion↩︎

In this work, we uncover the latent PV concepts within the activation space of LLMs. These concepts reflect the intrinsic awareness of recognizing policy violations, which persists even when LLMs are subjected to PI attacks. Based on this finding, we propose PVDetector, a training-free framework that detects PI attacks by measuring hidden-state alignment with these concepts during LLM inference. Experimental results show that PVDetector is capable of effectively detecting various PI attacks, outperforming existing baselines.

8 Additional Experimental Results↩︎

8.1 Results on TripPlanner Agent↩︎

To further validate the generality of PVDetector, we conduct additional experiments on the TripPlanner agent. The results are summarized in Table ¿tbl:tab:results95on95TripPlanner?. It can be observed that our method still outperforms most baselines. Although AttentionTracker [@hung2025attention] achieves marginally better overall detection performance than PVDetector on Llama3.1 (8B), it exhibits a high FNR (reach up to 48.67% in some cases) on Qwen2.5 (7B), suggesting limited generalization across different models. These results show that PVDetector works well across different scenarios and LLMs.

8.2 Distribution of Violation-Aware Layers↩︎

Figure 7 shows the mean PVS differences between policy-violating samples and benign samples across all layers in different LLMs. The target agent is RecipeMaster. We can see that the PVS differences exhibit a consistent upward trend with increasing layer depth across all evaluated models, indicating a strengthened representation of policy-violating semantics in deeper layers. Notably, while Llama3.1 (8B) sustains this upward trajectory through the final layer, the other two assessed models demonstrate a slight reduction in divergence at the output layer. This attenuation is likely attributed to a shift in representational priorities: as models prioritize response generation in the last layer, attention originally allocated to the semantic analysis of policy constraints is reallocated toward generating coherent and contextually appropriate outputs.

8.3 Performance under Image-Based Attacks↩︎

For a comprehensive assessment, Section 6.2.4 includes a VLM evaluation of PVDetector, which suggests that PV concepts also persist and are recognizable in the latent space of multimodal models. Here, we further evaluate PVDetector on detecting those images with embedded malicious commands. Following the visual PI setting in GHVPI [@kimura2024empirical], we embed adversarial prompts (e.g., “Ignore previous instructions”) together with policy-violating queries into images. To ensure consistency, we use the same adversarial prompts and multimodal evaluation data as in Section 6.2.4. As reported in Table ¿tbl:tab:img95attack95detect?, PVDetector maintains very low FNRs (0%) on both VLMs, demonstrating that PV concepts remain detectable even when malicious instructions are embedded in images. This suggests that the proposed activation-space mechanism naturally extends to image-based PI attacks, further highlighting its relevance to multimedia applications.

9 Experimental Details↩︎

To ensure reproducibility, the temperature parameter of each LLM is set to 0.0 and the seed for the random number generator is fixed in our experiments. Our experiments are conducted on a machine equipped with an Intel(R) Xeon(R) Gold 5318Y CPU, 256GB RAM, and three NVIDIA A800 GPUs.

9.1 The Mapping of Model Code↩︎

We conduct evaluations on the Qwen2.5 family [@qwen2.5] (including two LLMs and one VLM) from Alibaba, Llama-3.1-8B-Instruct [@grattafiori2024llama] from Meta, and Phi-3.5-vision-instruct [@abdin2024phi3] from Microsoft. Table 2 provides the mapping of the model codes to the model names on HuggingFace¹.

9.2 Details of Datasets↩︎

In-Domain Query Generation. The styles of generated ID queries include open-ended questions and multiple-choice questions, with character lengths ranging from 30 to 500. The generation prompt for LLMs is shown in Figure 8. For illustration, we provide several ID query examples in Figure 11.

None

None

Out-of-Domain Query Generation. The Alpaca and MMLU datasets cover a wide range of domains and provide a rich source of questions beyond the scope of our agent scenarios. Therefore, we used the two datasets as the basis for constructing OOD queries. Specifically, we performed random sampling on each dataset and fed the samples to each agent. We then retained 300 rejected queries as the agent’s final OOD queries. For illustration, we provide several OOD query examples in Figure 12.

9.2.1 Evaluating Acceptance and Refusal Rates.↩︎

We define two evaluation metrics: ID acceptance rate \(\text{AR}_{\text{ID}}\) and OOD refusal rate \(\text{RR}_{\text{OOD}}\). \(\text{AR}_{\text{ID}}\) denotes the fraction of ID queries that the agent accepts, while \(\text{RR}_{\text{OOD}}\) denotes the fraction of OOD queries that the agent rejects. The results are shown in Table ¿tbl:tab:ID95acceptance95OOD95refusal?.

Recall that for both the Alpaca and MMLU datasets, we performed random sampling and retained only 300 rejected queries as the final OOD queries. Therefore, we estimate \(\text{RR}_{\text{OOD}}\) as the ratio of these 300 rejected queries to the total number of sampled queries. In this experiment, we adopt a string matching strategy to evaluate acceptance and refusal. Specifically, we append a suffix to each query with the instruction shown in Figure 9. If the agent’s response begins with “[No]”, we consider it a refusal; otherwise, it is classified as an acceptance.

9.3 Attack Setup↩︎

The introduction and setup of five attack methods are described as follows:

• Ignore Attack [@perez2022ignore]. In the ignore attack, an ignoring instruction is first appended, and then the injected instruction is placed in the subsequent content. An example is shown in Figure 13, where the adversarial prompt is highlighted in red.

 

• Fake Completion Attack [@willison2023delimiters]. This attack aims to trick the LLM into believing that the original system instruction has been finished. Figure 14 shows an example.

 

• Combined Attack [@liu2024formalizing]. This method combines the attack methods mentioned above. An example is shown in Figure 15.

 

• GCG Attack [@zou2023universal]. The Greedy Coordinate Gradient (GCG) is a strong optimization-based attack which optimizes an adversarial suffix to make the LLM output begin with a target prompt (e.g., “Sure, I can ...”). Figure 16 shows an example for this attack. In our implementation, we set the number of iterations to 300, the top-k value to 256, and batch size to 512. For all datasets, we adopt “Sure, I can help with that.” as the target prompt. Due to the high computational cost, we randomly sample 100 entries from each OOD dataset to generate GCG and Ig-GCG attack instances, while using the full OOD datasets for all other attack types.

 

• Ig-GCG Attack. This method, introduced in this paper, combines the Ignore attack and the GCG attack. An example is shown in Figure 17.

9.4 Evaluating Attack Success Rate↩︎

We evaluated the ASR of each PI attack against the LLM and VLM agents constructed in this work. The results are shown in Table ¿tbl:tab:ASR95Evaluation?. In this experiment, following [@zou2023universal], we adopt a string matching strategy to evaluate attack success by detecting refusal prefixes at the start of agent responses. Specifically, an attack is deemed unsuccessful if the agent’s response begins with any prefix in the refusal set, and successful otherwise. The refusal set is shown in Figure 10.

None

9.5 Baseline Setup↩︎

PPL [@alon2023detecting]. This method uses the average negative log-likelihood (NLL) of the user input as an anomaly score, where higher scores indicate more suspicious inputs. Specifically, NLL is calculated only over the tokens corresponding to the user message, focusing on semantic anomalies in the user’s input. We estimate the detection threshold from 100 randomly sampled benign examples, setting it to ensure an FPR of at most 1%.

ProtectAI [@protectai2024deberta]. This approach employs a fine-tuned prompt injection detector based on the DeBERTa-v3-base model, trained on English prompt injection data to classify inputs as either “SAFE” or “INJECTION”. In our experiments, we load the official model weights for deberta-v3-base-prompt-injection-v2² and perform inference using the Hugging Face pipeline, adopting the model’s original argmax prediction as the final decision.

Prompt-Guard-2 [@promptguard2]. This method uses a lightweight binary classifier based on Llama-Prompt-Guard-2-86M³, a DeBERTa-style sequence classifier, to detect jailbreak or prompt injection attacks in user inputs. The method assigns each raw user message a score corresponding to the predicted probability of the malicious class (in the range [0, 1]). Following the official recommendation in our evaluation, inputs are truncated to 512 tokens, and a default classification threshold of 0.5 is used; inputs with a malicious probability exceeding 0.5 are labeled as attacks.

DataSentinel [@liu2025datasentinel]. This method adopts a game-theoretic prompt injection detector that frames training as a minimax optimization: the inner maximization models adaptive attackers that jointly evade detection and mislead the backend LLM, while the outer minimization trains a dedicated detector LLM to distinguish clean from contaminated inputs under a known-answer protocol. At inference, a fixed detection instruction (e.g., “Repeat [secret key] once while ignoring the following text:”) is prepended to the user message. If the detector’s output lacks the secret key, the input is flagged as malicious. In our experiments, the inputs are formated as [instruction] + [user message]. For the detector, we employ the open-source QLoRA-fine-tuned Mistral-7B model released by the original authors⁴, with its binary prediction results (0 = benign query, 1 = prompt injection attack) used for quantitative evaluation.

JailGuard [@zhang2025jailguard]. This method detects prompt injection by analyzing the semantic consistency of model responses. It generates multiple semantically preserved variants of the user input (e.g., via word replacement), then uses a fixed system prompt to query the target LLM and collect corresponding responses. If the average semantic divergence among these responses exceeds a threshold, the input is classified as an attack. To achieve optimal detection performance, in our setup, we generate 8 variants per sample using the policy-based augmentation strategy. We estimate the detection threshold from 100 randomly sampled benign examples, setting it to ensure an FPR of at most 1%.

AttentionTracker [@hung2025attention]. This method leverages the distraction effect in prompt injection attacks: during an attack, certain critical attention heads shift focus from the system instruction to the malicious content. These vulnerable heads are identified using a small set of benign and adversarial examples. At inference, the input is flagged as malicious if the aggregated attention scores of these heads on the original instruction fall below a threshold. In our experiments, we use the open-source code⁵ and the default settings of AttentionTracker. Note that although AttentionTracker does not require additional model inference, it still incurs extra runtime overhead due to its reliance on attention weights⁶.

10 Interpretation Tokens↩︎

Figure 18 presents the complete tokens obtained from the projection results at the layers 23-32 of Llama3.1 (8B). To ensure interpretability, we eliminated all unreadable Unicode characters, retaining only interpretable tokens in the final set.

11 More Conversation Examples↩︎

We show more real-world conversation examples (see Figure 19) to reflect the traces of “resistance awareness” of LLMs.

12 Agent System Prompts↩︎

We construct 4 distinct agent scenarios: RecipeMaster, TripPlanner, CareerCounselor, and LineMonitor. For each agent scenario, we design a purpose-specific prompt, as illustrated in Figure 20, Figure 21, Figure 22, and Figure 23.

None

None

None

None

None

None

None

None

None

None

None

None

None

---

1. https://huggingface.co/models↩︎

2. https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2↩︎

3. https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M↩︎

4. https://github.com/liu00222/Open-Prompt-Injection↩︎

5. https://github.com/khhung-906/Attention-Tracker↩︎

6. This reliance restricts the model from utilizing optimized attention implementations (e.g., FlashAttention [@dao2022flashattention]), indirectly increasing the inference overhead.↩︎