1 Introduction↩︎

Although modern vision models achieve strong benchmark performance, these metrics do not guarantee that the features they use are semantically meaningful. Models often use spurious patterns for predictions; these models perform well on benchmarks, but fail to transfer over to real-world scenarios [1]. Ilyas [2] further argues that adversarial examples are features learned from these poorly generalized, spurious patterns. Additionally, both natural and model audits have shown that ImageNet contains harmful spurious features, class-associated frequency shortcuts, and systematic failure cases [3]–[5]. These findings suggest that ordinary, unpoisoned datasets may contain statistical structure that can be leveraged to cause model failures.

Moosavi-Dezfooli [6] demonstrated that a single adversarial direction can affect many images and transfer across different model architectures. Other work also shows that incorporating information about a target-class distribution can improve cross-model attacks [7]. These adversarial directions, however, are generally obtained through optimization against a single victim or surrogate model.

Similarly, frequency-domain attacks use structured frequency components to efficiently search for adversarial perturbations [8]. These adversarial directions are identified through responses from the attacked model. Information-geometric attacks similarly derive adversarial directions from Fisher geometry from a trained neural network [9]. Therefore, while prior work established that numerous different types of attack directions exist, these directions had to be identified through model-mediated signals.

Our paper seeks to answer what these prior papers do not: can target-specific failure directions can instead be constructed from the statistics of the original source dataset, without optimizing a victim or surrogate attack objective? Establishing these directions would connect dataset-level statistics to model sensitivities that are shared across multiple different architectures. We refer to these data source derived directions that induce target-specific failures as statistical adversaries. Our study is focused on four major questions:

1. Do models trained on ordinary, unpoisoned data exhibit sensitivity to adversarial directions derived only from the dataset?

2. Can these directions be constructed without victim-model gradients, queries to a model, or surrogate attack objectives?

3. Can the resulting directions induce target-specific, model-independent failures?

To answer these questions, we construct bounded perturbation directions solely from class-conditional ImageNet statistics and evaluate them on held-out, target-negative images across CNN and transformer classifiers. We compare raw class-mean directions against more controlled band-pass diagonal-whitened and Hellinger-motivated frequency-statistical directions, using random, low-pass, spectrum-matched, global-mean, and wrong-target controls. We find that selected source-statistical directions increase evidence for absent target classes beyond repeated matched controls. These effects primarily appear as calibrated false-positive inflation, rank movement, and top-(k) entry rather than consistent top-1 target takeover.

Our contributions are:

• We introduce and define statistical adversaries: directional perturbations derived only from the training data, which we evaluate through a held-out protocol with norm, frequency, and spectrum-matched controls.

• We propose source-only direction families based on band-pass diagonal-whitened class contrasts and Hellinger-motivated frequency statistics, requiring no victim-model gradients, queries, or surrogate attack optimization.

• We show that selected source-statistical directions induce target-specific false-positive inflation, rank movement, and top-\(k\) entry across CNN and transformer classifiers, while raw class-mean directions provide a negative baseline.

2 Related Works↩︎

2.1 Adversarial Attacks↩︎

The current literature on adversarial attacks comes from the observation that imperceptible perturbations can significantly alter the predictions of neural networks. Szegedy et al. were the first to demonstrate the existence of adversarial attacks [10]. Goodfellow et al. later found that vulnerability to adversarial attacks results from local linearity in multidimensional models, and introduced the Fast Gradient Sign Method (FGSM) [11]; however, Projected Gradient Descent (PGD) is considered the strongest type of attack and serves as the benchmark for evaluating model robustness [12].

More relevant to our evaluation settings are studies showing that a single perturbation can effectively attack a significant portion of the dataset for a given model [6], and in some cases even across different architectures by taking into account information about the target class distribution [7], by adjusting the statistics of intermediate features of a given class instead of directly attacking the logits [13] or by correlating decision boundaries with geometric properties [14]. Black-box attacks from a frequency-domain perspective [8], [15] and attacks based on information geometry from Fisher’s geometry of a trained network [9], further demonstrate that the shape of an effective perturbation is constrained by a structure that is not specific to any particular model. Nevertheless, these approaches determine perturbations using information from the model. In contrast, we explore whether this step can be completely eliminated and the direction determined solely based on the dataset.

2.2 Decision-Making Features of Neural Networks↩︎

Deep learning classifiers tend to focus on features that differ from human perception and are not related to the target class, a behavior that was observed long before the current era of artificial intelligence [16]. The best-known example is the recognition of wolves in snowy scenes [1] using local texture statistics instead of a global shape representation [17]–[19]. Similar examples occur also on ImageNet, where the classifier learned e.g. to recognize birds based on the background [20]. Attributes characterized by imbalance between groups may dominate predictions for minority subgroups [21]. Large-scale audits have revealed the existence of thousands of spurious features—correlated with classes and recognizable by humans—directly within the ImageNet dataset [3], [4], [22], [23], such as by adapting to different frequency ranges that may misdefine a class [5], [24], [25]. SilverLining showed that both spatial and spectral shortcuts can simultaneously emerge in real datasets and that removing such cues requires explicitly controlling frequency-domain confounders [26]. These audits are framed as issues related to bias and interpretability: they explain why the model fails in the case of an atypical example, but they do not show that the same spurious structure can be transformed into a controlled, repeatable perturbation of the data able to fool the model. This literature points to the frequency domain as a natural place to explore class-conditional structures and directly provides information on the statistics we use in our work to investigate potential gaps in the ImageNet dataset. Most work on shortcut learning focuses on analyzing failures under distribution shifts. We, however, investigate whether such statistical regularities themselves can serve as perturbation directions.

2.3 Backdoor Attacks and Natural Triggers↩︎

“Backdoor” attacks involve introducing triggering patterns during training to cause the model’s predictions to shift in the direction chosen by the attacker. BadNets was the first to demonstrate the effectiveness of “backdoor” poisoning [27]. Subsequent work introduced “clean-label” attacks [28], hidden triggers [29], and semantic backdoors [30]. However, poisoned examples leave a detectable spectral trace in feature space [31], although that study uses them to detect an injected trigger rather than to rule out naturally occurring triggers, which is the focus of our research. Ordinary, naturally photographed objects can act as triggers when associated with a target label [32], and naturally occurring reflection patterns on glass or water can serve the same function [30]. The most important findings come from [33], [34], showing that normally trained, unpoisoned models already contain input transformations resembling triggers, which the authors refer to as natural backdoors. Our work shares the main thesis of this article, but differs in terms of mechanism and evaluation: instead of reverse-engineering a trigger for a specific, trained model, we derive a potential direction based solely on source-class statistics, without access to any victim model, and then verify whether this single, model-independent direction reproduces backdoor-like behavior across several held-out architectures at once.

The literature on dataset bias shows that ImageNet exhibits a structure that can be exploited; the literature on frequency bias narrows down the range in which this structure typically occurs; the literature on transfer attacks shows that a single direction can generalize to different models; and the literature on natural backdoors shows that models not subjected to poisoning may already contain vulnerabilities resembling triggers. To the best of our knowledge, no previous work combines all four observations into a single process that (i) derives a potential direction exclusively from class-conditional source statistics, (ii) rules out trivial explanations for this effect using norm-matched control groups, frequency, and false targets - not just a pure baseline-and (iii) evaluates the remaining directions without fine-tuning or access to queries, on architectures where that direction was never built or tested during search. This is a gap that Statistical Adversaries are intended to fill.

3 Problem Formulation↩︎

We construct universal targeted perturbations from training-set statistics alone: a single perturbation is built per target class with no access to the victim models at construction time, and is then evaluated for its effect on a family of pretrained classifiers. Figure 1 summarizes the source statistics used to build these directions.

3.1 Setup↩︎

3.1.0.1 Images and labels.

Images are \(\boldsymbol{x}\in\mathcal{X}=[0,1]^{C\times H\times W}\) with \(C=3\) channels and \(H=W=224\), giving flattened dimension \(d=CHW\). Class labels are drawn from \(\mathcal{Y}=\{1,\dots,K\}\) with \(K=1000\).

3.1.0.2 Source dataset.

Perturbations are functions of a source training set \(\mathcal{D}_{\mathrm{tr}}=\{(\boldsymbol{x}_i,y_i)\}_{i=1}^{N}\) alone; we write \(\mathcal{D}_{\mathrm{tr}}^{c}\) for its class-\(c\) subset and the number of training samples \(n_c=\lvert\mathcal{D}_{\mathrm{tr}}^{c}\rvert\). Evaluation uses a disjoint validation set \(\mathcal{D}_{\mathrm{val}}\).

3.1.0.3 Victim models.

We attack a fixed family of \(M\) pretrained classifiers \(\{f^{(m)}\}_{m=1}^{M}\), where each \(f^{(m)}\colon\mathcal{X}\to\mathbb{R}^{K}\) returns a vector of \(K\) class logits. For each target \(t\), evaluation is restricted to the target-negative validation images \[\mathcal{V}_t= \{\boldsymbol{x}\in\mathcal{D}_{\mathrm{val}}:y(\boldsymbol{x})\neq t\},\] so movement toward \(t\) is an induced false positive rather than a correct prediction.

3.1.0.4 Perturbation model.

For each target \(t\) we build a single perturbation \(\boldsymbol{\delta}_t\in\mathbb{R}^{d}\), apply it additively, and clamp the result to the image range under an \(\ell_\infty\) budget \(\varepsilon\): \[\begin{align} \tilde{\boldsymbol{x}} &= \operatorname{clip}_{[0,1]}(\boldsymbol{x}+\boldsymbol{\delta}_t), \\ \lVert\boldsymbol{\delta}_t\rVert_\infty &\le\varepsilon,\qquad \varepsilon\in\{8,16,32\}/255 . \label{eq:budget} \end{align}\tag{1}\] Construction is model-free: \(\boldsymbol{\delta}_t=\mathcal{A}(\mathcal{D}_{\mathrm{tr}},t)\) with \(\partial\mathcal{A}/\partial f^{(m)}\equiv0\), so it uses no gradients and issues no queries to any \(f^{(m)}\). The same \(\boldsymbol{\delta}_t\) is reused across every image in \(\mathcal{V}_t\) and across all \(M\) models, making the attack universal in both inputs and models.

3.2 Evaluation metrics↩︎

Our headline metric treats each \(f^{(m)}_t\) as a one-vs-rest detector for class \(t\). On the clean target-negative images we fix a threshold \(\tau^{(m)}_t\) at the \((1-\alpha)\) empirical quantile of the clean target logits, with \(\alpha=0.05\), and then hold that threshold fixed after perturbation.The resulting false-positive-rate shift is \[\begin{align} \Delta\mathrm{FPR}_\alpha &= \mathbb{E}_{\mathcal{V}_t} \mathbb{1}\{f^{(m)}_t(\tilde{\boldsymbol{x}})>\tau^{(m)}_t\} -\alpha, \\ \mathrm{lift} &= 1+\Delta\mathrm{FPR}_\alpha/\alpha . \end{align}\] We report \(\Delta\mathrm{FPR}_\alpha\) in percentage points and as extra false fires per 1,000 images. Secondary metrics record the target-logit shift, the target-rank improvement, and targeted ASR@1/5. Target-rank and ASR definitions are provided in Appendix 9.1, and inference rules are provided in Appendix 9.4.

4 Source-Statistical Adversary Construction↩︎

Every perturbation is produced as follows: (i) form a target-specific direction from first- and second-moment statistics of \(\mathcal{D}_{\mathrm{tr}}\); (ii) optionally reshape it with a linear operator \(\mathcal{O}\) that retains only part of the signal; and (iii) project the result to the \(\ell_\infty\) budget. Figure 2 illustrates the full construction pipeline.

4.1 Source-statistical construction↩︎

4.1.0.1 Class means.

From the class subsets of \(\mathcal{D}_{\mathrm{tr}}\) we form the class, global, and complement means \[\begin{align} \boldsymbol{\mu}_c &=\frac{1}{n_c}\sum_{\boldsymbol{x}\in\mathcal{D}_{\mathrm{tr}}^{c}}\boldsymbol{x}, & \boldsymbol{\mu} &=\frac{1}{N}\sum_c n_c\boldsymbol{\mu}_c, \\ \boldsymbol{\mu}_{\bar{t}} &=\frac{1}{N-n_t}\sum_{c\neq t}n_c\boldsymbol{\mu}_c . \end{align}\] The raw class-contrast direction is the difference between the target and non-target means, \[\boldsymbol{g}_t=\boldsymbol{\mu}_t-\boldsymbol{\mu}_{\bar{t}}.\] To prevent a few high-variance coordinates from dominating, we form the diagonally whitened contrast \[\boldsymbol{w}_t=\boldsymbol{g}_t\oslash(\boldsymbol{s}+\eta), \qquad s_j=\Bigl(\tfrac{1}{N}\textstyle\sum_i(x_{i,j}-\mu_j)^2\Bigr)^{1/2}, \label{eq:whiten}\tag{2}\] where \(\oslash\) denotes coordinatewise division, \(\boldsymbol{s}\) is the vector of per-coordinate standard deviations over \(\mathcal{D}_{\mathrm{tr}}\), and \(\eta>0\) stabilizes low-variance coordinates.

4.1.0.2 Operator family and projection.

A perturbation family is defined by applying an operator \(\mathcal{O}\) to either \(\boldsymbol{g}_t\) or \(\boldsymbol{w}_t\). The final perturbation is obtained by scaling the resulting direction \(\boldsymbol{v}_t\) to the budget, \[\boldsymbol{\delta}_t = \Pi_\varepsilon(\boldsymbol{v}_t) = \varepsilon\,\frac{\boldsymbol{v}_t}{\lVert\boldsymbol{v}_t\rVert_\infty}, \qquad \lVert\boldsymbol{\delta}_t\rVert_\infty=\varepsilon . \label{eq:proj}\tag{3}\] The confirmation experiments focus on FFT-Hellinger and bandpass diagonal-whitened directions. Appendix 10 gives the full operator definitions and the complete perturbation-family table.

5 Experimental Protocol↩︎

We test whether class-conditioned statistics estimated from ImageNet can be converted into universal perturbations that increase false positives for a chosen target class. The perturbations are constructed without using victim-model gradients, model-specific optimization, or image-specific updates. Our primary outcome is target-specific false-positive-rate (FPR) inflation. We report targeted ASR@1 and ASR@5 as secondary outcomes.

To determine whether an observed effect is specific to the proposed class-conditioned direction, we compare it with five controls: Gaussian-random noise, lowpass-random noise, spectrum-random noise, the global ImageNet mean, and a direction constructed for the wrong target. Each control is evaluated on the same images and under the same perturbation budget as the corresponding proposed direction. The perturbation budget limits the largest change made to any individual RGB channel value.

5.1 Data and Evaluation↩︎

5.1.0.1 Source-statistic construction set.

We compute the source statistics from all 1,281,167 images in the ImageNet-1K training split. The split contains 1,000 classes, with between 732 and 1,300 training images per class. We use the training split only to estimate the statistics from which the perturbations are constructed. We do not train or fine-tune any of the victim models, and we do not compute FPR, ASR, or attack-success results on the training images.

ImageNet-1K images do not have consistent dimensions. Thus, we resized each image using bicubic interpolation such that its shorter side was 256 pixels. We then center-cropped the image to \(224\times224\) and converted to an RGB tensor with values in \([0,1]\). From these images, we calculate per-class and global mean images, first and second moments at each pixel location and color channel, and summaries of how image energy is distributed across spatial frequencies.

5.1.0.2 Candidate selection and validation sets.

The ImageNet-1K validation split contains 50,000 images, with exactly 50 images from each class. Candidate discovery requires searching over target classes, perturbation constructions, and perturbation budgets. Evaluating the final candidates on the same images used to select them would therefore overestimate their performance.

Thus, we shuffle the 50 validation images within each class using seed 17 and divide the resulting ordering into non-overlapping slices. Positions 0–4 form the concept-check slice, positions 5–14 form the candidate-validation slice, and positions 15–24 form the confirmation slice. The candidate panel is fixed before the confirmation slice is evaluated, and only results from this slice are used for the headline confirmation analysis.

5.1.0.3 Frozen confirmation panel.

The confirmation panel contains 11 target–construction–budget combinations evaluated on all four victim models, giving 44 model-level evaluations. Twenty-four correspond to FFT-Hellinger and twenty correspond to bandpass diagonal-whitened mean. Since the panel was assembled during the development stages, the reported results summarize these selected candidates rather than every ImageNet target.

5.1.0.4 Statistical tests and Benjamini–Hochberg correction.

We test paired target-threshold crossings with a one-sided exact binomial test, equivalently a one-sided exact McNemar test, and apply Benjamini–Hochberg correction across the 44 confirmation cells. This testing protocol is detailed more thoroughly in Appendix 11.

5.2 Victim Models and Controls↩︎

5.2.0.1 Victim models.

We evaluate four pretrained ImageNet classifiers: ResNet-50, ConvNeXt-Tiny, ViT-B/16, and Swin-T. All models use the default pretrained weights provided by Torchvision. Their parameters remain frozen, and inference is performed in evaluation mode. The same target-specific perturbation is evaluated on every model.

5.2.0.2 Perturbation application

We add our perturbations in the RGB image space prior to ImageNet normalization. After adding the perturbation, pixel values are clipped to the valid RGB range of \([0,1]\). We then apply standard ImageNet normalization. Our perturbation budgets, \(\ell_\infty\) are \(8/255\) and \(16/255\). No single RGB value for a given pixel may change more than the perturbation budget.

5.2.0.3 Controls.

Each proposed direction is compared with controls evaluated on the same target class, target negative images, victim model, and perturbation budget. The Gaussian-random control uses 10 random pixel-space noise directions. The lowpass-random and spectrum-random controls use 30 seeds each. The global-mean and wrong-target controls are deterministic.

3.6pt

6 Results↩︎

We first report whether raw class averages are recognizable to the models and which frequency components of the mean-based directions appear useful. We then report the held-out confirmation results on the frozen candidate panel. We follow this with comparisons on the matched controls in addition to a breakdown by construction, architecture, and secondary attack metrics. Figure 3 shows representative confirmation-slice examples used to illustrate these aggregate effects.

The raw class-average images are rarely classified as their own class. Across the four victim models, own-class top-1 accuracy ranges from 1.1% to 2.5%, and own-class top-5 accuracy ranges from 4.2% to 5.7% . The median own-class rank is also far from the top of the prediction list, ranging from 212.5 to 282.0. This suggests that the final effect is not explained by class averages simply looking like recognizable examples of their corresponding classes. Instead, the useful signal appears to require the transformations described in Section 4, such as class contrasts, whitening, and frequency selection.

6.1 Proposed Directions Increase Target-Specific FPR↩︎

The frozen confirmation panel contains 11 target-construction-budget candidates evaluated on four models, for 44 model-candidate cells. Each cell is evaluated on 9,990 target-negative confirmation images, giving 439,560 condition-level image evaluations. These are condition-level evaluations rather than unique images, since the same validation image can appear under different targets, constructions, budgets, and models.

Our confirmation panel contains 22,000 targeted false-positives on our clean, unperturbed images. After applying the proposed directions, this increases to 42,589 target false-positive decisions. This is a net increase of 20,589 false positives, or 46.84 additional false positives per 1,000 target-negative images. In other words, the proposed directions increase target-specific FPR from 5.005% to 9.689%, a 1.94\(\times\) lift over the clean baseline (Table 1). The increase is positive in 43 of 44 cells, and 40 of 44 clean-to-perturbed comparisons remain significant after Benjamini–Hochberg correction.

The target-specific increase is not accompanied by a general increase in false positives for arbitrary wrong classes. The generic any-wrong-class FPR decreases from 5.005% to 4.452%. Figure 4 provides a qualitative view of one such target-directed change.

6.2 Controls Explain Some, but Not All, of the Effect↩︎

The matched controls demonstrate several findings with our controls. First, Gaussian-random noise does not reproduce the effect. Its average perturbed FPR is 4.792%, which is slightly below our clean baseline. Our proposed directions exceed the Gaussian-random mean in 43 of 44 cells. Because this control has only 10 seeds, the smallest possible empirical \(p\)-value is \(1/11=0.0909\). Therefore, our one-sided empirical randomization test over control seeds cannot produce \(p<.05\) for this control.

Lowpass-random perturbations produce some false-positive inflation, raising FPR to 5.941%. The proposed directions remain larger by 3.748 percentage points on average and exceed the lowpass-random mean in 37 of 44 cells. Thirty-five of those comparisons remain significant after Benjamini–Hochberg correction. While this shows that smooth low-frequency noise can move target scores, its shift is relatively negligible and it significantly underperforms our proposed method.

Spectrum-random is the strongest control. It raises FPR to 7.681%, which means that matching the proposed direction’s frequency magnitude accounts for part of the effect. Even so, the proposed directions remain larger on average by 2.008 percentage points and exceed the spectrum-random mean in 37 of 44 cells. Although 21 comparisons have raw \(p\leq.05\), none survive Benjamini–Hochberg correction. Therefore, we treat spectrum matching as an important partial explanation as opposed to a ruled-out null.

The deterministic controls are much weaker. The global-mean control barely outperforms the clean baseline, raising FPR only to 5.207%. The wrong-target control raises FPR to 6.209% over the 40 valid cells, but the proposed direction still exceeds it in 32 of those 40 cells. This indicates that target identity matters. Simply applying another class-statistical direction is not equivalent to applying the direction for the intended target. Table 2 summarizes these matched-control comparisons.

4pt

The Gaussian-random test has minimum attainable \(p=1/11=.0909\). The lowpass-random and spectrum-random tests have minimum attainable \(p=1/31=.0323\). \(^\dagger\)

Four unavailable wrong-target comparisons are excluded.

4pt

6.3 FFT-Hellinger and Bandpass-Whitened Directions Both Contribute↩︎

FFT-Hellinger and bandpass-whitened directions play different roles in the confirmation panel. FFT-Hellinger is the frequency-profile construction motivated by the class-level frequency-energy summaries. It is positive in all 24 of its cells, increases FPR from 5.005% to 7.994%, and is Benjamini–Hochberg significant against clean in 21 of 24 cells. Bandpass-whitened is the stronger empirical construction in this panel. It increases FPR from 5.005% to 11.723%, is positive in 19 of 20 cells, and is Benjamini–Hochberg significant against clean in 19 of 20 cells.

The confirmation panel is not balanced across methods, targets, and budgets. Thus, direct method comparisons should use matched subsets. For FFT-Hellinger, targets 267 and 789 appear at both \(8/255\) and \(16/255\). Within these matched cells, the \(16/255\) result is larger for 7 out of 8 of them. For bandpass-whitened, target 789 appears at both budgets. The \(16/255\) result is larger for all four models. At \(16/255\), FFT-Hellinger and bandpass-whitened share targets 140, 207, and 789. On those matched cells, bandpass-whitened is larger for 10 out of 12 comparisons. These comparisons are descriptive because the candidate panel was selected during development. Table 3 summarizes these matched subsets.

6.4 The Effect Transfers Across Architectures↩︎

2.6pt

The same frozen perturbations are evaluated across all four architectures. The effect is positive for nearly every architecture-candidate pair, but its magnitude differs sharply by model. ResNet-50 and ConvNeXt-Tiny show smaller average increases, while ViT-B/16 and Swin-T are much more sensitive. On Swin-T, the mean target-specific FPR rises to 15.111%, a 10.106 percentage-point increase over clean. On ViT-B/16, it rises to 11.615%. The two convolutional models also show positive movement, but at a smaller scale (Table 4).

6.5 Frequency ablation↩︎

The exploratory frequency ablation shows that the effect is not driven by high-frequency residuals. Bandpass, raw mean, and zero-DC mean directions all increase target-specific FPR on average, while the highpass mean direction decreases it. Highpass also produces negative average target-logit and target-rank movement. We therefore carry forward the frequency-restricted bandpass direction, along with the FFT-Hellinger construction, into the confirmation experiment. We use this ablation is to motivate method selection. It is exploratory and not used as evidence for the final claims.

7 Conclusion↩︎

We show that source-statistical perturbations can induce target-specific false positives without model gradients, queries, or image-specific optimization. On the frozen confirmation panel, FFT-Hellinger and bandpass-whitened directions increase target-specific FPR from 5.005% to 9.689%, with positive movement in 43 of 44 model–candidate cells and 40 cells remaining significant after Benjamini–Hochberg correction. The effect is strongest for thresholded FPR, target logits, and target ranks.

The controls clarify what this result does and does not show. Gaussian random noise and global mean remain near the clean baseline, while lowpass-random produces a smaller effect. Spectrum-random is the strongest control and explains part of the behavior, but the proposed directions remain larger on average and exceed the spectrum-random mean in most cells. The current evidence therefore supports the existence of selected, transferable target-specific false-positive vulnerabilities induced by class-conditioned image statistics.

8 Prior-Work Comparison↩︎

Table 5 summarizes the closest prior-work families and how our setting differs from them.

2.6pt

Notes. Neuhaus et al. is a model-based spurious-feature discovery, benchmarking, and mitigation framework rather than an adversarial perturbation method. Its entries therefore describe the NPCA discovery pipeline, not the downstream models evaluated on Spurious ImageNet. “N/A” denotes settings with direct model access or no corresponding victim-query threat model.

9 Detailed Problem Formulation↩︎

9.1 Target rank and secondary metrics↩︎

We summarize the standing of a target class \(t\) on a single image by the integer \(r^{(m)}_t(\boldsymbol{x})\), defined as the position of \(t\) in the descending ordering of the logits \(f^{(m)}_1(\boldsymbol{x}),\dots,f^{(m)}_K(\boldsymbol{x})\), \[r^{(m)}_t(\boldsymbol{x}) = 1 + \sum_{k=1}^{K} \mathbb{1}\bigl\{f^{(m)}_k(\boldsymbol{x}) > f^{(m)}_t(\boldsymbol{x})\bigr\}. \label{eq:r}\tag{4}\] Here \(r^{(m)}_t(\boldsymbol{x})\in\{1,\dots,K\}\): it equals \(1\) when \(t\) is the top-scoring class and \(K\) when \(t\) is scored last, so a smaller \(r\) means a stronger standing for \(t\). A perturbation that promotes \(t\) decreases \(r\); the amount by which it decreases is the \(r\)-improvement \(\Delta r\).

For a model \(f^{(m)}\) and image \(\boldsymbol{x}\in\mathcal{V}_t\) with perturbed counterpart \(\tilde{\boldsymbol{x}}\) we record three per-image quantities, each averaged over \(\mathcal{V}_t\). The target-logit shift \[\Delta z^{(m)}_t(\boldsymbol{x}) = f^{(m)}_t(\tilde{\boldsymbol{x}})-f^{(m)}_t(\boldsymbol{x})\] is positive when the perturbation raises the score of \(t\). The \(r\)-improvement \[\Delta r^{(m)}_t(\boldsymbol{x}) = r^{(m)}_t(\boldsymbol{x})-r^{(m)}_t(\tilde{\boldsymbol{x}})\] counts how many positions class \(t\) climbs in the logit ordering, and is positive when \(t\) is promoted. The top-\(k\) targeted success \[\mathrm{ASR}^{(m)}_k = \mathbb{E}_{\mathcal{V}_t}\, \mathbb{1}\bigl\{r^{(m)}_t(\tilde{\boldsymbol{x}})\le k\bigr\}, \qquad k\in\{1,5\},\] is the fraction of target-negative images for which \(t\) is pushed into the top \(k\) predictions.

9.2 Calibrated FPR threshold↩︎

On the clean target-negative images we fix a threshold \(\tau^{(m)}_t\) at the \((1-\alpha)\) empirical quantile of the clean target logits, with \(\alpha=0.05\), \[\tau^{(m)}_t:\quad \mathbb{E}_{\mathcal{V}_t}\, \mathbb{1}\bigl\{f^{(m)}_t(\boldsymbol{x})>\tau^{(m)}_t\bigr\}=\alpha,\] so the clean false-positive rate equals \(\alpha\) by construction. We report \(\Delta\mathrm{FPR}_\alpha\) both in percentage points and as \(1000\,\Delta\mathrm{FPR}_\alpha\) extra false fires per \(1000\) images, with a paired nonparametric bootstrap confidence interval over \(\mathcal{V}_t\). To confirm that the effect is genuinely targeted, we verify that the untargeted any-wrong-class false-positive rate is not inflated.

9.3 Controls↩︎

We compare each \(\boldsymbol{\delta}_t\) against budget-matched controls that share its \(\ell_\infty\) norm but lack the target-class signal. All controls are projected to the same budget \(\varepsilon\) via 3 , and the spectral and statistical operators they invoke are defined in 10.

• random: \(\boldsymbol{v}\sim\mathrm{Unif}[-1,1]^{d}\), an unstructured baseline.

• lowpass-random: \(G_{\mathrm{low}}\) applied to a random direction, matching the smoothness of low-frequency perturbations without any class signal.

• spectrum-random: a phase-randomized version of \(\boldsymbol{\delta}_t\) that preserves its power spectrum, \(\mathcal{F}(\boldsymbol{v})=\lvert\mathcal{F}(\boldsymbol{\delta}_t)\rvert\odot e^{i\theta}\) with \(\theta\sim\mathrm{Unif}[0,2\pi)\) under Hermitian symmetry, isolating spectral magnitude from spatial structure.

• global / statistical: the dataset-wide mean image \(\boldsymbol{v}=\boldsymbol{\mu}\), a class-agnostic statistical direction.

• wrong-target: a perturbation \(\boldsymbol{\delta}_{t'}\) built for a different class \(t'\neq t\), testing target specificity.

9.4 Selection and inference protocol↩︎

For any metric \(\Theta\) we draw \(S\) seeds per control to form a null distribution \(\{\Theta^{\mathrm{null}}_s\}_{s=1}^{S}\), and summarize the position of the observed value \(\Theta^{\mathrm{main}}\) by \[\hat{p} = \frac{1+\lvert\{s:\Theta^{\mathrm{null}}_s\ge\Theta^{\mathrm{main}}\}\rvert}{S+1}, \qquad z = \frac{\Theta^{\mathrm{main}}-\bar{\Theta}^{\mathrm{null}}}{\operatorname{sd}(\Theta^{\mathrm{null}})}.\] The \(p\)-value is floored at \(1/(S+1)\), so \(z\) quantifies the margin by which the perturbation exceeds the null. Although construction is model-free, the reported triple \((t,\text{method},\varepsilon)\) is selected from a candidate grid by a model-dependent score; to guard against this selection we report a candidate as transferring only when its paired \(\Delta\mathrm{FPR}_\alpha\) confidence interval excludes \(0\) and it exceeds every null seed, simultaneously on all \(M\) models.

10 Detailed Construction Operators↩︎

Each operator takes an image-shaped direction \(\boldsymbol{v}\in\mathbb{R}^{C\times H\times W}\), where \(\boldsymbol{v}\) is either the contrast \(\boldsymbol{g}_t\) or its whitened form \(\boldsymbol{w}_t\), and retains a chosen part of it.

10.0.0.1 Spectral operators.

Let \(\mathcal{F}\) denote the two-dimensional discrete Fourier transform applied independently to each color channel, and \(\mathcal{F}^{-1}\) its inverse; thus \(\mathcal{F}\) maps a channel from pixel space into frequency space and \(\mathcal{F}^{-1}\) maps it back. For a frequency location \(\boldsymbol{\xi}\) we write its radial frequency \(\rho(\boldsymbol{\xi})=\lVert\boldsymbol{\xi}\rVert_2\) (the distance from the zero, or DC, frequency), and define the radial band mask \[\mathcal{M}_{[a,b]}(\boldsymbol{\xi}) = \mathbb{1}\bigl\{a\le\rho(\boldsymbol{\xi})\le b\bigr\}.\] Multiplying a spectrum by such a mask and inverting yields a band-limited operator; the low-pass, high-pass, and band-pass operators are \[\begin{align} G_{\mathrm{low}}(\boldsymbol{v}) &= \mathcal{F}^{-1}\!\bigl(\mathcal{M}_{[0,\rho_c]}\odot\mathcal{F}\boldsymbol{v}\bigr),\\ G_{\mathrm{high}}(\boldsymbol{v}) &= \mathcal{F}^{-1}\!\bigl(\mathcal{M}_{[\rho_c,\infty)}\odot\mathcal{F}\boldsymbol{v}\bigr),\\ G_{\mathrm{band}}(\boldsymbol{v}) &= \mathcal{F}^{-1}\!\bigl(\mathcal{M}_{[\rho_\ell,\rho_h]}\odot\mathcal{F}\boldsymbol{v}\bigr), \end{align}\] with cutoff \(\rho_c\) and band edges \(\rho_\ell<\rho_h\). Intuitively, \(G_{\mathrm{low}}\) keeps smooth, blurry structure, \(G_{\mathrm{high}}\) keeps sharp edges and texture, and \(G_{\mathrm{band}}\) keeps mid-frequency structure.

10.0.0.2 Channel and DC operators.

The per-channel mean (DC component) of a direction is \[\mathrm{DC}(\boldsymbol{v})_c = \frac{1}{HW}\sum_{h,w} v_{c,h,w}.\] From it we form the zero-DC operator, which removes each channel’s average and keeps only spatial pattern, \[\mathrm{ZDC}(\boldsymbol{v})_c = \boldsymbol{v}_c-\mathrm{DC}(\boldsymbol{v})_c\,\boldsymbol{1},\] and the channel-mean operator, which keeps only the overall per-channel color shift, \[\mathrm{CM}(\boldsymbol{v})_c = \mathrm{DC}(\boldsymbol{v})_c\,\boldsymbol{1},\] where \(\boldsymbol{1}\) is the constant all-ones image. Thus \(\mathrm{ZDC}\) discards a global color cast while preserving structure, and \(\mathrm{CM}\) does the opposite.

10.0.0.3 FFT–Hellinger operator.

This operator uses \(\mathcal{D}_{\mathrm{tr}}\) to identify the radial frequency bands whose share of spectral power differs most between target-class images and the dataset as a whole, and reweights the contrast \(\boldsymbol{g}_t\) to emphasize those bands. Reusing the Fourier transform \(\mathcal{F}\) and radial frequency \(\rho(\boldsymbol{\xi})\) above, we partition the frequency plane into \(B\) radial bands and write \(b(\boldsymbol{\xi})\in\{1,\dots,B\}\) for the band containing \(\boldsymbol{\xi}\). The radial power of an image \(\boldsymbol{x}\) in band \(b\) is its channel-averaged Fourier power summed over that band, \[P_b(\boldsymbol{x}) = \sum_{\boldsymbol{\xi}:\,b(\boldsymbol{\xi})=b} \frac{1}{C}\sum_{c=1}^{C} \bigl\lvert(\mathcal{F}\boldsymbol{x})_{c}(\boldsymbol{\xi})\bigr\rvert^{2}.\] Averaging \(P_b\) over the target-class images and over all images yields the two radial power profiles \(\bar{P}^{t}=(\bar{P}^{t}_b)_b\) and \(\bar{P}=(\bar{P}_b)_b\), which we normalize into probability vectors over the \(B\) bands, \[p^{t}_b=\frac{\bar{P}^{t}_b}{\sum_{b'}\bar{P}^{t}_{b'}}, \qquad p_b=\frac{\bar{P}_b}{\sum_{b'}\bar{P}_{b'}}.\] We measure the discriminativeness of each band through the Hellinger chart, i.e. the signed difference of the square-root profiles, mean-centered so the weights add no net energy, \[h_t(b) = \Bigl(\sqrt{p^{t}_b}-\sqrt{p_b}\Bigr) - \frac{1}{B}\sum_{b'=1}^{B}\Bigl(\sqrt{p^{t}_{b'}}-\sqrt{p_{b'}}\Bigr).\] A positive \(h_t(b)\) marks a band carrying relatively more target power than the dataset average and a negative one a band carrying relatively less; before centering, \(\tfrac{1}{2}\sum_b\bigl(\sqrt{p^{t}_b}-\sqrt{p_b}\bigr)^2\) is exactly the squared Hellinger distance between the two profiles. Broadcasting the band weights back to the frequency plane as \(W(\boldsymbol{\xi})=h_t\bigl(b(\boldsymbol{\xi})\bigr)\), the operator modulates the spectrum of the contrast, inverts, and removes each channel’s spatial mean, \[\boldsymbol{v}^{\mathrm{fft\text{-}hel}}_t = \mathrm{ZDC}\!\Bigl( \mathcal{F}^{-1}\!\bigl(W\odot\mathcal{F}\boldsymbol{g}_t\bigr) \Bigr).\]

Table 6 summarizes the perturbation families used in the candidate screen and confirmation panel.

11 Detailed Experimental Protocol↩︎

11.0.0.1 Validation slices.

Thus, we have to separate the datasets we use for candidate development and final evaluation. In order to achieve this in a random but still reproducible way, we shuffle the 50 validation images within each class using seed 17 and divide the resulting ordering into non-overlapping slices.

Positions 0–4 form the concept-check slice. They contain five images per class and 5,000 images in total. We use this slice to evaluate an initial set of candidates, check whether they produce target-directed changes in FPR, logits, and ranks. We also compare them with matched controls, and remove candidates that are ineffective or affected by obvious implementation artifacts.

Positions 5–14 form the candidate-validation slice. They contain 10 images per class and 10,000 images in total. We use this larger, disjoint slice to reevaluate the retained candidates with additional random-control seeds and to select the final target–construction–budget combinations.

Positions 15–24 form the confirmation slice, also containing 10 images per class and 10,000 images in total. The candidate panel is fixed before this slice is evaluated, and only results from this slice are used for the headline confirmation analysis.

Positions 25–49, are comprised of the remaining 25,000 validation images. They are not used in the current experiments.

11.0.0.2 Class-average prototype check.

Because several of our perturbation families are derived from class-level image averages, we first evaluate whether raw class averages contain any model-visible evidence of their own class. We construct one average image for each of the 1,000 ImageNet classes using all available training images from that class. We then evaluate each class-average image on all four victim models, for a total of 4,000 model–prototype evaluations and recorded whether the corresponding class appears in the model’s top-1 or top-5 predictions along with its logit and rank. Table 7 reports this class-average recognition check.

3.2pt

11.0.0.3 Exploratory frequency ablation.

Alongside the main runs, we ran a small screen on the mean-based perturbation. The four versions differed only in the frequency filter applied before normalization: no filtering for the raw mean, removal of the image-wide offset for zero-DC, retention of lower-to-middle spatial frequencies for bandpass, and removal of frequencies below the highpass cutoff for highpass. We used the screen to choose which mean-based variants to carry into the confirmation run.

This differs from the lowpass-random control used in the confirmation experiment. The frequency ablation still starts from the class-statistical direction; the random control starts from noise. Thus, the ablation asks which part of our direction matters, while the random control asks whether a non-class-statistical perturbation with similar smoothness can produce the same effect.

This experiment uses three target classes, ResNet-50 and ViT-B/16, and perturbation budgets of \(8/255\), \(16/255\), and \(32/255\). The resulting grid contains \(3\times2\times4\times3=72\) proposed-direction conditions. Each condition is evaluated on 1,998 target-negative validation images. We use this stage to identify which frequency band is carried forward into the confirmation experiment. Table 8 reports the resulting exploratory frequency ablation.

4pt

11.0.0.4 Broad candidate screen.

We conduct an initial screen over 100 target classes, consisting of five previously identified prototype candidates and 95 targets selected using a fixed random sample. For each target, we evaluate zero-DC mean, diagonal-whitened mean, bandpass diagonal-whitened mean, and FFT-Hellinger directions at \(8/255\) and \(16/255\). The screen uses ResNet-50, ViT-B/16, and one validation image per class. For every target, perturbation method, perturbation budget, and victim model, we generate one lowpass-random direction and one spectrum-random direction. This stage is used to reduce a broad target and method search to a smaller set of candidates.

11.0.0.5 Focused concept check.

Eight targets from the broad screen are subsequently used in the concept-check slice described above. We evaluate four candidate constructions, two perturbation budgets, and all four victim models, giving \(8\times4\times2\times4=256\) model-level proposed-direction conditions. Stochastic controls are evaluated with 10 seeds in this stage. We retain candidates that show positive target-FPR, target-logit, and target-rank movement, compare favorably with the lowpass-random and spectrum-random controls, transfer across more than one model family, and do not exhibit obvious implementation artifacts or severe corruption. Candidate selection at this stage is rule-guided rather than based on a preregistered scalar score.

11.0.0.6 Candidate validation.

The retained candidates are reevaluated on the larger candidate-validation slice using all four victim models. This stage focuses on FFT-Hellinger and bandpass diagonal-whitened mean at \(8/255\) and \(16/255\). Gaussian-random controls use 10 seeds per condition, while lowpass-random and spectrum-random controls use 30 seeds each. Global-mean and wrong-target controls are deterministic. Results from this stage are used to freeze the final target–construction–budget panel before the confirmation slice is evaluated. Table 9 lists the frozen panel used for the confirmation-slice evaluation.

11.0.0.7 Confirmation controls.

Each proposed direction is compared with controls evaluated on the same target class, target negative images, victim model and perturbation budget. We use 10 independently random sampled pixel spaced noise directions for the Gaussian-random control. The lowpass-random control uses 30 random directions smoothed towards low spatial frequencies to test if smooth random structure is sufficient. The spectrum-random control uses 30 phase-randomized directions with the same Fourier magnitude as the proposed direction, testing whether only matching the frequency content can explain the direction. The global-mean control uses ImageNet-wide statistics rather than target-specific statistics. The wrong-target control uses a source-statistical direction from a different target class.

There is no wrong target comparison for the bandpass-whitened \(8/255\) target-789 candidate because it is the only target in that construction–budget group. Thus, we omit the wrong-target control for those four model cells but retain it’s proposed direction and all of its other corresponding controls.

11.0.0.8 Statistical tests and Benjamini–Hochberg correction.

For each model–candidate pair, we compare how often an image crosses the target threshold after perturbation with how often it falls back below the threshold. A newly crossed example is below the target threshold when clean and above it after perturbation. A recovered example is above the threshold when clean and below it after perturbation. Under the null hypothesis that the perturbation has no directional effect on target-threshold crossings, these two outcomes are equally likely. We therefore use a one-sided exact binomial test, equivalently a one-sided exact McNemar test, and apply Benjamini–Hochberg correction across the 44 confirmation cells so that the expected fraction of false discoveries among the significant cells is controlled at 5% [40].

Because the panel comprises \(m=44\) model–candidate cells, some form of multiplicity adjustment is required. We adopt the Benjamini–Hochberg procedure, which controls the false discovery rate, in preference to family-wise schemes such as Bonferroni. The latter sacrifices considerable power across a panel of this size, while the former limits the expected fraction of erroneous rejections among the cells declared significant, which is the quantity of interest for a confirmatory panel. Ordering the one-sided \(p\)-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\), the procedure rejects every hypothesis up to the largest rank \(k\) satisfying \[p_{(k)} \leq \frac{k}{m}q, \qquad q=0.05.\] Tables 1, 2, and 3 in the main paper report the corresponding Benjamini–Hochberg-corrected counts and matched-control summaries; we do not repeat those tables here.

For stochastic controls, we compare the proposed direction with the control-seed distribution within the same cell. The empirical one-sided \(p\)-value counts the fraction of control seeds whose FPR change is at least as large as the proposed direction, with a standard plus-one correction. The number of control seeds also limits the smallest empirical \(p\)-value we can report. With 10 Gaussian-random seeds, the smallest value is \(1/11\). With 30 lowpass-random or spectrum-random seeds, the smallest value is \(1/31\). We apply Benjamini–Hochberg correction separately within the Gaussian-random, lowpass-random, and spectrum-random families. Because global-mean and wrong-target controls are deterministic, we report paired effect differences for those controls instead of seed-based \(p\)-values.

References↩︎