1 Introduction↩︎

Machine unlearning [1], [2] typically removes specified samples, classes, or concepts from a trained model while preserving performance on the remaining task [3]. Spurious-correlation settings raise a different target: not a class or attribute itself, but a learned relationship between the target label and an attribute. For example, Waterbirds correlates bird type with background, and CelebA-Blond correlates hair color with gender [4]–[6]. We refer to this evaluation setting as association unlearning: disabling a learned label–attribute shortcut while preserving the task label.

Evaluating association unlearning requires separating output behavior, representational readability, and functional use. Output metrics such as worst-group accuracy (WGA) measure whether the model currently avoids shortcut-driven errors, but not whether the shortcut remains encoded internally. Representation-level audits probe frozen features to test whether supposedly removed information remains readable [7], [8]. However, readability is not functional use: a feature can remain decodable without being used by the original classifier head.

Restoration-aware evaluation has recently exposed similar failures in class-unlearning settings, where models can satisfy output-level forgetting metrics while retaining recoverable class information  [7], [9], [10]. These works recover forgotten behavior using lightweight probes, relearning, or feature steering [7], [10]–[12]. However, class restoration tests recovery of a forgotten class, whereas association restoration tests reactivation of a learned label–attribute shortcut.

To fill this gap, we propose the Association Restoration Test (ART), a post-hoc feature-space diagnostic for functional restorability of label–attribute associations. ART estimates class-conditional association directions in frozen feature space, amplifies each example’s residual component along the corresponding direction, and applies the original classifier head to the modified feature to obtain the restored prediction. We use ART to audit visual shortcut-mitigation methods across Waterbirds, CelebA, SpuCoDogs, and a multiclass ISIC timestamp-artifact extension [4]–[6], [13]–[16]. We audit a spectrum of methods that reduce or suppress label–attribute shortcuts, including group-robust debiasing references, post-hoc classifier reweighting methods, and association-adapted unlearning mechanisms.

Our audit shows that output robustness, feature readability, and functional restorability can diverge. Shortcut attributes often remain readable from frozen features even when WGA improves. ART further separates readable-but-decoupled shortcuts from restorable shortcuts: some methods appear to decouple the classifier head from retained shortcut structure, whereas others leave associations that ART can reactivate, causing WGA drops and increased shortcut-consistent errors.

Overall, our contributions are:

• Association Restoration Test. We introduce ART, a post-hoc diagnostic for testing whether retained label–attribute associations remain functionally restorable under the original classifier head.

• Association-level restoration auditing. We extend restoration-aware evaluation from class-level unlearning to learned label–attribute shortcuts.

• Empirical evidence of evaluation divergence. Across visual shortcut benchmarks, we show that output robustness, feature readability, and functional restorability can diverge across shortcut-mitigation and association-adapted unlearning methods.

2 Related Work↩︎

2.1 Machine Unlearning and Association-Level Targets↩︎

Machine unlearning aims to remove specified information from a trained model without full retraining [1], [2]. In vision, standard unlearning targets are usually defined by a forget set at the sample, class, or concept level. Representative methods include negative-gradient forgetting, SCRUB, SalUn, and SSD [3], [17]–[19].

Beyond standard sample-, class-, and concept-level unlearning, recent work has begun to study unlearning-style interventions for spurious associations and shortcut dependence [20]–[22]. These settings motivate an association-level target: disabling a learned relation between the task label and a spurious attribute, rather than removing a sample, class, or attribute in isolation. Our work studies this target from an evaluation perspective, asking whether such relations are functionally disabled or merely suppressed.

2.2 Shortcut Mitigation for Label–Attribute Associations↩︎

Shortcut mitigation methods aim to improve robustness when labels are spuriously correlated with attributes. Representative methods include GroupDRO, DFR, and JTT, which use group-robust optimization, classifier retraining, or example reweighting  [4], [23], [24]. These methods reduce shortcut-driven behavior through robustness objectives or reweighting rather than specifying a relation-level forget target.

Other shortcut-mitigation approaches reduce reliance on spurious attributes through biased-model failure signals, inferred environments, or contrastive objectives [25]–[27]. These methods can reduce shortcut-driven errors, but output robustness alone does not determine whether the underlying label–attribute association remains encoded or functionally restorable. We therefore audit both shortcut-mitigation references and association-adapted unlearning variants.

2.3 Evaluating Unlearning Beyond Output Metrics↩︎

Standard unlearning evaluations focus on output behavior: class-unlearning methods are typically judged by forget-set and retain-set accuracy, while shortcut and association settings often use subgroup accuracy or worst-group accuracy [4], [23]. These metrics capture current prediction behavior, but they do not determine whether the target information remains encoded in the representation or can be functionally recovered.

Recent work has therefore examined unlearning beyond output metrics. Representational audits test whether supposedly removed information remains readable in hidden features. For example, Gao et al. use linear probes and nearest-class-center classifiers to show that forgotten-class information can remain separable after class unlearning [7]. Yu et al. evaluate remaining forget-set information through probe recovery, alignment, separability, and layer-wise analyses [8]. These works show that output forgetting does not imply feature-level removal. However, feature readability alone does not establish that the retained information remains usable by the model’s original classifier head.

Functional restoration methods provide a stronger test by applying lightweight recovery procedures and measuring whether forgotten behavior returns. Existing studies recover forgotten behavior through prototype relearning, post-hoc recovery from residual output signals, or inference-time feature steering  [10]–[12]. These approaches reveal that models can satisfy output-level forgetting metrics while retaining recoverable information. However, they remain largely class-centric: they primarily test whether a forgotten class or concept can be recovered.

Association unlearning raises a different restoration question: whether a retained label–attribute relation can be reactivated, rather than whether a forgotten class can be recovered. ART fills this association-level gap by auditing functional restorability under the original classifier head, while probes provide a companion measure of feature readability.

3 Association Restoration Test (ART)↩︎

ART is a post-hoc feature-space diagnostic for functional restorability of label–attribute associations. Given a trained classifier, ART finds class-conditional association directions in frozen feature space, gates unreliable directions, and restores each example’s residual component along the corresponding direction before applying the original classifier head. A strong restoration effect indicates that the association remains functionally connected to the classifier.

3.0.0.1 Setup.

In the binary setting, each example is a tuple \((x_i,y_i,a_i)\), where \(y_i\in\mathcal{Y}=\{y_1,y_2\}\) is the target label and \(a_i\in\mathcal{A}=\{a_1,a_2\}\) is the spurious attribute. Each subgroup is a label–attribute pair \((y,a)\). Let \(\pi(a)\) denote the label most associated with attribute value \(a\) in the biased training distribution. Pairs with \(y=\pi(a)\) are aligned, and pairs with \(y\neq \pi(a)\) are conflicting.

Let \(f=\psi\circ\phi\) be a trained model, where \(h_i=\phi(x_i)\in\mathbb{R}^d\) is the frozen representation and \(\psi\) is the classifier head. ART uses a labeled audit split to estimate association directions and gates, and a disjoint test split to evaluate restored predictions.

3.0.0.2 Find.

For each class \(c\), ART estimates a residual direction that separates the two attribute groups within that class. Conditioning on \(y=c\) removes the trivial between-class difference, so the remaining separation reflects within-class attribute structure.

Let \(\mu_c=\mathbb{E}[h\mid y=c]\) be the class mean, and let \(\bar\mu=\mathbb{E}[h]\) be the global feature mean, both estimated on the audit split. We define the global label subspace as \[S_y = \operatorname{span}\{\mu_c-\bar\mu: c\in\mathcal{Y}\}.\] In the binary case, this is equivalent to the one-dimensional span of the between-class mean difference. Let \(P_{S_y}\) denote the orthogonal projection onto this subspace. We center features within class \(c\) and apply a partial label-null correction: \[\tilde{h}_i^{(c)} = (I-\rho P_{S_y})(h_i-\mu_c), \qquad y_i=c.\] Here, \(\rho\in[0,1]\) controls the strength of label-null correction. We use \(\rho=0.5\) by default to reduce label-direction leakage while preserving shortcut signal.

ART then contrasts the two attribute groups inside class \(c\): \[\begin{align} u_c &= \operatorname{norm} ( \mathbb{E}[\tilde{h}^{(c)}\mid y=c,a=a_2]\\ &- \mathbb{E}[\tilde{h}^{(c)}\mid y=c,a=a_1] ), \end{align}\] where \(\operatorname{norm}(v)=v/(\|v\|_2+\epsilon)\). The direction \(u_c\) captures within-class attribute structure that ART later tests for functional restorability.

3.0.0.3 Gate.

To avoid restoring noisy or poorly supported directions, ART gates each candidate direction using its projected subgroup separation. On the audit split, we project residual features from class \(c\) onto \(u_c\): \[s_i^{(c)} = u_c^\top \tilde{h}_i^{(c)} .\] Let \(\mu_{c,j}^s\) and \((\sigma_{c,j}^s)^2\) be the mean and variance of \(s^{(c)}\) for subgroup \((c,a_j)\). We compute the separation score \(T_c\) between the projected subgroups by: \[T_c = \frac{ \left|\mu_{c,2}^{s}-\mu_{c,1}^{s}\right| }{ \sqrt{ \frac{1}{2} \left[ (\sigma_{c,1}^{s})^2 + (\sigma_{c,2}^{s})^2 \right] + \epsilon} } .\] Therefore, the gate is \[m_c = \mathbf{1} \left[ T_c > \tau \;\wedge\; \min\left(n_{c,a_1}, n_{c,a_2}\right) \ge n_{\min} \right],\] where \(n_{c,a_j}\) is the number of audit examples in subgroup \((c,a_j)\). The gate keeps only directions with sufficient projected separation and enough examples in both attribute groups. If \(m_c=0\), ART applies no restoration for class \(c\). This prevents noisy or poorly supported directions from being treated as restorable shortcuts.

3.0.0.4 Restore.

Finally, ART tests whether the residual association remains usable by the original classifier head. For each test example, ART selects the direction using its true label, \(c_i=y_i\), and amplifies the gated residual component: \[h_i'(\beta) = h_i+\beta\,m_{c_i}\,P_{U_{c_i}}(h_i-\mu_{c_i}), \qquad U_c=\operatorname{span}(u_c).\] Here, \(\beta\) controls restoration strength, and the projection extracts the example’s residual component along the estimated association direction. ART therefore amplifies shortcut structure already present in the representation rather than injecting an arbitrary attribute feature.

The restored prediction is computed using the original classifier head: \[\hat{y}_i'(\beta)=\psi(h_i'(\beta)).\] Therefore, ART asks whether this controlled feature-space restoration can bring back shortcut behavior.

4 Experimental Setup↩︎

4.0.0.1 Datasets.

We audit methods on three binary spurious-correlation benchmarks: Waterbirds, CelebA, and SpuCoDogs. We also evaluate a multiclass ISIC 2019 timestamp-artifact extension in Sec. 5.6. Table 1 summarizes the label–attribute semantics and biased training-set group counts.

3.2pt

4.0.0.2 Methods under audit.

All methods use an ImageNet-pretrained ResNet-50. We include ERM as the biased reference model and Balanced Retrain as a group-balanced reference trained from the ImageNet initialization. Because there is no standardized suite of association-unlearning baselines, we audit methods that target the same label–attribute shortcuts.

We audit shortcut-mitigation references GroupDRO, DFR, and JTT [4], [23], [24]. These are not unlearning algorithms, but they reduce shortcut-driven behavior through robust optimization, classifier retraining, or example reweighting. We also audit association-adapted unlearning variants A-NegGrad+, A-SCRUB, A-SalUn, and A-SSD [3], [17]–[19]. The prefix “A-” denotes an association-specific adaptation of the original unlearning mechanism. Instead of defining the forget target as a sample set or class, we define it through a class-conditional association objective that suppresses predictability of the spurious attribute from frozen features within each target class, while preserving task-label performance. A-NegGrad+ replaces the forget-set loss with this association loss; A-SCRUB keeps the teacher–student preservation term but uses the association loss as the adversarial forget term; A-SalUn computes saliency with respect to the association loss before applying masked updates; and A-SSD replaces forget-set importance with association-direction importance before selective dampening. Thus, A-* variants preserve the original mechanisms while changing the target to label–attribute association suppression.

4.0.0.3 Evaluation metrics.

We use representation probes and ART as complementary diagnostics: probes test feature readability, while ART tests functional restorability under the original classifier head.

Representational probes. We use two standard post-hoc probes on frozen penultimate features: a linear probe (LP) [28] and a nearest class-center classifier (NCC) [29]. LP trains a linear readout, while NCC predicts by nearest probe-class centroid. Both are trained after model training and leave the original model unchanged.

We report held-out probe accuracy for three targets: the global spurious attribute (a), the class-conditional attribute (a) within each target class (y=c), and the joint subgroup identity ((y,a)). The class-conditional attribute probe is our main readability diagnostic because association unlearning does not require globally erasing the attribute; it asks whether attribute structure remains within each label class.

ART restoration metrics. To evaluate functional restorability, we compare model behavior before and after applying ART. Our main output robustness metric is worst-group accuracy, \[\mathrm{WGA} = \min_{(y,a)\in\mathcal{Y}\times\mathcal{A}} \mathrm{Acc}(y,a).\] We also measure shortcut-consistent errors on conflicting groups. Let \[\mathcal{D}_{\mathrm{conf}} = \{(x_i,y_i,a_i)\in\mathcal{D}: y_i\neq \pi(a_i)\}.\] The conflict shortcut rate is \[\mathrm{CSR} = \frac{1}{|\mathcal{D}_{\mathrm{conf}}|} \sum_{(x_i,y_i,a_i)\in\mathcal{D}_{\mathrm{conf}}} \mathbf{1}[\hat{y}_i=\pi(a_i)] .\] Higher CSR indicates more shortcut-consistent errors on conflicting examples.

We measure the effect of ART using the WGA drop and CSR gain after restoration: \[\Delta_{\mathrm{WGA}} = \mathrm{WGA}_{\beta=0}-\mathrm{WGA}_{\beta}, \qquad \Delta_{\mathrm{CSR}} = \mathrm{CSR}_{\beta}-\mathrm{CSR}_{\beta=0}.\] Larger \(\Delta_{\mathrm{WGA}}\) or \(\Delta_{\mathrm{CSR}}\) indicates stronger functional restoration of shortcut behavior under the original classifier head.

4.0.0.4 Training and ART protocol.

Binary benchmark experiments use an ImageNet-pretrained ResNet-50 with a two-way classifier head; the ISIC extension uses the corresponding multiclass head. Checkpoints are selected by validation WGA, and results are averaged over five seeds.

ART is run post hoc on frozen penultimate 2048-dimensional features. Held-out training features are split into subgroup-stratified direction and gate sets; the test set is used only once to evaluate restored predictions. We use \(\rho=0.5\), \(\tau=0.5\), \(n_{\min}=20\), and \(\beta=2\) by default, and ablate these choices in Sec. 5.5.

5 Results↩︎

5.1 Initial Output Behavior↩︎

Before applying ART, we evaluate standard output behavior. Table 2 reports WGA and CSR on all three binary benchmarks before restoration. Some methods improve WGA or reduce CSR relative to ERM, indicating fewer shortcut-driven mistakes before restoration. However, these metrics only measure current behavior; they do not show whether the shortcut is removed or merely hidden but usable.

5.2pt

5.2 Feature Readability under Post-hoc Probes↩︎

We probe frozen penultimate features using LP and NCC readouts. Table 3 reports LP / NCC accuracies for three probe targets: the class-conditional attribute, the global attribute, and the joint group.

Across datasets, probes show little evidence of representational deletion. Class-conditional attribute scores remain close to ERM, with small standard deviations across methods. This indicates that the spurious attribute remains readable within each target class. Thus, the worst group accuracy improvements of the methods in Sec. 5.1 generally do not correspond to shortcut removal from feature space. However, this result alone does not imply functional failure: a readable shortcut may still be decoupled from the original classifier head.

3pt

5.3 Functional Restoration with ART↩︎

7pt

We apply ART to test whether residual shortcut associations are functionally restorable. Table 4 reports WGA drop and CSR gain at the default setting \((\beta=2,\rho=0.5)\).

Table 4 shows that ART separates low-restoration references from methods with functionally restorable shortcut associations. Balanced Retrain provides the strongest low-restoration reference, with small WGA drops and little or negative CSR gain. DFR and GroupDRO are often less vulnerable than the association-adapted unlearning variants, although they are not uniformly stable across all datasets. In contrast, A-NegGrad+, A-SCRUB, A-SalUn, and A-SSD frequently show large WGA drops together with large CSR gains, especially on Waterbirds and SpuCoDogs. Thus, many methods improve output behavior while leaving residual associations that ART can reactivate.

5.3.0.1 Qualitative illustration.

Fig. 2 shows representative stable and vulnerable cases. In feature space, A-NegGrad+ exhibits a larger ART-induced shift than Retrain. In Grad-CAM visualizations, ART increases saliency on shortcut-associated background regions for A-SalUn, whereas GroupDRO remains more focused on the object. These visualizations illustrate the quantitative WGA and CSR patterns.

5.4 Joint Interpretation of Readability and Restorability↩︎

Fig. 3 compares feature readability and ART restorability. The x-axis is class-conditional attribute probe accuracy; the y-axis is WGA drop at \(\beta=2\). We use this plot as a qualitative taxonomy rather than a hard decision rule. We treat probe accuracy above \(70\%\) as clear feature-space readability and a WGA drop above \(15\) points as a substantial restoration effect. Together, these guides divide the plot into four regions. Low probe accuracy and low ART vulnerability suggest representational deletion: the association is neither readable from the representation nor easily restored into model behavior. High probe accuracy but low ART vulnerability suggests functional decoupling: the association is still encoded in the features, but the classifier head cannot easily restore it. High probe accuracy and high ART vulnerability indicate restorable shortcuts: the association remains readable and can still be reactivated into shortcut-consistent predictions. Finally, low probe accuracy but high ART vulnerability indicates a possible probe miss, where the association is not captured by the probe but can still affect the classifier through the ART intervention.

Most methods fall in the high-probe region, indicating little evidence of representational deletion. DFR and GroupDRO are closer to functional decoupling in several settings, whereas A-NegGrad+, A-SCRUB, A-SalUn, and A-SSD often fall in the suppressed/restorable region. Thus, similar output improvements can reflect different internal states: some methods reduce the functional influence of retained shortcuts, while others preserve associations that ART can reactivate.

5.5 Sanity Checks and Ablations↩︎

5.5.0.1 Control perturbations.

Control perturbations test whether ART reflects association-specific restoration rather than generic feature-space disruption. We compare ART with controls that use the same operation but replace the association direction with shuffled or random directions. Table 5 shows that these controls produce near-zero WGA drops, while ART produces much larger drops across datasets. Thus, the effect depends on the estimated class-conditional association direction, not generic feature perturbation.

4pt

5.5.0.2 Hyperparameters.

Our default ART configuration uses \(\beta=2\) and \(\rho=0.5\). We use \(\beta=2\) as a restoration stress test because it makes stability differences clearly measurable. We set \(\rho=0.5\) as a partial label-null correction to reduce leakage from the global label direction without making the association estimate overly conservative.

We ablate both choices to test sensitivity to this default setting. Increasing \(\beta\) consistently strengthens ART restoration; for example, on CelebA, the mean WGA drop increases from \(4.2\) to \(32.2\) points as \(\beta\) increases from \(0.25\) to \(2\). In contrast, increasing \(\rho\) weakens restoration by removing more label-aligned variation before direction estimation; for example, on CelebA, the mean WGA drop decreases from \(38.7\) points at \(\rho=0\) to \(1.9\) points at \(\rho=1\). Thus, \(\rho=0.5\) provides a useful tradeoff between suppressing label leakage and preserving shortcut signal.

5.5.0.3 Design choices.

We ablate three design choices in the ART find–gate–restore pipeline, as summarized in Table 6. All entries are averaged over non-retrain audited methods.

For Find, we compare the proposed class-conditional association direction with a global attribute direction and a label direction. The class-conditional ART direction gives the clearest shortcut-restoration signal: it produces large WGA drops while also yielding the largest CSR gains across all three datasets. In contrast, the label direction can sometimes reduce WGA, but it produces little or no CSR increase, indicating that it does not specifically restore shortcut-consistent errors.

For Gate, we compare the default gated version with an ungated variant. Removing the gate makes the intervention more aggressive and can increase the WGA drop, but it does not consistently increase shortcut-consistent restoration. We therefore use the gate as a conservative specificity control that filters weak or poorly supported directions.

For Restore, we compare true-label selection with predicted-label selection. Predicted-label ART remains effective, but it is consistently weaker because misclassified examples can be routed to the wrong class-conditional direction. Thus, true-label selection provides a cleaner controlled diagnostic, while the predicted-label variant shows that the restoration effect does not depend entirely on oracle label routing.

4.5pt

5.6 Extension to Multiclass Associations↩︎

We also evaluate ART on a multiclass ISIC2019 skin-lesion classification setting [14]–[16]. We create a controlled shortcut by adding date stamps more often to MEL training images, encouraging a timestamp–MEL association. At evaluation, each test image is run in clean and timestamped form. We compute WGA over groups defined by class and timestamp presence, and measure the timestamp shortcut rate by checking how often timestamped non-MEL images are incorrectly predicted as MEL.

ART extends naturally to this setting by estimating a separate timestamp direction within each lesion class. The results mirror the binary benchmarks. Balanced Retrain remains largely stable, suggesting the timestamp association is not easily restored after balanced training. In contrast, several audited methods still show restoration effects. DFR and A-SSD show the clearest WGA degradation, with drops of \(31.8\) and \(17.9\) points, respectively. Other methods, such as GroupDRO and A-NegGrad+, show restoration mainly through increased timestamp-to-MEL shortcut errors rather than WGA degradation. A-SCRUB and A-SalUn show smaller but still non-negligible increases in this shortcut rate. Overall, the ISIC extension suggests that ART can reveal restorable shortcut associations beyond binary label–attribute settings.

5.7 ART-Targeted Mitigation↩︎

Finally, we test whether ART directions can serve as a head-level robustness signal. Starting from A-NegGrad+, we freeze the backbone and retrain only the final linear head using both clean and ART-restored features. This head-only procedure does not remove representation-level shortcuts; instead, it tests whether ART can guide the classifier away from functionally restorable associations.

Table 7 shows that the base A-NegGrad+ head is highly vulnerable: applying ART causes large WGA drops and substantial CSR increases. After retraining the classifier head on both clean and ART-restored features, this vulnerability is largely removed. In Waterbirds and CelebA, the ART-perturbed features even yield higher WGA than clean features, producing negative \(\Delta\). We interpret this not as representation-level shortcut deletion, but as head-level functional decoupling: the retrained head no longer maps the ART-amplified association direction to shortcut-consistent predictions. This benefit may come at a cost, as seen in CelebA where clean WGA decreases. Thus, ART-head is a mitigation probe demonstrating that ART directions can guide classifier decoupling, rather than a complete association-unlearning method.

3.2pt

6 Conclusion, Limitations, and Future Work↩︎

We introduced ART as a post-hoc diagnostic for testing whether retained label–attribute associations remain usable by the original classifier head. Across datasets, class-conditional shortcut structure often remains readable in frozen features even when output robustness improves. ART shows that readable associations differ in restorability: some methods decouple the head from retained shortcut structure, while others leave reactivatable associations that produce WGA drops and CSR gains.

Our work has several limitations. Low ART vulnerability indicates only that the specific restoration tested by ART does not readily reactivate the association; alternative nonlinear interventions may still recover it. ART also requires an audit split with target and attribute labels, and our implementation uses linear class-conditional directions in penultimate feature space. Other layers, architectures, nonlinear structures, or real-world multi-attribute associations may require different restoration estimators.

Future work should develop methods that more directly eliminate shortcut associations from internal representations, rather than primarily decoupling them from the classifier head. ART should also be extended beyond linear penultimate-space directions to nonlinear, layer-dependent, and multi-attribute shortcuts. Extending restoration-aware association audits to generative and vision-language models is another promising direction.

References↩︎

---

1. These authors contributed equally to this work.↩︎