1 Introduction↩︎

Every major machine unlearning benchmark shares a structural blind spot: causal knowledge. Why-type questions—probing the relational and causal chains that underlie factual knowledge—comprise less than \(0.06\%\) of CounterFact, \(0.6\%\) of ZSRE, \(1.2\%\) of TOFU, \(0.5\%\) of MUSE, and \(1.2\%\) of WMDP-Cyber (Table 1). This is not an oversight in any single benchmark—it is a systematic property of how these datasets were constructed: all derive from entity-centric knowledge graphs and relation-extraction corpora that inherently underrepresent causal and relational knowledge. The consequence is a critical measurement gap: any unlearning method that fails on causal knowledge can score highly in aggregate, and this failure is statistically undetectable without balanced evaluation.

1.0.0.1 Why Causal Knowledge Resists Unlearning.

The gap is not merely quantitative—causal facts are qualitatively harder to unlearn. Why-type answers average \(40.1\) tokens versus \(4.2\)–\(10.5\) for other categories, and \(44\%\) involve multi-hop reasoning chains compared to \(\leq 2\%\) for other categories (Table 7). These properties cause severe gradient dilution: the ascent signal is spread across long token spans with no dominant direction to target. Crucially, our encoding analysis (Appendix 13) shows this is not because Why-type facts are encoded differently—all 5W categories share uniform distributed encoding across layers. The difficulty is relational complexity and gradient dilution, not a unique weight-space footprint.

1.0.0.2 The MaatFramework.

We introduce Maat, a three-phase unlearning framework that operates directly on LoRA adapter weights without merging them into the base model. Rather than applying uniform gradient pressure, Maatperforms structured adapter surgery: (1) gradient projection orthogonalising forget updates against the retain gradient only when the two conflict; (2a) SVD-based pruning of Multi-Layer Perceptron(MLP) adapter dimensions to concentrate the forgetting signal on rank components specifically activated by forget-set inputs; (2b) task vector negation on the top-\(k_F\) forget-scored rank dimensions; and (3) hybrid KL–hidden-state retain repair with an entropy term preventing the repair phase from re-learning forgotten content. Evaluated under a Qwen 2.5-7B  [1] using LLM-as-a-Judge, Maatis the first method to simultaneously achieve high forgetting and high retention on Why-type causal knowledge—a new operating point on the forget–retain Pareto frontier that no baseline reaches.

1.0.0.3 Contributions.

1. 5WBench: a balanced 5,000-sample benchmark providing 1,000 examples per 5W question category (Who, What, When, Where, Why), exposing the causal knowledge gap in existing unlearning evaluation through structured taxonomic coverage.

2. Maat: a three-phase structured LoRA adapter unlearning framework that achieves a new forget–retain operating point on Why-type causal knowledge, outperforming all baselines on the aggregate forget–retain tradeoff across both Llama 3.2-3B and Gemma 3-4B.

2 Related Work↩︎

2.0.0.1 Gradient-Based and Preference-Based Unlearning.

The dominant paradigm for LLM unlearning applies gradient ascent (GA) directly on the forget set to maximize loss on target facts. KL-regularised GA [2] augments this with a divergence penalty against the original model’s outputs on retain samples, while Gradient Difference [3] combines forget-loss maximisation with retain-loss minimisation. A common failure mode across all gradient-based methods is that aggressive forgetting degrades model utility while conservative steps result in under-forgetting—particularly on long causal spans where the gradient signal is diffuse. More recently, unlearning has been framed as a preference alignment problem: Negative Preference Optimization (NPO) [4] treats forget data as a rejected distribution, applying DPO-style objectives without positive samples; SimNPO [5] removes reference-model bias entirely, improving robustness to relearning attacks [6]. Despite these advances, none of these methods account for the internal structure of adapter weight spaces or the geometry of the retain manifold.

2.0.0.2 Localization, Weight Saliency, and Structured Editing.

A parallel line of work identifies and targets specific weight subspaces associated with the forget target. Rank-One Model Editing (ROME) [7] and Mass-Editing Memory in a Transformer (MEMIT) [8] localise factual associations in mid-layer MLP weights via causal tracing and apply rank-one or distributed updates. AlphaEdit [9] improves specificity by projecting updates into the null space of the retained knowledge covariance—a conceptual predecessor to Maat’s gradient projection phase. SalUn [10] restricts updates to weights with the highest gradient-magnitude saliency on the forget set, providing the first principled weight-selection mechanism for unlearning. Mechanistic Unlearning [11] uses circuit-level localization via path patching to identify and fine-tune only fact-lookup components, producing edits robust to adversarial probes. All localization-based methods assume target knowledge is stored in identifiable, localised positions—an assumption our encoding analysis (Appendix 13) shows does not differentiate Why-type from other categories; the challenge is relational complexity and gradient dilution, not a unique encoding footprint.

2.0.0.3 Representation-Based and Adapter-Aware Methods.

Representation Misdirection for Unlearning (RMU) [12] steers intermediate activations toward a random direction in forget inputs while preserving retain representations, achieving state-of-the-art on WMDP. Circuit Breakers [13] reroute harmful representations to be orthogonal to the original hidden states via a LoRA [14] adapter, providing strong robustness under adversarial attacks. On the parameter-efficient side, LUNE [15] fine-tunes LoRA adapters on negative examples to overwrite targeted knowledge, and KGA [16] aligns the knowledge gap between two reference models. LoKU and FILA [15] apply Fisher Information to isolate forget-relevant parameters into LoRA adapters, providing the closest existing treatment of Fisher-guided adapter unlearning to Maat. Task vector negation [17] computes \(\tau = \theta_{\text{ft}} - \theta_{\text{base}}\) and subtracts it to approximate forgetting—the conceptual foundation of Maat’s Phase 2b, which refines this to target only the top-\(k_F\) forget-scored rank dimensions rather than the full adapter delta. Maatdiverges from all of the above by treating LoRA matrices as structured spaces where rank dimensions can be explicitly scored via SVD, selectively pruned, and negated—without requiring negative example construction or full adapter replacement.

2.0.0.4 Second-Order and Geometry-Aware Methods.

Natural Gradient Descent [18] preconditions updates with the inverse Fisher Information Matrix, yielding parameter-space geometry-aware updates. Selective Synaptic Dampening (SSD) [19] uses Fisher Information Matrix ratios between training and forget distributions to dampen forget-specific parameters without retraining. SOUL [20] establishes a connection between second-order optimization and influence-function unlearning, applying Sophia-based Hessian updates as a drop-in optimizer replacement for existing unlearning objectives, consistently outperforming first-order methods on TOFU [21]. Maatdiffers from SOUL [20] in that it uses first-order gradient projection combined with SVD rank [22] scoring to achieve structural suppression— circumventing full Hessian approximations while targeting the adapter’s forget subspace directly.

2.0.0.5 Benchmarks and Evaluation.

ZSRE [23] and CounterFact [24] dominate model editing evaluation but derive from entity-centric Wikidata triples and carry near-zero Why-type coverage (\(<1\%\)). TOFU [21] provides clean fictitious-fact unlearning splits but no structured question taxonomy, and is dominated by What-type biographical attributes (\(84.7\%\)). WMDP [12] targets hazardous capability suppression; MUSE [25] evaluates six unlearning desiderata including privacy leakage and sustainability; RWKU [26] provides zero-shot real-world entity unlearning with adversarial probes. None provide balanced causal Why-type coverage.

Recent work has further questioned benchmark reliability [6], [27], [28]: benchmark modifications expose residual accessible information, and fine-tuning on small auxiliary datasets reverses supposedly-unlearned knowledge.

5WBenchdirectly addresses the causal coverage gap by providing 1,000 balanced Why-type samples—making systematic failures on causal knowledge quantifiable for the first time.

3 The 5WBenchBenchmark↩︎

4.5pt

5pt

3.1 Dataset Construction↩︎

5WBenchis derived from the Factify-5WQA corpus [29], a multi-document fact-verification dataset with structured 5W question-answer annotations. Construction proceeds in four steps. (1) Subject extraction. Stanford CoreNLP dependency parsing [30] extracts the primary subject entity, which becomes the edit target. (2) Stratified sampling. We sample exactly 1,000 examples per 5W label, drawing uniformly within each category. Factify-5WQA [29] has sufficient Why-type entries—a property not shared by ZSRE or CounterFact. (3) Forget/Retain split. Each label’s 1,000 samples are split equally: 500 forget, 500 retain. Evaluation uses 100 samples per label per set (500 forget \(+\) 500 retain total), stratified to ensure equal 5W representation. (4) Format standardisation. Each sample is formatted as a (question, answer, label, rephrases) tuple compatible with EasyEdit [31]. The label reflects the semantic type of the relation queried, not the surface question word.

3.1.0.1 Sample Format.

Each 5WBenchrecord is a JSON object containing the question, ground-truth answer, the 5W label, and up to three rephrased question variants generated to test robustness of editing methods to surface-form variation. Table 2 shows a representative what-type instance from the forget split, drawn from the ZSRE source.

3.2 Why-Type Facts Are Structurally Different↩︎

Why-type facts encode causal and relational chains (e.g., “Smoking causes lung cancer because it introduces carcinogens into lung tissue”). Answer spans average 40.1 tokens versus 4.2–10.5 for other types, and a staggering 44% involve multi-hop reasoning chains (Table 7, Appendix 10). This complexity means gradient ascent on a long token span cannot produce a coherent unlearning signal. 5WBenchprovides sufficient Why-type samples to study and quantify this failure mode for the first time.

4 The MaatFramework↩︎

Maataddresses the unlearning challenge by operating on the structure of the LoRA adapter’s parameter space (Figure 1). All three phases act exclusively on adapter matrices \(\{\mathbf{A}_l, \mathbf{B}_l\}\); base model weights remain frozen throughout.

4.1 Phase 1: Gradient-Projected Unlearning↩︎

Standard gradient ascent applies a forget update \(\mathbf{g}_f\) uniformly. If \(\mathbf{g}_f\) has components aligned with the retain gradient \(\mathbf{g}_r\), those components erode retained knowledge. Maatremoves this component via conditional orthogonal projection—applied only when the forget and retain gradients actively conflict (\(\mathbf{g}_f \cdot \mathbf{g}_r > 0\)): \[\mathbf{g}_f^{\perp} = \begin{cases} \mathbf{g}_f - \dfrac{\mathbf{g}_f \cdot \mathbf{g}_r}{\|\mathbf{g}_r\|^2 + \epsilon} \,\mathbf{g}_r & \text{if } \mathbf{g}_f \cdot \mathbf{g}_r > 0 \\[6pt] \mathbf{g}_f & \text{otherwise} \end{cases} \label{eq:gradproject}\tag{1}\] where \(\epsilon > 0\) is a small numerical stabiliser. When the gradients do not conflict, the full forget-ascent direction is preserved without attenuation, maintaining unlearning signal strength. The KL reference distribution is fixed to the pre-unlearning adapter state \(\mathbf{W}_{\text{ref}}\). The unlearning objective to be minimised is: \[\mathcal{L}_{\text{unlearn}} = -\mathcal{L}_{\text{forget}}(q, a) \;+\; \lambda\,\mathrm{KL}\!\left( p_{\mathbf{W}} \;\|\; p_{\mathbf{W}_{\text{ref}}} \right)_{\mathcal{D}_R} \label{eq:unlearn95obj}\tag{2}\] where \(\lambda > 0\) controls the retain-anchoring strength.

4.2 Phase 2a: SVD Rank-Dimension Pruning (MLP-only)↩︎

Residual forget-fact signal may persist in low-magnitude adapter directions not captured by the projected gradient. For each MLP LoRA pair \((\mathbf{A}_l, \mathbf{B}_l)\), rank dimensions are scored by their gradient-column norm over a sample of forget-set inputs: \[s_k = \sum_{x \in \mathcal{D}_F^{\text{score}}} \left\|\nabla_{\mathbf{B}_{l}}\mathcal{L}(x)\right\|_{\text{col-}k} \label{eq:svd95score}\tag{3}\]

The top-\(\rho\) fraction of rank dimensions by score are zeroed in both A and B matrices on MLP modules (down_proj, up_proj, gate_proj) only. Attention modules are excluded: pruning attention rank dimensions at non-trivial ratios destroys the instruction-following pathway.

4.3 Phase 2b: Task Vector Negation↩︎

Phase 2b introduces weight-space negation targeting the forget subspace. The top-\(k_F\) fraction of forget-scored rank dimensions of each LoRA B matrix are masked to form a forget task vector \(\boldsymbol{\tau}_l^F\), then subtracted: \[\mathbf{B}_l \gets \mathbf{B}_l - \alpha \cdot \boldsymbol{\tau}_l^F, \quad \alpha > 0 \label{eq:taskvec95negate}\tag{4}\]

By confining negation to the highest forget-scoring rank dimensions (here \(k_F = 50\%\)), Phase 2b achieves targeted suppression without inadvertently erasing retain-associated directions.

4.4 Phase 3: Retain Repair with Hybrid Objective↩︎

SVD pruning and gradient ascent may partially degrade retain performance. Phase 3 recovers it through a hybrid repair objective combining four terms: \[\begin{gather} \mathcal{L}_{\text{repair}} = w_{\text{KL}}\,\mathrm{KL}(p_{\mathbf{W}} \| p_{\text{ref}})_{\mathcal{D}_R} \\ + w_{\text{HS}}\,d_{\text{rep}}(h_{\mathbf{W}}, h_{\text{ref}})_{\mathcal{D}_R} - w_{\text{ent}}\,\mathcal{H}_F(p_{\mathbf{W}}) \\ + w_{\text{TV}}\!\sum_l \cos(\mathbf{B}_l, \boldsymbol{\tau}_l^F)^+ \label{eq:repair} \end{gather}\tag{5}\]

where \(d_{\text{rep}}(h, h') = 1 - \cos(h, h')\) is the hidden-state representation distance, \(\mathcal{H}_F(p_{\mathbf{W}})\) is the output entropy of the current model evaluated on forget-set answer tokens, and the final term penalises cosine similarity between current LoRA \(\mathbf{B}_l\) weights and the forget task vector. Crucially, the entropy term enters with a negative sign: minimising \(-w_{\text{ent}}\mathcal{H}_F\) is equivalent to maximising entropy on forget-set predictions, discouraging the model from recovering forgotten content during the repair phase.

4.5 Knowledge Implantation Protocol↩︎

To benchmark unlearning in a controlled setting, we first implant target facts via LoRA fine-tuning (rank \(r\), scaling factor \(\alpha_{\text{LoRA}}\), targeting q, k, v, o, gate, up, down projections on layers \([l_s, l_e]\)) with 4-bit NF4 quantisation (learning rate \(\eta_0 > 0\) for \(E\) epochs). Crucially, the LoRA adapter is not merged into the base model: all Maatphases operate directly on adapter weights.

4.6 Maat Algorithm↩︎

We summarize the full Maat training procedure in Algorithm 2.

5 Experimental Setup↩︎

5.0.0.1 Models.

We evaluate on two instruction-tuned models from distinct architecture families: LLaMA 3.2-3B-Instruct [32] and Gemma 3-4B-Instruct [33]. Both fit on a single consumer GPU (\(\leq\)​24 GB VRAM). Inference: greedy decoding, temperature \(= 0\), max_new_tokens \(= 100\). For hyperparameters on 5WBench using Maat refer to (Appendix 12, Table 8).

5.0.0.2 Datasets.

5WBench (see §3 for details): 1,000 per label (5,000 total); experiments use 100 per label per set (forget500 \(+\) retain500), stratified to ensure equal 5W representation. TOFU [21]: We use the forget05/retain95 split. Although TOFU is not statistically useful for label-wise semantic computation due to severe category imbalance (see Appendix 14 for distributions), it is included to ensure comparability and completeness with existing literature. TOFU contains no meaningful Why-type questions.

5.0.0.3 Baselines.

All baselines operate on the fine-tuned LoRA adapter.

• Gradient Ascent (GA): Direct loss maximisation on the forget set.

• Gradient Ascent with KL Divergence (GA+KL): GA regularised by KL divergence.

• Adapter Negation (AN): Full task vector negation, \(\alpha = 1.0\).

• Retain-Only Fine-Tuning (RO-FT): Fine-tuning exclusively on the retain set.

5.0.0.4 Evaluation Metrics.

LLM-as-Judge (primary).[34] We evaluate Forget Success Rate () and Retain Success Rate () using a Qwen 2.5-7B judge that determines whether the model’s output semantically contains the ground-truth answer. \(\text{\text{FSR}{}} = 1\) when the ground truth is not present (successful forgetting); \(\text{\text{RSR}{}} = 1\) when it is present (successful retention). Full prompt template and evaluation rules are provided in Appendix 15.

ROUGE: We report ROUGE-1, ROUGE-2, and ROUGE-L  [35] on the forget and retain splits before and after unlearning in the Appendix 8. Lower forget ROUGE indicates more successful token-level erasure; higher retain ROUGE indicates better preservation of non-targeted knowledge.

6 Results↩︎

Tables 3 and 4 present results on 5WBenchand TOFU respectively. Appendix 16 provides qualitative examples illustrating how each method handles Why-type questions on both forget and retain splits. 5WBench’s balanced 5W coverage exposes performance differences between methods that TOFU’s category-skewed evaluation cannot resolve.

3.5pt

5pt

6.1 5WBench Results (forget500/retain500)↩︎

6.1.0.1 MAAT dominates the forget–retain Pareto frontier.

Maatachieves the best aggregate – balance on both models: \(77.4\%\) / \(71.6\%\) on Llama 3.2-3Band \(64.0\%\) / \(61.8\%\) on Gemma 3-4B. The cleanest single comparison is against RO-FT: both methods reach identical \(77.4\%\) average on Llama 3.2-3B, yet Maatdelivers this with \(71.6\%\) versus RO-FT’s \(35.2\%\)—a \(+36.4\)-point retention gain at zero forget cost. More strikingly, Maatis the only method to simultaneously exceed \(60\%\) and \(60\%\) on all five 5W categories on Llama 3.2-3B(Who: \(83/72\), When: \(79/77\), What: \(82/71\), Where: \(80/73\), Why: \(63/65\)); no baseline achieves this threshold on any single category.

6.1.0.2 Baseline tradeoff failure modes.

Each baseline occupies a distinct failure region of the Pareto frontier. GA achieves moderate average (\(57.4\%\) Llama 3.2-3B, \(61.2\%\) Gemma 3-4B) at the cost of lagging Maatby \(18\) and \(11\) points respectively. GA+KL (\(33.8\%\) on Llama 3.2-3B) sacrifices forgetting for retention—the lowest average of any real method. Adapter Negation (AN) maximises (\(99.8\%\) Llama 3.2-3B, \(82.0\%\) Gemma 3-4B) but triggers catastrophic forgetting uniformly across all 5W categories: collapses to \(0.4\%\) on Llama 3.2-3B and \(25.0\%\) on Gemma 3-4B. This confirms that full task vector negation removes both forget and retain knowledge indiscriminately—the entire adapter delta encodes both simultaneously. The partial resistance on Gemma 3-4B(25.0% ) compared to Llama 3.2-3B(0.4% ) suggests architecture-dependent sensitivity to full adapter negation.

6.1.0.3 Why-type results and the under-forgetting artifact.

On the Why category, Maatachieves \(63\%\) / \(65\%\) on Llama 3.2-3Band \(55\%\) / \(56\%\) on Gemma 3-4B. GA’s higher Why- (\(78\%\) on Llama 3.2-3B) is not evidence of better retention—it is a mechanical artifact of under-forgetting. At \(44\%\) Why-, more than half the target causal facts remain in the model, so retention is preserved by default rather than by design. Maatforgets \(19\) percentage points more Why-type knowledge (\(63\%\) vs \(44\%\)) while losing only \(13\) points (\(65\%\) vs \(78\%\))— the only method that meaningfully removes causal knowledge without indiscriminate retain damage. GA+KL (\(22\%\) Why- / \(93\%\) Why- on Llama 3.2-3B) represents the extreme of this artifact: near-perfect retention because almost nothing has been forgotten.

6.1.0.4 Per-category localized failures.

Beyond the Why category, 5WBench’s balanced coverage reveals localized failures in other categories. GA’s What-type on Llama 3.2-3B(\(48\%\)) approaches its Why-type (\(44\%\)), and Maat’s gap over GA is actually larger on What (\(+31\) points) than on Why (\(+19\) points)—showing that GA’s weakness is not unique to causal knowledge. GA+KL’s When- on Gemma 3-4Bcollapses to \(39\%\) while all other categories remain \(62\)–\(71\%\), indicating a localized temporal knowledge failure under KL regularization on this architecture. RO-FT’s Where- on Llama 3.2-3B(\(23\%\)) is dramatically lower than its other categories (\(35\)–\(42\%\)). 5WBench’s balanced \(500\)-sample splits per category provide the statistical power to detect these localized failures—a granularity unavailable in existing benchmarks.

6.1.0.5 Consistency and architecture sensitivity.

Maat’s ranges \(63\)–\(83\%\) across categories on Llama 3.2-3B, compared to GA’s \(44\)–\(71\%\), indicating more consistent unlearning behavior regardless of question type—an important practical property since real-world forget requests may contain any mix of knowledge types. All methods degrade moving from Llama 3.2-3Bto Gemma 3-4B, but differently: RO-FT swings dramatically ( \(-26.0\), \(+35.2\)), indicating high architecture-dependence, while Maat degrades more uniformly ( \(-13.4\), \(-9.8\)), maintaining a balanced operating point across both architectures.

6.2 TOFU Results (forget05/retain95)↩︎

On TOFU, Maatachieves the highest among methods with \(>60\%\) on Llama 3.2-3B(\(67\%\) / \(46.6\%\) ). AN again triggers catastrophic forgetting (\(100\%\) / \(0.2\%\) ), replicating the indiscriminate-negation pattern seen on 5WBench. On Gemma 3-4B, methods cluster tightly: Maat(\(61.5\%\)/\(48.7\%\)), RO-FT (\(63.0\%\)/\(46.2\%\)), and GA+KL (\(54.0\%\)/\(47.4\%\)) differ by \(1\)–\(3\) points—within the noise range of LLM-judge evaluation. TOFU’s category imbalance (\(84.7\%\) What-type, \(\leq 5\%\) each for When/Who/Where, \(1.2\%\) Why) means aggregate – is dominated by a single question type. 5WBench’s balanced \(500\)-sample splits per category provide the statistical power to surface per-category differences that TOFU’s distribution cannot resolve. TOFU results are included for comparability with existing literature.

7 Conclusion↩︎

We introduced 5WBench, a balanced 5,000-sample benchmark that makes causal unlearning failures quantifiable for the first time, and Maat, a three-phase structured LoRA adapter unlearning framework that concentrates the forgetting signal on rank dimensions specifically activated by forget-set inputs.

Using 5WBench, we established that no existing baseline simultaneously achieves high forgetting and high retention on Why-type causal knowledge—and that Maatis the first method to do so, exceeding \(60\%\) and \(60\%\) on all five 5W categories on Llama 3.2-3B, including Why-type causal questions where all baselines face a fundamental forget–retain tradeoff.

The performance gap between Llama 3.2-3B(\(77.4/71.6\)) and Gemma 3-4B (\(64.0/61.8\)) reveals that adapter-based unlearning inherits the base model’s knowledge encoding structure: forget and retain knowledge are more separable in Llama 3.2-3B’s adapter rank dimensions, while Gemma 3-4B’s interleaved local/global attention pattern distributes knowledge more diffusely, reducing separability. Architecture-aware adapter rank selection is an important open direction for future work.

Beyond unlearning, 5WBench’s balanced 5W taxonomy and EasyEdit-compatible format make it directly applicable to model editing evaluation—including insertion and modification on causal knowledge, a category no existing editing benchmark adequately covers.

Limitations↩︎

7.0.0.1 Evaluation protocol.

and are computed using a single Qwen 2.5-7B LLM judge. While this provides a reproducible, semantically-aware alternative to exact-match metrics, it may have calibration differences compared to proprietary judges or human evaluation. Judge availability and versioning may affect reproducibility across time. Future work should complement LLM-judge evaluation with adversarial paraphrase probing to assess whether unlearning is genuine or merely suppresses verbatim recall.

7.0.0.2 Dataset scope.

5WBenchis derived from the Factify-5WQA corpus [29], which was designed for fact verification rather than knowledge editing. The subject-extraction pipeline may introduce noise for facts with implicit or pronominal subjects. Answer length and multi-hop statistics in Table 7 are computed on the full annotated dataset. The complete benchmark will be released upon publication.

7.0.0.3 Model and benchmark scope.

Evaluation covers two model families at 3–4B scale on 5WBenchand TOFU; extension to 7B+ checkpoints and additional architectures is left for future work. The performance gap between Llama 3.2-3Band Gemma 3-4Bsuggests that adapter rank separability is architecture-dependent, and optimal hyperparameters for Phase 2a pruning ratio \(\rho\) and Phase 2b mask \(k_F\) may require architecture-specific tuning. While 5WBench’s format supports insertion and modification operations, this paper evaluates only unlearning; extending to other editing operations is left for future work.

Ethical considerations↩︎

The unlearning methods developed in this paper address legitimate privacy and safety needs including GDPR compliance and correction of harmful associations. We acknowledge a dual-use concern: the same techniques could be applied adversarially to remove safety-relevant knowledge from aligned models.

8 Ablation Study↩︎

To understand the contribution of each Maatphase, we conduct ablation experiments on a 200-sample 5WBenchsubset (20 forget \(+\) 20 retain per label) on Llama 3.2-3B. We first confirm successful knowledge implantation (Table 5), then evaluate four ablation conditions (Figure 3) that progressively introduce Maatcomponents. Post-unlearning ROUGE scores (Table 6) complement the FSR/RSR analysis by quantifying token-level erasure and preservation.

8.0.0.1 Knowledge implantation verification.

Table 5 reports ROUGE scores of the fine-tuned LoRA adapter before any unlearning on both 5WBenchsplits. High scores on the forget set confirm that target facts have been successfully implanted; high scores on the retain set confirm that general knowledge is intact. These serve as upper-bound references: lower forget ROUGE after unlearning indicates successful erasure, while retain ROUGE should remain as close to these baselines as possible.

4.5pt

8.0.0.2 Ablation conditions.

Each condition adds or modifies one component relative to the previous, isolating its effect on the forget–retain tradeoff.

Condition A (Gradient projection \(+\) SVD \(+\) KL-only repair; no task vector negation, no attention pruning): Phase 1 gradient-projected ascent and Phase 2a SVD rank-dimension pruning on MLP modules only, paired with a repair phase that uses only the KL-divergence term from Eq. 5 . Achieves 48% and 83% . The strong retention reflects effective anchor protection from gradient projection; however, the absence of structural erasure components beyond MLP pruning limits overall unlearning efficacy. Notably, a Why- of 45% indicates that gradient projection alone provides a stable baseline for unlearning causal chains.

Condition B (Condition A \(+\) hybrid repair; no task vector negation, no attention pruning): Same as Condition A, but Phase 3 is upgraded from KL-only to the full hybrid repair objective (Eq. 5 ), incorporating the KL-divergence, hidden-state representation distance, and negative forget-set entropy terms. Drops to 43% while driving up to 90%. The richer repair phase over-corrects: without a structural erasure anchor, it dominates the parameter updates and inadvertently recovers forgotten content.

Condition C (Condition B \(+\) attention pruning; no task vector negation): Extends Condition B by applying SVD rank-dimension pruning to attention modules (q_proj, k_proj, v_proj, o_proj) at a pruning ratio of \(\rho_\text{attn} = 0.01\), in addition to MLP pruning. Sharply shifts the balance: jumps to 70% but drops to 54%. Structural attention pruning strongly suppresses forget-set knowledge, but induces severe collateral damage to retained distributions—empirically confirming the design decision in §4.2 to exclude attention modules from pruning.

Condition D (Full Maat): Removes attention pruning from Condition C and instead introduces Phase 2b task vector negation on the top-\(k_F\) forget-scored rank dimensions, alongside the full hybrid repair phase (Phase 3) including the task-vector cosine penalty term. The complete pipeline: Phase 1 (gradient-projected ascent) \(+\) Phase 2a (SVD pruning, MLP only) \(+\) Phase 2b (task vector negation) \(+\) Phase 3 (full hybrid repair with all four loss terms). Achieves 71% and 76% . Task vector negation provides comparable forgetting strength to attention pruning (71% vs.% ) with far better retention (76% vs.% ), serving as the critical differentiator that reclaims retain performance while maintaining effective unlearning—particularly on multi-hop causal structures (Why- = 65%).

8.0.0.3 Post-unlearning ROUGE analysis.

Table 6 reports ROUGE scores on the forget and retain splits after applying each ablation condition, complementing the FSR/RSR analysis with token-level overlap metrics. Lower forget ROUGE indicates more successful knowledge erasure; higher retain ROUGE indicates better preservation of non-targeted knowledge. Compared to the pre-unlearning baselines in Table 5, these values quantify the retain–forget ROUGE tradeoff induced by each component.

Condition A (KL-only repair) provides a balanced initial baseline, maintaining a high retain R-1 (0.859) due to effective anchor protection. Upgrading to hybrid repair in Condition B further elevates retain preservation to an experimental maximum (R-1 = 0.892), but over-corrects: forget R-1 inflates to 0.614, confirming that the repair phase recovers forgotten content without a structural erasure anchor. Condition C sharply reverses this dynamic: attention pruning drives forget R-1 down to 0.325, but incurs heavy collateral damage (retain R-1 drops to 0.644). Full Maat(Condition D) resolves this tradeoff: by pairing MLP pruning with task vector negation instead of attention pruning, it maintains deep forget-set erasure (R-1 = 0.319) while recovering retain performance (R-1 = 0.721)—a \(+\)​0.077 retain improvement over Condition C at equivalent forget-set erasure depth.

4pt

9 Complete Unlearning Metric Profiles Across All Methods↩︎

Figure 4 presents the full per-label unlearning metric profiles for all five methods on both LLaMA 3.2-3B and Gemma 3-4B, evaluated across three complementary aggregation functions:

• Linear Mean: (\(\text{FSR} + \text{RSR} - 100\)). Centred at zero; positive values indicate net forgetting gains exceed retention losses.

• Geometric Mean: (\(\sqrt{\text{FSR} \times \text{RSR}}\)). Collapses to zero if either metric is zero, penalising catastrophic imbalance.

• Harmonic Mean: (\(\frac{2 \times \text{FSR} \times \text{RSR}}{\text{FSR} + \text{RSR}}\)). The most conservative aggregator, heavily penalising extreme imbalances.

Key observations from the full profiles. Across all three metrics, Maatconsistently occupies the Pareto-dominant region—achieving both high forgetting and high retention—while each baseline exhibits a characteristic failure mode visible in the metric profiles. AN’s Geometric and Harmonic scores collapse near zero on LLaMA due to near-zero retention, even though its Linear score is moderately positive. This illustrates why single-metric or forget-only reporting conceals retention failures. GA+KL’s Geometric and Harmonic means are competitive on Who and Why on LLaMA where its high retention partially compensates for low forgetting, but its Linear scores are negative, revealing that retention gains come at the price of under-forgetting. GA’s profile is the most consistent across metrics, but remains below Maat on all three aggregators. The When category is the hardest across both models and all methods, corroborating the short-answer, single-entity structure that limits gradient-ascent effectiveness.

10 Per-Category Statistics↩︎

Table 7 summarizes the per-category characteristics of 5WBench. Notably, Why-type questions exhibit substantially longer answers and higher multi-hop complexity, making them particularly challenging for standard unlearning methods.

6pt

11 Label-Wise Harmonic Mean Efficiency Across Methods↩︎

Figure 5 visualises the label-wise harmonic mean aggregated as an average across all evaluated methods per 5W category for both models. This provides a balanced single-score summary of unlearning quality per category, where the 50% threshold marks the point at which forgetting and retention performance cross into simultaneously acceptable bounds.

Gemma achieves its highest harmonic efficiency on Who (59.1%) and What (54.7%), and its lowest on When (41.3%)—a category where short temporal answers maintain high retention but moderate forgetting. LLaMA’s profile is more uniform, ranging from 41.8% (Why) to 49.0% (Where), staying just below the 50% threshold across all categories. Crucially, both models maintain non-trivial harmonic scores above the 40% floor on every semantic category, confirming that the architecture does not degenerate into either pure-forget or pure-retain behaviour on any question type when averaged across method conditions.

12 Hyperparameter Details↩︎

We detail the complete configuration of hyperparameters for each model across all experimental pipeline stages in Table 8. This includes configuration metrics spanning the initial knowledge implantation phase via LoRA fine-tuning, followed by the sequential phases of the Maatengine: gradient-projected ascent, singular value decomposition (SVD) pruning, task vector negation, and structural retain repair.

4pt

13 Encoding Analysis↩︎

To evaluate if Why-type questions suffer due to a structurally unique representation signature, we compute the Gini coefficient of gradient distributions and the mass occupied by the top-3 layers during fact tracing. Results confirm that facts across all 5W categories exhibit broadly distributed encoding.

As presented in Table 9, both Llama 3.2-3Band Gemma 3-4B exhibit low average Gini coefficients and small top-3 layer mass fractions across all 5W categories, confirming distributed encoding. Gemma 3-4B shows Gini values ranging from 0.152 to 0.177 with top-3 mass of 17.8–19.3%, while Llama 3.2-3Bexhibits slightly higher absolute values (Gini 0.258–0.277, top-3 mass 28.1–29.9%) but remains equally uniform across categories. Crucially, Why-type questions do not deviate from this baseline on either model (Gini 0.172 on Gemma 3-4B, 0.276 on Llama 3.2-3B), confirming that the difficulty of unlearning causal knowledge stems from relational complexity and gradient dilution, not a unique encoding footprint.

3pt

14 TOFU Label Distribution↩︎

While TOFU is used for comparative evaluation, Table 10 displays the highly skewed label inferences across its splits. Because the Why category accounts for negligible volume (e.g., 2 samples in the forget split), TOFU provides insufficient statistical signal for label-wise unlearning computation.

6pt

15 Judge Prompt Details↩︎

The Qwen 2.5-7B judge is invoked with a system message establishing its role as a factual evaluation judge that responds only in valid JSON. The user prompt template below is applied identically for and evaluation; interpretation changes by verdict direction only: false \(\Rightarrow\) \(\text{FSR} = 1\); true \(\Rightarrow\) \(\text{RSR} = 1\). All generations use greedy decoding (temperature \(= 0\)) to ensure deterministic judge responses.

16 Qualitative Examples: Why-Type Unlearning↩︎

Tables 11 and 12 show representative generation traces from Llama 3.2-3Bon Why-type evaluation samples, illustrating how each method handles causal knowledge on the forget and retain splits.

16.0.0.1 Forget set (Table 11).

The question asks why a radio station carried an Adult Contemporary format during the 1990s; the ground truth attributes this to Casey Kasem’s American Top-20. Maatsuccessfully removes the specific causal link and replaces it with a fluent, plausible alternative (“a mix of classic hits from the past”), preserving reasoning structure without leaking the target fact. GA and GA+KLfail entirely, reproducing “Casey Kasem’s American Top-20” almost verbatim. AN achieves superficial forgetting (\(\text{FSR}= 1\)) but only through catastrophic parameter degradation—its output collapses into incoherent medical terminology. RO-FT avoids the target fact but resorts to a generic refusal (“I don’t have specific information…”), signaling a loss of generation flexibility rather than targeted erasure.

16.0.0.2 Retain set (Table 12).

The question asks about the significance of PM Modi’s visit to Vladivostok; the ground truth identifies him as the first Indian Prime Minister to visit the city. Maatpreserves this core fact with minor paraphrasing (“India’s growing ties with Russia in the Far East”), matching the retention fidelity of GA, GA+KL, and RO-FT. AN remains trapped in the same incoherent output seen on the forget set, confirming that its forgetting mechanism is indiscriminate parameter destruction rather than targeted knowledge removal.

6pt

References↩︎