1 Introduction↩︎

Machine learning models trained on sensitive medical data face a critical challenge: how to remove the influence of specific training examples while preserving overall model utility. This capability, known as machine unlearning, is essential for healthcare applications where patients may request data deletion under privacy regulations like GDPR’s “right to be forgotten” [1], or where erroneous or outdated medical records must be excised from trained models without costly full retraining. Despite growing interest in machine unlearning across various domains, the medical field lacks standardized benchmarks for evaluating unlearning methods for clinical question-answering tasks.

To address these gaps, we introduce AMNESIA: A Large Scale Medical UNlearning BEnchmark Suite with Disease-Informed Analysis. AMNESIA provides the first large-scale benchmark for evaluating machine unlearning in medical question-answering (QA), comprising 70,560 question-answer pairs derived from 8,820 patient notes across 11 disease categories. Unlike prior unlearning benchmarks built on synthetic or general-domain QAs (Appendix 8), AMNESIA is grounded in real patient notes, spanning thousands of patients across many disease conditions. This is essential in medical contexts, where consent withdrawals, diagnostic revisions, and record corrections continually generate new unlearning targets. AMNESIA also distinguishes factual questions (direct recall) from reasoning questions (clinical inference) and supports both random patient-level and disease-level forgetting scenarios, enabling researchers to test whether models can forget patients with particular medical conditions.

Our contributions are as follows:

• First large-scale clinically-grounded unlearning benchmark: 70,560 Question-Answer pairs (QAs) from 8,820 patient notes across 11 disease categories. Each patient has 4 factual (recall) and 4 reasoning (inference) QAs. ²

• Multi-level evaluation across diverse forget/retain splits: Evaluation across random patient-level and disease-level forget/retain splits.

• Comprehensive unlearning analysis: Four representative unlearning methods are evaluated and analyzed.

• Medically-aware unlearning evaluation: Novel disease-focused keyword evaluation for forget performance.

• Release of medical models and dataset: Our models are publicly available. Our detailed data construction pipeline makes AMNESIA reproducible and easy to extend.

2 Related Work↩︎

2.0.0.1 Machine Unlearning Benchmarks.

Machine unlearning aims to remove the influence of specific training data from a model without full retraining [2]. TOFU [3] establishes the standard benchmark for LLM unlearning using synthetic author profiles, with methods including Gradient Difference [4], KL Minimization [5], and Negative Preference Optimization [6]. R-TOFU [7] extends TOFU to Large Reasoning Models, showing that answer-only objectives leave residual forget traces in chain-of-thought reasoning. MUSE [8] evaluates forgetting in LLMs but does not address clinical or cross-modal structures. Recent work further shows that existing methods fail on structured, multi-hop knowledge [9], motivating more realistic evaluation settings. We follow OpenUnlearning [10], a unified framework integrating 13 unlearning algorithms and 16 evaluations across TOFU, MUSE, and WMDP [11], to inform our selection and standardized comparison of unlearning methods.

2.0.0.2 Medical Question Answering.

Clinical NLP has benefited from large-scale datasets such as MIMIC-CXR  [12], with models like LLaVA-Med [13] and Asclepius [14] demonstrating that LLMs fine-tuned on patient notes acquire patient-specific knowledge. Recent benchmarks have pushed medical QA in several directions: evaluating LLM reasoning and explanation on challenging clinical cases [15], detecting hallucinations in medical outputs [16], assessing multi-hop biomedical reasoning across knowledge graphs [17], and supporting argumentative explanation of diagnoses [18]. However, these benchmarks do not address machine unlearning in a medical QA setting.

2.0.0.3 Machine Unlearning in Medical AI.

Privacy regulations, including HIPAA [19] and GDPR [1], motivate the selective removal of patient data from trained models, with prior defenses such as federated learning [20] offering only partial protection. Other work has explored unlearning multimodal patient information in clinical imaging settings [21]. MedForget [22] is the closest prior work, providing a hierarchical multimodal benchmark of 3,840 Visual Question Answering (VQA) pairs across 64 patients from MIMIC-CXR. In contrast, AMNESIA scales to 70,560 clinical note–grounded QAs and introduces disease-level unlearning, a clinically motivated setting not addressed by prior work.

3 AMNESIA Dataset↩︎

Figure 1 illustrates the AMNESIA framework. Given a patient note, AMNESIA includes factual and reasoning questions, each with gold-standard answers. After unlearning, models should produce substantially different (uninformative) responses to questions about forgotten patients while maintaining accurate answers for retained patients.

3.1 Data Construction (Figure 1.a)↩︎

3.1.0.1 Dataset Selection and Sampling

We use PMC-Patients-v2 [23], a large-scale publicly available dataset of de-identified patient notes extracted from reports in PubMed Central [24]. From this corpus, we randomly sampled 8,820 patient notes to create a computationally feasible yet substantial dataset for our study.

3.1.0.2 Disease Category Assignment

We used GPT-5.4-nano to assign a disease from Centers for Disease Control and Prevention (CDC)’s classification of common diseases ³ (11 diseases listed in Table 1) to each patient based on its note title. The note title is a single sentence describing the patient note. The prompt is designed to enforce GPT-5.4-nano to map each note’s title to a single disease category (see Appendix 9.1). This categorization enables AMNESIA’s disease-informed analysis of machine unlearning methods. We evaluate these categories in the .

3.1.0.3 Question-Answer Generation

For each patient, we use GPT-5-nano to generate 8 QAs: 4 factual QAs testing direct information recall from the patient’s note and 4 reasoning QAs requiring inference across multiple findings within the note, making them more challenging than factual questions. The generation prompt (Appendix 9.2) enforces strict quality criteria: questions must be answerable only from the specific patient’s details (not general medical knowledge), must avoid any personal information, and must maintain clinical specificity. This generation strategy produced 70,560 total QAs (35,280 factual and 35,280 reasoning), ensuring coverage of both knowledge retrieval and clinical reasoning capabilities. Table 1 presents the final dataset composition across disease categories.

3.1.0.4 Human-in-the-Loop Evaluation

We validate both disease labels and QAs under majority vote with a three-judge LLM panel: a medical-specialized model (MedGemma-27B [25]), a general-purpose reasoning model (Qwen3-32B [26]), and a frontier model (GPT-5-mini [27]). For disease labels, we sample 900 patient notes (\(\sim\)​10% of total notes) from the three most populated diseases (300 each from Cancer, Infectious/Immune, and Cardiovascular). For QAs, we sample 500 patients and take 1 factual and 1 reasoning QA from each, yielding 1,000 QAs. The judge panel supports \(\mathbf{91.8\%}\) of disease labels and rates \(\mathbf{97.6\%}\) of QA triples as valid against four criteria (clarity, correctness, derivability, clinical meaning); a biology student’s annotations of the same patient notes agree with the judges’ majority vote on \(\mathbf{90.4\%}\) of disease labels. Together, these results indicate that AMNESIA’s QAs and disease labels are trustworthy for unlearning evaluation. Full protocols, prompts, and per-judge breakdowns are in Appendices 13 and 14.

3.2 Data Partition (Figure 1.b)↩︎

Our partition strategy creates hierarchical forget/retain splits at two levels: random patient-level and disease-level.

3.2.0.1 Random Patient-Level Splits

Similar to existing unlearning benchmarks [3], [22], we create forget splits by randomly selecting patients at 5%, 10%, 15%, 20%, and 25% of the total number of patients, where each smaller forget set is a proper subset of the next larger one (i.e., forget_5 \(\subset\) forget_10 \(\subset\) forget_15 \(\subset\) forget_20 \(\subset\) forget_25). For each forget split, the corresponding retain split contains all remaining patients.

3.2.0.2 Disease-Level Splits

We build disease-level splits for the three most prominent diseases in our dataset: Cancer, Infectious/Immune, and Cardiovascular. Each disease-level split matches the size of the corresponding random patient-level split, sampled only from the target disease. The size of the largest possible forget set for each disease depends on the total number of patients having that disease in the dataset (Table 1):

• Cancer (30.3% of all patients): 5%, 10%, 15%, 20%, 25% forget splits.

• Infectious/Immune Diseases (22.3% of all patients): 5%, 10%, 15%, 20% forget splits.

• Cardiovascular (16.2% of all patients): 5%, 10%, 15% forget splits.

Each disease-level split maintains two key properties: (1) the subset relationship holds within each disease (e.g., cancer_forget_5 \(\subset\) ... \(\subset\) cancer_forget_25), and (2) all patients with a given disease label that appear in the random patient forget split are also included in the corresponding disease-level forget split at the same percentage (e.g., all cancer patients in forget_5 are guaranteed to appear in cancer_forget_5. Appendix 10 shows the size of each data split.). The 900 samples selected for human-in-the-loop evaluation of disease categories are drawn from cancer_forget_5, cardio_forget_5, and infectious/immune_forget_5 splits, which are shared among all forget sets of that disease (See ).

3.2.0.3 Holdout Dataset

To evaluate generalization, we create a holdout set by withholding a factual and a reasoning QA per patient (17,640 total), never seen during fine-tuning and unlearning. Complete statistics for all splits are provided in Appendix 10.

4 Unlearning Benchmark (Figure 1.b)↩︎

We establish a medical unlearning benchmark through a four-stage workflow: (1) Pre-training on patient notes, (2) instruction fine-tuning on QAs, (3) applying unlearning methods on the fine-tuned model, and (4) evaluation across multiple metrics.

4.1 Base Language Model↩︎

We select LLaMA 3-8B as the base language model, following established practices in medical AI [14]. We perform continued pre-training on 8,820 patient notes using standard next-token prediction, followed by instruction fine-tuning on 52,920 QAs (see Appendix 11 for hyperparameter details). The resulting fine-tuned model serves as the baseline for all unlearning experiments.

4.2 Unlearning Methods↩︎

We evaluate four top-performing unlearning methods from OpenUnlearning [10], covering diverse algorithmic families. All methods operate on the forget set \(\mathcal{D}_f\) (patients to unlearn) and the retain set \(\mathcal{D}_r\) (patients to preserve), applied to the fine-tuned model. Following the instruction fine-tuning configuration, all unlearning methods are trained with forget/retain patient notes and their QAs as input:

RMU [11] steers internal representations of forget set inputs toward random targets at intermediate layers while preserving retain set representations, avoiding output-level gradient manipulation. In our experiments, steering is applied at layer 7, consistent with the OpenUnlearning default values.

GradDiff [4] jointly maximizes the loss on \(\mathcal{D}_f\) and minimizes the loss on \(\mathcal{D}_r\) via the objective \(\mathcal{L}_{\text{GD}} = -\mathcal{L}(\mathcal{D}_f, w) + \mathcal{L}(\mathcal{D}_r, w)\), where \(\mathcal{L}(\mathcal{D}_f, w)\) is implemented as gradient ascent on the forget set negative log-likelihood. Rather than fine-tuning the full model, we perform layer-selective unlearning: all parameters are frozen except those in layer 7. Freezing all other layers limits parameter drift on \(\mathcal{D}_r\), reduces catastrophic forgetting and training cost.

KL-Min(imization) [5] uses the same GradDiff forget objective and the same layer-7 parameter mask. KL-Min applies gradient ascent on \(\mathcal{D}_f\) while regularizing \(\mathcal{D}_r\) outputs to match the original model via KL divergence, preventing model collapse.

SimNPO [28] treats forgotten samples as negative preferences without reference-model dependency, addressing bias issues in standard NPO [6]. Like GradDiff and KL-Min, we update only layer 7.

5 Evaluation Metrics (Figure 1.c)↩︎

To assess the trade-off between knowledge removal and utility preservation, we use four metrics from [7] and one from [10]. We also introduce a novel disease-informed leakage evaluation metric.

5.1 Disease Keyword Leakage Evaluation↩︎

Standard unlearning metrics measure semantic and lexical similarity to references, but do not test whether disease-specific terminology about forget patients still surfaces in outputs. We therefore introduce a leakage metric based on disease-specific token matching.

5.1.0.1 Keyword Extraction

For each QA instance, we build a disease-specific keyword set from the note, question, and answer. We parse each field with scispaCy [29] and collect biomedical named entities and noun chunks, dropping generic clinical terms (e.g., patient, history). Relevance to diseases is scored lexically against curated seed lists for Cancer (e.g., chemotherapy, metastasis), Infectious/Immune (e.g., sepsis, autoimmune), and Cardiovascular (e.g., stent, troponin). Complete seed lists and generic terms are detailed in Appendix 12.

For a candidate phrase \(p\) and seed \(s\), we assign lexical relevance \(\mathrm{rel}(p)\in[0,1]\) using exact or partial phrase matching. Each candidate receives a weight \(s_i\) based on its frequency in that field. We then merge the top K keywords from the notes and all surviving keywords from QAs.

5.1.0.2 Leakage Score (LS)

Let \(y\) be the model’s generated answer. Leakage is the weighted fraction of keywords that reappear in \(y\) as a contiguous token subsequence, \[\mathrm{LS}(y)= \frac{\sum_i s_i \cdot \mathbb{1}\!\left[k_i \in_{\mathrm{tok}} y\right]}{\sum_i s_i} \in [0,1],\] where \(\mathbb{1}[k_i \in_{\mathrm{tok}} y]\) is \(1\) if the keyword \(k_i\) occurs consecutively in \(y\) after tokenization (case-insensitive, alphanumeric only). We report \(\mathrm{LS}\) as a percentage (mean over forget set QAs with at least one keyword). Higher values indicate more reproduction of disease-specific terms from the note.

5.2 Established Metrics↩︎

5.2.0.1 Cosine Similarity (CS)

We measure semantic similarity using PubMedBERT embeddings [30].

5.2.0.2 Entailment Score (ES)

We employ PubMedBERT-MNLI-MedNLI [31] to verify factual consistency between model responses and reference answers.

5.2.0.3 Syntactic Overlap

We calculate ROUGE-L recall and position-aligned token Exact Memorization (EM) [10] to track lexical retention.

5.2.0.4 Token Entropy (TE)

We monitor normalized unigram entropy [7] to identify instances of model collapse or repetitive output (higher indicates more diverse output).

We utilize the harmonic mean (\(H\)) to compute primary performance indices. This ensures that a model cannot achieve high scores by over-optimizing for a single metric while failing in others:

5.2.0.5 Model Utility (MU)

Measures the preservation of medical knowledge in the retain set, \[MU = H(\text{ROUGE, EM, CS, TE, ES})\]

5.2.0.6 Answer Forget Efficacy (AFE)

Quantifies unlearning success on the forget set by aggregating the inversion of quality metrics (\(1 - \text{score}\)), \[AFE = H(1-\text{ROUGE, } 1-\text{EM, } 1-\text{CS, } 1-\text{ES})\]

Following [7], we exclude TE from this aggregate since it measures how repetitive the text is, which is unrelated to whether facts were actually forgotten. Instead, we report degradation score.

5.2.0.7 Degradation Score (DS)

To detect gibberish output, we report a DS,

\[\text{DS} = \frac{|\{x \in \mathcal{D}_f : TE(x) = 0\}|}{|\mathcal{D}_f|},\] which is the fraction of forget set generations with zero mean TE. These correspond to fully repetitive token outputs; a high DS indicates output collapse rather than knowledge removal (Table 2).

6 Results and Analysis↩︎

We evaluate 4 unlearning methods across two patient selection strategies: random patient-level and disease-level. For all experiments, MU is measured on the retain set, and AFE is measured on the forget set. We expect the MU for unlearned models to be close to the baseline MU, and the AFE to exceed the baseline AFE. MU of the holdout set is measured on the unseen QAs of retain patients, and AFE is measured on unseen QAs of forget patients. We report results separately for factual and reasoning QAs.

6.1 Do Models Forget Random Patients Regardless of Their Disease Profile?↩︎

Figure 2 presents random patient-level results, examining whether unlearning methods can forget patient information across different forget set sizes from 5% to 25%. On training (seen) questions (solid bars), RMU and SimNPO maintain high MU (above 0.9 in factual and above 0.75 in reasoning QAs) and low AFE (< 0.15) across all forget set sizes, indicating ineffective unlearning in the random patient-level setting. KL-Min exhibits the opposite pattern: high AFE (above 0.9 for both QA types) but low MU (\({\sim}0.0\)), achieving forgetting by generating repeated tokens (Table  2) rather than selective erasure. GradDiff balances both metrics at 5–15% splits (MU \({\sim}0.9\) for both QA types) but collapses at 20–25%, similar to KL-Min, losing model utility as more patients are forgotten.

Holdout QAs (hatched bars) yield much lower MU baseline values than train QAs (\({\sim}0.6\) vs \({\sim}1.00\) for factual and \({\sim}0.2\) vs \({\sim}0.95\) for reasoning) since these QAs are unseen. The large gap between factual and reasoning holdout MU values suggests that unlearned models recall direct facts from the note but fail to reason from it. Holdout AFE also remains low for RMU and SimNPO in both factual and reasoning QAs and close to baseline. This suggests that models struggle with novel QAs, creating a false sense of forgetting. Also, it means that unlearned models forget specific QAs from training data rather than truly erasing learned patient information (notes), which remains accessible via alternative questions.

3pt

6.2 Does Forgetting Patients with the Same Disease Differ from Forgetting Random Patients?↩︎

Analysis in 6.1 showed that model unlearning for forgetting random patients either collapses (high AFE and low MU) or is ineffective (high MU and low AFE). We now examine whether this behavior changes when forgotten patients share the same disease.

The evaluation of the disease-level (Figure 3) reveals a fundamentally different pattern from the random-patient unlearning. We plot forget set AFE versus retain set MU for the three most populated diseases (Cancer, Infectious/Immune, Cardiovascular) across different forget set sizes. Perfect unlearning appears in the top-right corner (high MU, high AFE). Unlike random patients, patients sharing the same disease are interconnected through shared medical characteristics. KL-Min and GradDiff behave similarly across all three diseases. Both achieve moderate MU (\(\sim\)​0.4–0.7) while preserving high AFE (\(\sim\)​0.90–0.97). The pattern is noisy and non-monotonic for both QA types.

For RMU and SimNPO, MU also remains high across all three diseases, but AFE now increases with forget set size. For instance, RMU forgetting 5% of cancer patients achieves AFE below 0.2, but forgetting 25% raises AFE to \(\sim\)​0.6. So, forget set size strongly influences RMU and SimNPO’s unlearning effectiveness when patients share diseases. This occurs because removing more patients from the same disease progressively erodes the shared medical knowledge, making it harder for the model to answer questions about that disease.

In Figures 4 (a) and 4 (b), for RMU and SimNPO, MU on seen questions (averaged over factual and reasoning) for same-disease retained patients degrades substantially with forget set size: in the cancer cohort, both methods drop from \({\sim}0.83\) to \({\sim}0.37\), while in the infectious/immune cohort, RMU declines from \({\sim}0.87\) to \({\sim}0.62\) and SimNPO from \({\sim}0.84\) to \({\sim}0.53\). However, MU for out-of-disease patients remains high (\({\sim}0.90\)–\(0.95\)). GradDiff and KL-Min, by contrast, collapse to near-zero MU for same-disease retained patients, but maintain moderate MU (\({\sim}0.50\)–\(0.77\)) for out-of-disease patients, indicating that their unlearning is disease-targeted rather than a global model collapse.

Holdout evaluation in Figure 3 mirrors the trends observed for seen QAs: AFE increases with forget set size, and MU shows disease-dependent trajectories. MU on reasoning QAs is consistently lower than MU on factual QAs, reflecting the greater difficulty of clinical inference. Critically, holdout AFE values remain comparable to training AFE, unlike random patient unlearning, where holdout AFE values dropped to \(\sim\)​0.2. This indicates that disease-level patient unlearning operates at the patient-note level rather than just the QA level.

6.3 Do Models Reproduce Exact Medical Terms From Forgotten Patients?↩︎

We now examine a novel privacy criterion: disease keyword leakage. Even if a model has high AFE, leakage of disease-related keywords from patient notes constitutes a privacy violation, as it confirms the model has retained specific information.

Table 2 reports disease-specific keyword leakage rates on the forget set across disease categories and forget set sizes. GradDiff and KL-Min achieve near-zero leakage score (LS) (\(\sim\)​0.001–0.007) across all diseases and forget set sizes, but this occurs because these methods generate largely incoherent outputs (high degradation scores in Table  2) rather than because they have erased targeted information.

Disease-level forgetting shows a different pattern than random patient forgetting for RMU and SimNPO. For random patients, LS remains stable at \(\sim\)​14–15% (factual) and \(\sim\)​25% (reasoning) across all forget set sizes for RMU and SimNPO, consistent with the flat AFE patterns observed in Figure 2. In contrast, for RMU and SimNPO, disease-level forgetting shows declining leakage as forget set size increases, with degradation remaining near zero across all diseases. For example, for cancer, RMU’s reasoning LS drops from 24.6% (5%) to 12.7% (25%), and SimNPO’s drops from 24.1% to 11.6%. This mirrors the improving AFE trend in Figure 3, confirming that removing more patients from the same disease progressively erodes the shared medical knowledge, reducing leakage of medical terminology.

7 Conclusion↩︎

We introduced AMNESIA, a large-scale open source medical unlearning benchmark with 70,560 QAs from 8,820 patients, featuring random patient-level and disease-level splits. AMNESIA reveals the limitations of existing unlearning methods. For random patient-level splits, unlearning methods either fail to forget effectively or achieve forgetting through model collapse. Disease-level forgetting improves as the forget set grows, but at the cost of model utility on retained same-disease patients. Across both settings, clinical reasoning remains harder than factual recall. Holdout evaluation further shows that models forget specific QAs rather than the patient notes, since the underlying information remains accessible through alternative unseen questions. Our disease keyword leakage analysis also shows the importance of having a domain-specific question-answering evaluation metric in unlearning settings. Together, these findings highlight the need for medical-specific unlearning techniques that erase patient information without damaging shared medical knowledge and establish AMNESIA as an essential benchmark for advancing the field.

Limitations↩︎

Our evaluation of forgetting relies on answer-level metrics (AFE, MU) and exact-match keyword leakage. A model with high AFE may still encode patient information recoverable through paraphrased queries, and our exact-match leakage metric does not catch paraphrased reproductions of forgotten terminology. Therefore, what fraction of a patient’s information is truly removed remains difficult to quantify.

All experiments use a single base model (LLaMA 3-8B), and gradient-based unlearning at layer  7 to match the RMU configuration from OpenUnlearning.

We partition patients into 11 single disease categories, but real clinical conditions exist on a spectrum, and comorbidities are common.

Ethical Considerations↩︎

AMNESIA is built from publicly available, de-identified clinical notes and will be released without any restrictions on the source data.

Our central finding is that unlearning can coexist with substantial keyword leakage, and new questions recover information from supposedly forgotten patients. Deploying any of the methods we evaluate under the assumption that they fulfill a right-to-be-forgotten request would create a false sense of privacy compliance. We further show that forgetting a critical mass of patients sharing a disease degrades model utility for retained patients with the same condition, meaning batch deletion requests could compromise care quality for patients who never requested deletion. None of the models in this work is validated for clinical use.

Acknowledgments↩︎

We thank our human annotator for volunteering to do our human-in-the-loop evaluation.

We also acknowledge the use of Cursor ⁴ for coding assistance and Claude (Anthropic) ⁵ for writing assistance in preparing this manuscript.

Appendix↩︎

8 Comparison to Existing Benchmarks↩︎

9 Prompts Used for Data Generation↩︎

This appendix provides the complete prompts used in our automated data generation pipeline.

9.1 Disease Category Assignment Prompt↩︎

We used the following prompt with GPT-5.4-nano to assign disease categories to patient cases based on article titles:

{
  "article_id": "<article_id>",
  "article_title": "<article_title>",
  "category": "<one top-level category from the taxonomy>",
  "short_reason": "<brief reason using title keywords>"
}

9.2 Question-Answer Generation Prompt↩︎

We used the following prompt with GPT-5-nano to generate question-answer pairs for each patient case:

10 Data Statistics↩︎

11 Training Hyperparameters↩︎

We perform pre-training on patient clinical notes to adapt LLaMA 3-8B model to medical language and terminology. Patient notes are tokenized using the LLaMA 3 tokenizer with a maximum sequence length of 2048 tokens and then packed to maximize GPU utilization. We employ distributed training across 8 AMD MI250X GPUs. Gradient checkpointing is enabled to reduce memory consumption. Unlearning also occurred for 3 epochs, and all experiments are from the last checkpoint of each unlearned model (the same as finetuning).

During instruction fine-tuning, we use the following template to structure each training example:

12 Leakage Score (LS)↩︎

We introduce a disease-specific keyword extraction pipeline to detect whether unlearned models continue reproducing specific medical terminology from forget patients. This provides fine-grained evaluation beyond aggregate metrics.

12.1 Extraction Pipeline↩︎

For each patient in the dataset, we extract disease-relevant keywords from three text sources: patient notes, questions, and answers. The pipeline uses scispaCy [29] medical NLP models to identify biomedical entities and noun chunks. Specifically, we use en_core_sci_lg for patient notes, a large English biomedical model trained on scientific literature with word2vec embeddings that can process arbitrarily long texts without token-length restrictions. For questions and answers, the same model is used by default, though the pipeline supports alternative models such as en_core_sci_scibert (a BERT-based transformer model with higher run-time and a 512-token limit). We focus on three major disease categories: Cancer, Cardiovascular, and Infectious or Immune Diseases.

12.1.0.1 Disease-Specific Seed Terms

For each disease category, we maintain a curated list of seed terms (Table 5) covering core terminology, conditions, biomarkers, and treatments. These seed lists were curated by prompting Claude (Anthropic) to generate medical terminology for each disease category, followed by manual review and validation by the authors. We also filter out common clinical terms that lack patient-specific diagnostic value.

12.1.0.2 Candidate Extraction

From each scispaCy-parsed document, we extract:

1. Named entities identified by the biomedical entity recognizer

2. Noun chunks from the dependency parser

3. Filtering: remove common clinical terms, phrases \(<\) 3 characters, and phrases \(>\) 6 words

12.1.0.3 Relevance Scoring

Each candidate phrase receives a lexical disease relevance score based on its overlap with seed terms:

• Exact match: score = 1.0

• Multi-word phrase containment: score = 0.9 (e.g., “lung cancer” contains seed “lung cancer”)

• Partial word overlap: score = 0.0–0.6 (proportional to Jaccard similarity)

Phrases scoring below 0.5 are filtered. The final score combines relevance with frequency: \(\text{score} = \text{relevance} + \min(\text{freq}, 5) \times 0.05\).

12.1.0.4 Top-K Selection

For patient notes, we select the top-5 highest-scoring keywords after removing redundant longer phrases that contain shorter selected keywords (e.g., keep “lung cancer”, remove “small tumor lung cancer”). For questions and answers, we extract all keywords meeting the relevance threshold without top-k limitation, as these texts are shorter and contain fewer candidates.

12.2 Keyword Leakage Computation↩︎

For each model prediction \(y\) on a forget-set question, we compute the weighted keyword leakage score: \[\text{Leakage}(y) = \frac{\sum_{i} s_i \cdot \mathbb{1}[k_i \in y]}{\sum_{i} s_i}\] where \(k_i\) are the extracted keywords from the corresponding patient (note + question + answer), \(s_i\) are their relevance scores, and \(\mathbb{1}[k_i \in y]\) indicates the exact phrase match detected via tokenized n-gram comparison. The prediction and keywords are lowercased and tokenized (alphanumeric tokens only), then we check if each keyword appears as a contiguous sequence in the prediction tokens.

This metric quantifies the extent to which the model reproduces disease-specific medical terminology associated with forget patients. Algorithm 5 shows the entire pipeline.

13 Human-in-the-Loop Validation of Disease Category Labels↩︎

This appendix details the LLM-as-judge panel used to verify disease category assignments in AMNESIA, together with the human validation study used to verify the panel itself.

13.1 Panel Design and Models↩︎

We assembled a three-judge panel with deliberately heterogeneous families to limit single-model bias:

• MedGemma-27B-IT (google/medgemma-27b-it): A 27B-parameter medical-specialized instruction-tuned model by Google DeepMind, built on the Gemma 3 backbone and trained on clinical text, medical QA pairs, and EHR data. It achieves 87.7% on MedQA, making it one of the strongest open-weight models for clinical reasoning and medical question answering.

• Qwen3-32B (Qwen/Qwen3-32B): A 32.8B-parameter open-weight model by Alibaba released under Apache 2.0, featuring a hybrid architecture that switches between extended chain-of-thought thinking mode and fast non-thinking mode. It supports a 128K-token context window with strong performance in mathematics, coding, and agentic reasoning across 119 languages.

• GPT-5 mini (gpt-5-mini-2025-08-07): A closed-source proprietary model by OpenAI and the cost-efficient variant of the flagship GPT-5 (released August 2025), supporting a 272K-token context window with vision, function calling, and web search. It delivers approximately 45% fewer hallucinations than GPT-4o while targeting high-volume, low-latency workloads requiring reliable multi-step reasoning.

Each judge received the article title and assigned category and was required to return a JSON object with three fields: (i) a 1–2 sentence rationale, (ii) a verdict in \(\{\textit{Supported}, \textit{Unsupported}, \textit{Ambiguous}\}\), and (iii) the category the judge believed best fit the title, drawn from {Cardiovascular, Cancer, Infectious/Immune Diseases, Unclear}. The final per-title label was the majority verdict across the three judges; ties (one judge each on three different labels) were recorded as Disagree.

13.2 Panel Results on 900 Titles↩︎

The panel evaluated all 900 sampled patient titles spanning Cancer (300 titles), Cardiovascular (300 titles), and Infectious/Immune Diseases (300 titles). Table 6 shows the verdict distribution across models. Table 7 breaks the panel verdict down by assigned disease, and Table 8 reports pairwise inter-judge agreement.

13.3 Human Validation Protocol↩︎

To evaluate the panel, one biology student manually verified the aforementioned 900 titles. For each title, the annotator was shown the title and the assigned category and answered a binary question: “Is the assigned category label supported by the title?”(Yes/No). The annotator was blinded to all LLM judge outputs and to the panel verdict at annotation time. The resulting human label distribution was \(870\) Yes and \(30\) No.

We aligned the human Yes/No labels with the LLM Supported/ Unsupported scheme and computed agreement statistics per judge against the human. We report a confusion matrix per judge (Table 12). For each judge’s confusion matrix, that judge’s Ambiguous verdicts were excluded from the denominator (1 for MedGemma, 8 for Qwen, 3 for GPT-5-mini). For the panel matrix, the 1 Ambiguous and 2 Disagree panel outcomes were excluded, yielding 897 rows.

14 LLM-as-Judge Validation of QA Pairs↩︎

We sampled two QAs (one factual and one reasoning) per 500 randomly selected patients, resulting in \(1{,}000\) (patient note, question, answer) triples from the AMNESIA training pool. We submitted each to the same three-judge panel used for disease-label validation (Appendix 13): MedGemma-27B, Qwen3-32B,and GPT-5-mini.

14.1 Judge Prompt↩︎

Each judge was given the full patient note alongside the candidate question and answer, and asked for a strict binary verdict together with a brief rationale. The verbatim prompt is below.

14.2 Panel Results↩︎

Table 13 summarizes verdicts at the judge and panel level over the \(N=1{,}000\) triples, and Table 14 reports pairwise inter-judge agreement and Fleiss’ \(\kappa\).

References↩︎