1 Introduction↩︎

Modern language models increasingly rely on hybrid architectures that combine parametric knowledge with external memory. Limited Memory Language Models (LMLMs) are a prominent example of this paradigm, explicitly separating linguistic competence encoded in model parameters from factual knowledge stored in an external database [1]. As illustrated in Figure 1, an LMLM retains the linguistic competence of a standard language model but routes factual recall through an external database rather than holding it in parameters. This design enables deletion-based unlearning, where removing entries from the database is intended to eliminate access to specific facts without requiring retraining [1]. Such capabilities are particularly important for applications involving data governance, privacy, and model editing.

However, it remains unclear whether deletion in these systems truly removes knowledge. Existing evaluations of forgetting typically measure whether a model produces the correct answer before and after deletion, but do not distinguish the underlying mechanism of post-deletion correctness [1]. A model may still answer correctly due to residual parametric memory, alternative retrieval paths, or semantically related matches in the external database. As a result, current metrics cannot determine whether knowledge has been successfully externalized or whether it persists internally in the model.

In this work, we propose a causal auditing framework for analyzing forgetting in LMLMs. Our approach introduces three controlled interventions, FULL, where the database is intact and retrieval is enabled, DEL-ON, where relevant entries are deleted while retrieval remains enabled, and DEL-OFF, where the same deletion is applied and retrieval is disabled. By comparing model behavior across these conditions, we isolate the contribution of external retrieval and quantify residual internal knowledge. This enables a decomposition of post-deletion behavior into parametric leakage, retrieval-mediated correctness, and retrieval artifacts.

We implement this framework using facts drawn directly from the LMLM database [2] and other self-made databases¹, and apply a verified deletion procedure that removes all canonical and alias-equivalent representations of a fact. We further log retrieval traces during inference to attribute model outputs to explicit database evidence. This controlled setup provides a principled method for auditing whether knowledge has been successfully externalized in LMLMs.

2 Related Work↩︎

Our work sits at the intersection of three lines of research. First, retrieval-augmented language models pair parametric representations with external memory accessed at inference time [3], [4], and the LMLM architecture we audit [1] is a recent extension that pre-trains the model to limit internal storage of factual knowledge so that retrieval becomes the primary factual channel. This line of work shows that external retrieval can improve factual accuracy [5], but leaves open the question of whether knowledge is genuinely externalized or partly retained in parameters [6].

Second, knowledge-editing methods such as ROME [7] and MEMIT [8] address the same question from the opposite direction: rather than externalizing knowledge, they locate and modify parametric associations in place [9]. LMLMs aim for the cleaner separation that ROME and MEMIT bypass, which makes the LMLM setting a natural place to ask whether the separation actually holds.

Third, work on machine unlearning has probed whether specific training data can be removed from a model after the fact [10], and has tended to find that residual traces are difficult to fully eliminate [11], [12]. Our audit takes up this question in the LMLM setting, where the question becomes tractable in a new way: because the database is the intended factual store, deletion can be applied at inference time and the resulting behavior decomposed into a parametric channel and a retrieval channel separately. The framework we develop in Section 3 is, to our knowledge, the first to make that decomposition per-fact and to attribute surviving correctness to specific retrieval candidates.

3 Method Setup: Causal Audit Framework↩︎

We evaluate whether Limited Memory Language Models actually forget facts when those facts are removed from the external database. Each fact is represented as a subject–relation–object tuple, such as Geri Halliwell – Famous For – Spice Girls. At inference time, retrieval provides the model with relevant database entries as additional context before the model generates an answer. Thus, when retrieval is enabled, the model is not reading the entire database directly; instead, a retrieval step selects relevant facts and inserts them into the model’s input context. Appendix 10.1 gives examples of the prompt formulations used in the experimental grid.

Our audit compares three intervention states. In FULL, the target fact remains in the database and retrieval is enabled. This measures normal database-supported accuracy. In DEL-ON, the target fact is deleted but retrieval remains enabled. This tests whether the answer can still be recovered through alternative database entries, aliases, semantically related facts, or retrieval artifacts. In DEL-OFF, the target fact is deleted and retrieval is disabled. This isolates parametric recall, since the model must answer without retrieved evidence.

We use these interventions to decompose post-deletion correctness into three mechanisms. Parametric leakage occurs when the model answers correctly in DEL-OFF, indicating that the deleted fact may still be stored in the model parameters. Retrieval-mediated correctness occurs when the model is correct in DEL-ON but not in DEL-OFF, indicating that retrieval helped recover the answer after deletion. Retrieval artifacts occur when the model produces the correct answer even though the deleted fact is not directly available as retrieved evidence. Formally, for a fact \(f=(s,r,o)\) and intervention condition \(c\), let \(Y(f,c)\) denote the normalized model prediction. We define parametric leakage as \[L(f)=\mathbb{I}[Y(f,\mathrm{\small DEL-OFF})=o],\] which indicates whether the model can recover the deleted fact without retrieval.

We define retrieval-mediated correctness as \[R(f)=\mathbb{I}[Y(f,\mathrm{\small DEL-ON})=o \wedge Y(f,\mathrm{\small DEL-OFF})\neq o],\] which captures cases where retrieval enables the correct answer after deletion.

Across a fact set \(\mathcal{F}\), the empirical leakage rate is \[\hat{L}=\frac{1}{|\mathcal{F}|}\sum_{f\in\mathcal{F}} L(f) = \frac{1}{|\mathcal{F}|}\sum_{f\in\mathcal{F}} \mathbb{I}[Y(f,\mathrm{\small DEL-OFF})=o].\] Analogously, we estimate retrieval-mediated correctness by averaging \(R(f)\) over \(\mathcal{F}\).

4 Experimental Setup↩︎

In addition to the released LMLM database [2], we developed databases for countries, politicians and sports. Within each of these three themes, we built four database variants (Base, Alias, Noise, and Collision), for a total of twelve custom databases. These four variants are designed to stress-test a distinct mechanism by which a deleted fact could remain accessible after canonical removal. Base contains only the canonical \((s, r, o)\) triplet for each fact, leaving retrieval with no alternative path. Alias stores the same fact only under aliased subject and relation forms, with no canonical entry, and tests whether alias-closure deletion catches every surface realization. Noise augments Base with decoy triplets that route to the same object through paraphrased subjects (e.g., Government of United States \(\rightarrow\) Seat of Government \(\rightarrow\) Washington, D.C.), probing whether retrieval can recover the deleted answer via near-neighbor paraphrases. Collision augments Base with near-miss triplets that share the subject but route to a different object (e.g., United States \(\rightarrow\) Largest City \(\rightarrow\) New York City), probing whether retrieval drifts onto a confusable neighbor and returns a plausible but incorrect answer. Table 2 outlines an example of these four variants for the politician domain. We evaluate each target fact under six prompt formulations: direct questions, paraphrased questions, contextual questions, cloze prompts, continuations, and few-shot prompts; examples are provided in Appendix 10.1.

Crossing the six prompt sets, the three intervention states, and the thirteen databases yields a fully crossed evaluation grid in which every target fact is scored on matched inputs across all conditions. For each cell we record exact match, token-level precision, recall, and F1, together with the cross-state quantities \(L(f)\), \(R(f)\), and the retrieval artifact rate; retrieval traces are logged at every FULL and DEL-ON call so that post-deletion correctness can be attributed to explicit database evidence rather than implicit model behavior. Figure 2 outlines this pipeline.

For each target fact, we construct the deletion set by enumerating the canonical triplet and all alias-equivalent triplets whose subject, relation, and object match the target under our alias mapping. We then remove this set from the database and verify deletion by checking that no retained triplet is gold-equivalent to the target. The same prompt is evaluated under FULL, DEL-ON, and DEL-OFF, and the generated answer is normalized before scoring against the gold object. During FULL and DEL-ON runs, we also save the retrieved candidates returned to the model. A DEL-ON answer is counted as a retrieval artifact when the normalized answer matches the gold object but none of the retained retrieval candidates is gold-equivalent to the deleted fact.

5 Results↩︎

Our reference point throughout the results is the aggregate evaluation reported in the original LMLM paper [1], which uses a single FactScore [13] drop when the database is disabled and a single TOFU [14] forget-quality \(p\)-value per unlearning step. Both quantities collapse the post-deletion behavior of every fact into a single number, so they cannot indicate which channel a surviving correct answer came through. Our framework refines that aggregate signal into per-fact attributions across the three intervention states, and we read each result below against the corresponding aggregate quantity from the original paper.

We run the LMLM under all three interventions on every cell of the prompt \(\times\) database grid. Across the \(78\) (prompt file, database) cells, this yields \(12{,}228\) paired (DEL-ON, DEL-OFF) evaluations together with an equal number of FULL baselines, for a total of \(36{,}684\) model completions. All reported quantities are count-weighted averages over fact-paired groups, so a prompt file with more target facts contributes proportionally to the aggregate. We confine the present section to direct observations and defer cross-figure interpretation to Section 6.

We begin with the variant-level view. Figure 3 attributes DEL-ON correctness to its three components separately for the four custom variants and the released LMLM database. In particular, the retrieval artifact bar isolates cases in which DEL-ON returns the gold object even though no gold-equivalent triplet appears among the retrieval-trace candidates retained after deletion.

We next turn to the prompt-style axis. Figure 4 reports the same decomposition averaged over all four custom variants and the released LMLM database, so that variant identity is collapsed and only the effect of prompt formulation remains. As before, parametric leakage stays near zero across the panel, although a small red residue is visible under direct and few-shot prompts and is essentially absent under the others. The combined \(L(f) + R(f)\) stack varies from \(6.1\%\) on cloze prompts to \(9.9\%\) on direct questions, while the orange artifact bar matches the green \(R(f)\) bar within rounding in every prompt style.

Having attributed DEL-ON correctness, we now widen the lens to all three intervention states. Figure 5 reports weighted token F1 by prompt style and state, so that the post-deletion residual can be compared against both the FULL baseline and the retrieval-disabled DEL-OFF condition. Under FULL, F1 spans a wide range, from \(\sim\)​17% on few-shot prompts to \(\sim\)​57% on continuations. Under DEL-ON, by contrast, the same prompt styles compress into a narrow band of roughly \(7\)–\(10\%\). As a result, the largest FULL-to-DEL-ON drop falls on continuations (about \(48\) points), while the smallest falls on few-shot prompts (about \(10\) points). Finally, under DEL-OFF, F1 is visually indistinguishable from zero in every prompt style.

Figure 6 mirrors this three-state cut along the database-variant axis and uses exact match in place of token F1, so that the same trajectory can be read off the variants we constructed for the audit. Under FULL, accuracy is similar across Base, Alias, and Noise, all clustered near \(33\%\), and noticeably higher on Collision and the released LMLM database, both near \(46\%\). The corresponding FULL-to-DEL-ON drop is about \(30\) points on Base, \(27\) on Alias, \(39\) on Collision, \(19\) on Noise, and \(45\) on the released LMLM database. Under DEL-OFF, exact match remains at or near zero in every variant.

Tables 5 and 6 in Appendix 14 report the full per-cell numbers underlying Figures 3–6.

6 Analysis↩︎

Sections 6.1 and 6.2 establish the headline decomposition of post-deletion correctness into a parametric component and a retrieval-graph component. Sections 6.3 and 6.4 then ablate the two axes of the audit grid that the framework holds fixed, the retrieval-graph topology and the prompt formulation, to identify which axis controls the surviving residual. Section 6.5 closes the analysis with qualitative examples of the three failure modes the decomposition predicts.

6.1 Parametric externalization at the per-fact level↩︎

Our headline finding is that parametric leakage \(\hat{L}\) is near zero across the audit. The red \(L(f)\) component contributes a vanishingly small share of the \(L(f) + R(f)\) stack in every variant (Figure 3) and in every prompt style (Figure 4), and weighted DEL-OFF performance is visually indistinguishable from zero in every cell of Figures 5 and 6. At the level of model parameters, the LMLM checkpoint behaves close to a model that had never seen the fact, in contrast to standard language models, which are known to memorize and expose training-data content even when each example is seen only a handful of times [12]. The original LMLM paper [1] provides only an aggregate version of this result. Their Table 9 shows that disabling the database globally drops FactScore [13] by roughly \(19\) points, and their §5 explicitly labels that finding as “preliminary evidence” that LMLM does not memorize. Our audit converts the same intuition into a per-fact statement: across \(12{,}228\) alias-closure deletions, the parameters return the deleted answer at a rate of \(\hat{L} = 0.11\%\). Our per-fact result is consistent with, and adds resolution to, the preliminary aggregate finding in the original LMLM paper [1].

6.2 The residual lives in the retrieval graph↩︎

The residual that survives deletion is, in our audit, almost entirely attributable to the retrieval graph rather than to the parameters. The signature observation in Figures 3 and 4 is the near-equality of the green \(R(f)\) bar and the orange retrieval artifact bar: the two match within rounding for every variant and every prompt style. The following proposition shows that part of this equality is structural, which in turn clarifies what the figures empirically measure.

Proposition 1 reframes what the two attribution figures actually measure. Given the per-fact \(\hat{L} \approx 0\) established in Section 6.1, the empirical near-equality \(\hat{R} \approx \hat{A}\) is a confirmation that our alias-closure deletion procedure is complete in practice; the empirical content of the figures is therefore not the equality itself but the topology-dependent magnitude of the residual. Retrieval-mediated correctness almost never coincides with a gold-equivalent retained candidate. Instead, it coincides with near-neighbor candidates that share embedding-space similarity with the deleted entry. This decomposition is not present in the original LMLM paper. Their TOFU [14] evaluation reports a single forget-quality \(p\)-value per unlearning step and therefore aggregates over both gold-equivalent retrieval and near-neighbor retrieval. The retrieval-trace introspection used here separates the two and shows that, on retrieval graphs containing paraphrastic decoys, the second mechanism dominates. A standard FULL versus DEL-ON comparison would have shown only that some deleted facts are still answered correctly; the audit shows that those facts are not remembered, they are reconstituted at retrieval time.

6.3 Ablation: Retrieval-graph topology controls the residual↩︎

We ablate the retrieval-graph topology by holding the model, deletion procedure, and prompt distribution fixed and varying only the database variant across Base, Alias, Collision, Noise, and the released LMLM database; the dependent variable is the post-deletion residual \(L(f) + R(f)\). Reading down Figure 3, the residual rises from \(3.7\%\) on Base, to \(6.2\%\) on Alias, to \(7.2\%\) on Collision, and to \(13.6\%\) on Noise, before falling to \(0.7\%\) on the released LMLM database. This ordering tracks the retrieval-graph topologies described in Section 4: variants in which more surviving entries point to the gold object after alias-closure deletion produce larger residuals, with Noise as the worst case by construction and Base as the cleanest among the custom databases. The original paper [1] reports unlearning on a single retrieval graph, the annotated TOFU corpus [14], and on that graph achieves “ideal forgetting” with \(p > 0.05\). By varying the retrieval graph along controlled topologies, we show that the same architecture and the same alias-closure deletion can produce a DEL-ON residual anywhere from \(0.7\%\) to \(13.6\%\). The published TOFU result therefore measures the model on a benign retrieval graph [1], [14]; it does not, by itself, characterize what happens when the graph contains paraphrastic decoys.

6.4 Ablation: Prompt formulation moves baselines more than residuals↩︎

We ablate the prompt formulation by holding the model, deletion procedure, and database distribution fixed and varying only the prompt set across the six formulations introduced in Section 4; the dependent variables are the FULL baseline accuracy and the post-deletion residual under DEL-ON. Under FULL, prompt style produces a wide performance spread (token F1 between \(\sim\)​17% and \(\sim\)​57% in Figure 5, exact match between \(\sim\)​33% and \(\sim\)​46% in Figure 6), reflecting how easily each prompt style elicits a structured lookup. Under DEL-ON, that spread collapses: token F1 compresses into a \(7\)–\(10\%\) band, and the FULL-to-DEL-ON drop is dominated by the FULL baseline rather than by any property of the prompt itself. Continuations lose the most because they had the most to lose; few-shot prompts lose the least for the same reason. The original paper’s TOFU and FactScore evaluations each use a fixed prompt template, so this kind of decomposition is not available there [1], [13], [14]. Our six prompt families show that, once the supporting evidence is removed, prompt formulation has limited independent control over how much of a deleted fact survives. What survives is determined almost entirely by the retrieval graph.

6.5 Error Analysis↩︎

The decomposition in Sections 6.1 and 6.2 predicts three qualitatively distinct ways a deleted fact can still be answered correctly: parametric leakage, near-neighbor retrieval reconstruction on Noise, and confusable-neighbor drift on Collision. Table 3 in Appendix 10.3 shows one inference-time trace for each, with the Noise and Collision rows drawn from the corresponding custom-database files and the parametric-leakage row drawn illustratively from a cell that empirically contains leakage cases. The traces make the population-level rates concrete: in the Noise case, alias-closure deletion removes the canonical triplet for the gold fact, but a paraphrastic decoy with a different subject phrasing survives and routes retrieval to the same object, so DEL-ON returns the gold answer with no gold-equivalent candidate in the trace; in the Collision case, deletion is complete and no surviving candidate routes to the gold object, but a near-miss triplet sharing the subject pulls retrieval onto a confusable neighbor and DEL-ON returns a plausible but incorrect answer; and in the parametric-leakage case, retrieval returns unknown under DEL-OFF yet the model still produces the gold answer from its parameters, an outcome that the \(\hat{L} = 0.11\%\) headline rate predicts to be rare and that appears, in the cells where it occurs, to concentrate on high-frequency entities.

7 Conclusion↩︎

The audit’s contribution is not a single number but a decomposition: a retrieval-disabled control isolates parametric recall from retrieval-side correctness, and the retrieval trace separates correctness anchored in gold-equivalent evidence from correctness reconstituted by near-neighbor candidates. Across our grid, the parametric channel is essentially empty in our audit: the LMLM checkpoint almost never returns a deleted answer when retrieval is disabled, and the residual that survives is, in our experiments, primarily a property of the retrieval graph rather than of the model. Within the scope of our audit, LMLM deletion within the model appears to be a clean operation; outside the model, deletion is only as complete as the alias-and-paraphrase closure used to construct it, and our Noise variant produces a \(13.6\%\) residual at zero parametric leakage precisely by exposing this gap. Prompt formulation does not appear to move this number independently: once the supporting evidence has been removed, what predominantly determines how much of a deleted fact survives is the topology of the retrieval graph rather than the question’s surface form.

8 Limitations and Broader Impact↩︎

Our audit has several scope limitations. All experiments use a single LMLM checkpoint, the publicly released \(382\)M-parameter LLaMA2-style model, with retrieval threshold \(0.6\), fuzzy-match top-\(1\) fallback, and greedy decoding, so findings may not generalize across model sizes, retrieval architectures, similarity thresholds, or decoding strategies [2]. The custom databases used to stress-test deletion are also small and topology-controlled, on the order of \(100\)–\(240\) facts per (domain, variant) cell, which is what makes paired comparison possible but is several orders of magnitude smaller than the \(54.6\)M-triplet released LMLM database [1]; the variant-level findings therefore describe how deletion behaves when scale is held fixed and topology is varied, rather than how scale itself interacts with topology. We restrict attention to entity-level atomic facts in English, evaluated under an automatic, alias-aware exact-match notion of correctness, so multi-hop or compositional reasoning, languages other than English, sampling-based decoding, and human-judged correctness all remain outside the present scope. Finally, our alias-closure deletion procedure removes only triplets that alias-match the target on subject, relation, and object simultaneously; the Noise variant is designed to expose what survives this closure, so its residual is a feature of the audit rather than a methodological shortcoming, but it does mean that low parametric leakage should not be read as a guarantee that any user-facing notion of “deletion” fully removes access to sensitive information.

For applications such as data-deletion compliance or factual editing, the practical lesson from the audit is that the unlearning boundary in this class of LMLM is drawn primarily by the database administrator rather than by the model. Sound governance therefore requires a closure procedure that extends beyond entity aliases to retrieval-graph paraphrases of the object.

9 Future Work↩︎

The most direct extension of this work is on the database side. Our results identify retrieval-graph topology as the dominant determinant of post-deletion residuals, which suggests building or preprocessing the database so that alias-closure deletion has fewer surviving routes. One concrete direction is an extended deletion closure that, in addition to alias-equivalent triplets, also removes any triplet whose retained-candidate embedding falls within a similarity radius of the canonical entry, so that paraphrastic decoys are caught at deletion time. A second direction is canonicalization at write time, in which aliases and paraphrastic forms are stored as pointers into a single canonical record rather than as independent triplets. Both approaches are directly testable within our framework: re-running the audit on the modified database and measuring whether \(R(f)\) and the retrieval artifact rate fall below their current ranges would tell us how much of the residual is recoverable through database design alone.

Several axes of the audit itself also remain open. We hold the retriever (all-MiniLM-L6-v2) and the similarity threshold (\(0.6\)) fixed throughout, so sweeping the threshold and replacing the retriever with sparse, dense, or hybrid alternatives would clarify how much of the observed \(13.6\%\) Noise residual is a property of the embedding model rather than of the LMLM architecture. Scaling the audit to larger LMLM checkpoints and to the full \(54.6\)M-triplet released database would test whether the per-fact zero parametric leakage we report holds at production scale. Extending the framework beyond entity-level atomic facts to multi-hop and compositional knowledge, and beyond automatic exact-match scoring to semantic or human-judged correctness, would close the gap between the audit and end-user notions of forgetting. Finally, the same intervention set can be applied to retrieval-augmented and edited parametric models more broadly, which would allow a head-to-head comparison of which architectural family achieves the cleanest deletion under matched retrieval pressure.

Acknowledgements↩︎

We thank Akshat Gupta (Ph.D. student, UC Berkeley) for ongoing research feedback and direction; Yilun Hua (Ph.D. student, Cornell University) for further research feedback and direction on the LMLM framework; and Marcel Roed (Ph.D. student, Stanford University) for early feedback on the project proposal. This work used computing resources provided by Berkeley Research Computing through the Compton Spectrometer and Imager (COSI) mission (NASA Small Explorers (SMEX) Program).

langley00?

10 Architecture↩︎

This appendix details the two design axes that define the prompt \(\times\) database grid used throughout the audit: the six prompt formulations applied to every target fact, and the four custom database topologies constructed to stress-test alias-closure deletion. Together these specifications fix the inputs to the \(6 \times 3 \times 13\) evaluation grid summarized in Section 4.

10.1 Prompt formulations↩︎

Each target fact is presented to the LMLM under six prompt formulations, designed to vary surface form while holding the underlying fact constant. The formulations span direct question-answering, paraphrased rewordings, contextually framed prompts, cloze-style completions, free-form continuations, and few-shot demonstrations. Their purpose in the audit is to separate the effect of question phrasing on FULL baseline accuracy from any independent effect on the post-deletion residual; in our experiments, prompt formulation moves the FULL baseline considerably more than it moves the DEL-ON residual (Section 6.4). Table 1 illustrates each formulation using a single fact drawn from the released LMLM database.

10.2 Custom database variants↩︎

Alongside the released LMLM database, we constructed twelve custom databases spanning three domains (countries, politicians, sports) and four topological variants (Base, Alias, Noise, Collision). The variants are intended to isolate distinct mechanisms by which a fact could remain recoverable after canonical removal: Base provides a clean baseline with no alternative routes; Alias probes whether alias-closure deletion catches every surface realization of the subject and relation; Noise probes whether retrieval can reconstitute the deleted answer through paraphrastic decoys that point to the same object; and Collision probes whether retrieval drifts onto a near-neighbor sharing the subject but routing to a different object. In our audit, the post-deletion residual tracks this topology ordering, with the largest residual concentrated in the Noise variant (Section 6.3). Table 2 illustrates the four variants using a single politician-domain example.

10.3 Qualitative examples of post-deletion failure modes↩︎

Table 3 shows one representative inference-time trace for each of the three failure modes identified by the decomposition. For each example we report the gold fact, the variant the trace was drawn from, the canonical triplet that was removed by alias-closure deletion, the highest-similarity retrieval candidate retained at inference time, and the model’s DEL-ON (or DEL-OFF, for the parametric-leakage case) output. The deleted triplets and the surviving retrieval candidates in the Noise and Collision rows are taken verbatim from the custom databases under data/custom_databases/countries/. The parametric-leakage row is drawn from the direct_questions \(\times\) Base (sports) cell, which contains \(2\) leakage cases out of \(100\) paired facts. The specific triplet shown is illustrative of the kind of high-frequency sports fact this cell contains rather than a verbatim trace from the audit logs.

11 Hyperparameters↩︎

This section lists the configuration constants that directly affect the LMLM’s outputs in our audit: the loaded checkpoint, the retrieval stack, and the decoding policy. Settings inherited from the upstream LMLM release [2] are noted as such in the description column. Experimental-grid axes (prompt formulations, database variants, intervention states) and evaluation plumbing are described separately in Appendix 10 and Section 4. Table 4 lists each performance-relevant hyperparameter, the variable name in source, a short description, and the value used.

12 LMLM Audit Identities↩︎

This section collects the small set of equations that are specific to the three-state LMLM audit. Each identity links the per-fact indicators \(L(f)\) and \(R(f)\) from Section 3 and the per-fact retrieval artifact indicator \(A(f)\) from Proposition 1 to the paired contingency that the McNemar test in Appendix 13.1 consumes. Throughout, \(\mathcal{F}\) denotes the set of paired facts (those evaluated under both DEL-ON and DEL-OFF), \(n = |\mathcal{F}|\), and the paired contingency is \[\begin{align} a &= |\{f \in \mathcal{F} : Y(f,\texttt{DEL-ON}) = o \;\wedge\; Y(f,\texttt{DEL-OFF}) = o\}|,\\ b &= |\{f \in \mathcal{F} : Y(f,\texttt{DEL-ON}) = o \;\wedge\; Y(f,\texttt{DEL-OFF}) \neq o\}|,\\ c &= |\{f \in \mathcal{F} : Y(f,\texttt{DEL-ON}) \neq o \;\wedge\; Y(f,\texttt{DEL-OFF}) = o\}|,\\ d &= |\{f \in \mathcal{F} : Y(f,\texttt{DEL-ON}) \neq o \;\wedge\; Y(f,\texttt{DEL-OFF}) \neq o\}|. \end{align}\] The intervention-condition variable \(c\) from Section 3 does not appear elsewhere in this section, so we reuse the symbol for the McNemar-style off-diagonal count without ambiguity.

12.1 Decomposition of DEL-ON correctness↩︎

Proposition 2 explains why the \(\hat{L} + \hat{R}\) stack reported in Figure 3 reads almost identically to the DEL-ON exact-match column of Figure 6: the gap is exactly \(c/n\), which never exceeds \(4/600 \approx 0.7\%\) in any of our variants. The proposition also pins down what \(\hat{L} + \hat{R}\) would equal in a hypothetical setting where retrieval frequently hurts correctness; in that setting \(c/n\) would no longer be negligible and the stack and the DEL-ON bar would diverge by exactly that amount.

12.2 Recovering the paired contingency from aggregated rates↩︎

Proposition 3 is what allows the aggregated cross-state and per-state metrics produced by the audit to be re-expanded into the \((a, b, c, d)\) table that the McNemar statistic in Appendix 13.1 consumes, without re-running the audit on a per-fact basis. The recovery is exact under the same coverage condition that makes the McNemar test well defined, namely that every paired fact has been evaluated under both intervention states.

13 Additional Results↩︎

13.1 Paired McNemar test for DEL-ON vs.DEL-OFF↩︎

Figure 7 reports the discordant-pair counts that drive the paired McNemar test for DEL-ON against DEL-OFF, broken out by database variant. Both states share the same alias-closure deletion of the target fact; the only difference is that DEL-ON leaves retrieval enabled while DEL-OFF disables it. The count \(b\) measures facts that retrieval rescues after the canonical entry is removed, and \(c\) measures the reverse transition in which disabling retrieval somehow recovers an otherwise-incorrect answer. The concordant counts \(a\) and \(d\) are absorbed into the marginal \(n\) and do not enter the test.

Across the four custom variants the asymmetry is unambiguous: \(b\) ranges from \(69\) on Base to \(418\) on Noise while \(c\) never exceeds \(4\), and all four reject the null at \(p < 0.001\) with \(\chi^2\) statistics between \(61\) and \(407\). The Released LMLM column points the same way (\(b = 4\), \(c = 0\) on \(n = 600\)), but the discordant counts are too small for the exact binomial test to reject at \(\alpha = 0.05\) (\(p = 0.125\)); we read this as a power limit rather than a contradiction, since the released-database evaluation has both fewer paired facts and a lower base rate of exact-match correctness.

Taken together with the headline rates in the main paper, the McNemar evidence reinforces the claim that retrieval-mediated correctness is a real paired effect rather than a coincidence at the population level. The relation \(b \gg c\) holds inside every custom variant, which leaves little room for a reverse hypothesis in which gains from retrieval are merely noise of equal magnitude flipping in both directions.

14 Numeric Results↩︎

References↩︎

---

1. Code and custom databases are available at https://github.com/raeesiarya/LMLMAudit.↩︎