1 Introduction↩︎

Large language models (LLMs) derive much of their capability from parametric knowledge acquired during large-scale pretraining [1], [2]. While this enables strong performance across tasks, it also contributes to hallucination—generating plausible but ungrounded content—particularly in settings where factual accuracy is critical [3]–[5].

A common mitigation strategy is to ground generation in external evidence, e.g., via retrieval-augmented generation (RAG) [6]–[8]. However, standard LLMs are still pretrained to memorize and reproduce large amounts of factual information, creating a mismatch: at inference time, models are expected to rely on provided context, but their pretraining encourages parametric recall and they often fail to fully utilize available evidence [9], [10].

In this work, we address this tension at its source, asking: Can we modify the pretraining signal such that models rely less on parametric knowledge and more on external evidence, without sacrificing reasoning ability? To this end, we introduce Knowledgeless Language Models (KLLMs), pretrained on corpora in which named entities are systematically anonymized. This removes a primary channel of entity-linked knowledge supervision while preserving the structural and linguistic properties of the data. The result is a simple pretraining intervention that imposes an inductive bias toward evidence-grounded reasoning, without requiring architectural changes.

Our empirical findings reveal a consistent pattern. First, KLLMs exhibit a near-complete collapse in closed-book factual recall, confirming strong suppression of parametric knowledge. Second, when relevant information is provided as context, KLLMs match or outperform standard LLMs on factual QA, fact verification, and commonsense reasoning. Third, in retrieval-grounded settings, KLLMs achieve substantial gains (up to 20–25% relative), particularly under incomplete or noisy evidence, and exhibit more reliable abstention behavior. Finally, KLLMs demonstrate improved calibration across multiple metrics, indicating better alignment between confidence and correctness.

These results suggest that pretraining objectives fundamentally shape the epistemic behavior of language models: suppressing entity-linked supervision encourages reliance on external evidence while preserving core reasoning capabilities. As a result, KLLMs are well-suited to complement retrieval-augmented and tool-based systems, where robustness to imperfect evidence and reliable uncertainty estimation are essential. We scale training up to 20B tokens and perform ablations on anonymization strength, continued pretraining, and inference-time anonymization. The observed trends remain stable, supporting anonymization as the key mechanism driving the effect.

Overall, we present a controlled study of how modifying the pretraining signal reshapes the balance between parametric knowledge and evidence-grounded reasoning, and introduce a simple yet effective approach for improving reliability in language models.

2 Knowledgeless Language Modelling↩︎

In order to develop an LLM that minimizes retention of real-world factual knowledge, we propose to preprocess pretraining data to markedly reduce the model’s propensity to memorize specific factual knowledge from the data. In particular, we anonymize named entities by replacing their mentions with placeholder tokens, preventing the model from learning to associate any factual information with the names of entities such as people, places, or events. We train models from scratch on this preprocessed data without modifying the model architecture. During inference, the same anonymization procedure is followed to anonymize both the query and the context (where applicable), and entity names are restored as a postprocessing step. While some early work on reading comprehension [11] employed a similar entity anonymization scheme during supervised training, we are not aware of previous work that applied this to pretraining and with the explicit aim of pretraining a model that lacks entity-specific factual knowledge. 1 provides an example of our pretraining and inference pipeline.

2.1 Corpus Anonymization↩︎

2.1.0.1 Named Entity Recognition

The first step to effectively anonymize our pretraining data, is to identify all named entities present in the text. We employ a state-of-the-art Named Entity Recognition (NER) model from the Flair framework [12]. In particular, we use the large 18-class OntoNotes model based on XLM-R embeddings,² which obtained a reported 90.0% F1 on OntoNotes [13]. We use OntoNotes’ more fine-grained entity tagset [14], [15] for better control over which kinds of entity mentions are anonymized and to be able to construct more informative placeholder tokens based on the entity types.³ In this work, we only anonymize tokens corresponding to named entity types, not numerical or temporal values, which are also recognized by the NER model. This is based on the conjecture that numerical and temporal values may be crucial for a deeper understanding of the textual context. Retaining them ensures that the anonymized text remains coherent, readable, and semantically meaningful, which is essential for effective model training without introducing bias. The NER model does not have perfect recall; 10-13% of entity mentions are not anonymized in our pretraining corpus.⁴ However, retained entities are more likely to be rare long-tail entity names, which we postulate corresponds to a much smaller portion of entity-linked knowledge, while more aggressive anonymization would degrade syntactic integrity. Indeed, closed-book QA accuracy remains near-random across all model scales (§4.4, Table ¿tbl:tab:closedbook95results?), empirically validating that this level of anonymization successfully suppresses parametric knowledge.

2.1.0.2 Anonymization Strategy

We apply an anonymization procedure to each document in our dataset which replaces all identified entities with placeholders following the format EntX, where Ent denotes the entity type and X is a unique identifier assigned within the specific document (see 8). This strategy preserves the original sentence structure, enabling the model to discern general linguistic patterns and relationships without being biased by concrete real-world instances. By processing each document individually, we ensure that entity placeholders are unique within that document and consistent for repeated mentions of the same entity. This consistency helps maintain the coherence and logical flow of the text while still abstracting away factual details, thus preserving the underlying structure of general knowledge without exposing the model to specific facts. However, we do not perform coreference resolution and therefore different named mentions of the same entity (e.g., “Barack Obama" and”President Obama") will generally be assigned different tokens. 1 provides an example of text before and after applying our anonymization strategy.

This anonymization strategy limits models’ ability to acquire specific factual information from the text, since most factual knowledge statements are grounded in one or more entity names. However, in addition to small proportion of names not being anonymized due to NER errors, some entities can be identified uniquely through descriptive references, which may cause some leakage of entity-specific knowledge. For example, the sentence “The 44th President of the world’s most powerful country was born on an island state in that country.” still encodes implicit entity knowledge, although our anonymization procedure was designed to remove such information. However, such cases again represent a small minority of factual statements. Another important limitation concerns gender information arising from the use of gendered coreferences. Referring to an entity with pronouns such as “he” or “she” implicitly reveals gender information and may lead the model to internalize gender-related biases. Nonetheless, we chose not to anonymize pronominal coreferences, as they constitute a fundamental linguistic component of language and communication that we want the model to capture.

2.2 Pretraining Procedure↩︎

Knowledgeless language models are pretrained on large, diverse corpora similar to standard language models, the only difference being the application of the anonymization strategy described above that limits the model’s direct exposure to entity-related knowledge. This ensures that the model learns general linguistic patterns while minimizing reliance on memorized entity-specific information. We train autoregressive Transformer language models with the standard language modeling objective of predicting the next token, but the approach is not limited to any particular model architecture. The KLLM’s tokenizer is trained on the same anonymized corpus to ensure consistency. During fine-tuning and inference the same anonymization strategy is applied. This ensures that the model is learning how to gather information about entities and to reason about entities based on the given text through their placeholder tokens, without relying on memorization.

3 Experiments↩︎

Our experiments aim to isolate how suppressing entity-linked knowledge during pretraining affects parametric recall, contextual reasoning, and reliability. We employ controlled comparisons between standard language models (SLMs) and KnowledgeLess Language Models (KLLMs) under matched architectures, data, and training budgets.⁵

3.0.0.1 Pretraining Data: 2.5B tokens

We pretrain models at multiple data scales. The smallest data setup consists of two complementary English corpora. The CNN/DailyMail dataset [11] (272M tokens) contains news articles spanning diverse domains such as politics, business, sports, and technology, providing linguistically rich text with a high density of named entities. In contrast, English Wikipedia [16] (2.2B) offers broad coverage of world knowledge through structured expository text. Wikipedia is substantially larger and more diverse, whereas CNN/DailyMail provides denser entity mentions within a news-oriented domain. During preprocessing, we remove non-UTF-8 characters to reduce tokenizer artifacts. The corpora are concatenated and anonymized using the procedure described in 2.

3.0.0.2 Pretraining data: 10B–20B tokens

To assess scalability, we construct larger pretraining corpora of 10B and 20B tokens derived from the publicly available HuggingFaceTB/smollm-corpus. This dataset is a curated mixture of CommonCrawl, Wikipedia, Books, News, and WebText-like sources. To ensure comparability with the 2.5B-token setup, we apply the same preprocessing and anonymization pipeline and uniformly sample shards until reaching the desired token count, preserving domain proportions. The resulting corpus provides a controlled setting to evaluate how anonymization interacts with increased data scale.

3.0.0.3 Tokenization

We train BPE tokenizers [17] on the pretraining corpus. Anonymization tags (e.g.) are included as reserved tokens to prevent fragmentation. Given the relatively small scale of our training data compared to standard LLM pretraining, we use a vocabulary size of 30k tokens. The same tokenizer configuration is used for both KLLMs and SLM baselines to ensure comparability.

3.0.0.4 Model Pretraining

We pretrain models from scratch using the LLaMA architecture family [18], [19], specifically Llama-3.2-1B and Llama-3.2-3B, as well as SmolLM-135M, SmolLM-360M, and SmolLM-1.7B from the SmolLM family [20].

For each architecture, we train a matched pair of models: a standard language model (SLM) on the original corpus, and a KLLM on the anonymized corpus. The models share identical architectures, training data size, tokenizer, and optimization settings, differing only in the presence or absence of anonymization. This controlled setup allows us to attribute performance differences directly to the pretraining signal rather than to confounding factors. Pretraining loss curves are provided in Appendix 9. We additionally compare against the original pretrained model checkpoints to provide a reference point relative to large-scale pretraining.

3.0.0.5 Fine-tuning

Due to the relatively small pretraining scale, we fine-tune all models on each downstream benchmark using the corresponding training splits. To maintain consistency with the pretraining regime, fine-tuning data for KLLMs is anonymized using the same procedure. This ensures that models are trained to perform inference under the same input representation used at pretraining time.

3.0.0.6 Inference

At inference time, we follow a three-step anonymization pipeline. First, inputs and any accompanying context are anonymized using the same entity-masking scheme applied during pretraining. Second, the anonymized input is processed by the model. Finally, outputs are de-anonymized by restoring original entity names (see Figure 1). This allows evaluation against standard benchmarks while preserving consistency with the training distribution.

3.0.0.7 Evaluation Overview

We evaluate models across several complementary settings designed to probe different aspects of behavior: (i) closed-book factual recall, to measure reliance on parametric knowledge; (ii) context-based question answering and reasoning, to assess the ability to extract and use provided information; (iii) retrieval-grounded inference, to simulate realistic evidence-based settings; and (iv) calibration and selective prediction, to evaluate alignment between confidence and correctness. This combination of evaluations enables a systematic analysis of how knowledge-suppressed pretraining affects both performance and reliability.

4 Results↩︎

To assess the performance of our KLLMs, we evaluate them on diverse downstream tasks in several different setups. Across all tasks, we follow the standard evaluation protocols defined in the respective benchmark setups, ensuring comparability with prior work. The main evaluation metric is accuracy, which is reported as a percentage. We additionally verify that KLLMs retain strong linguistic and reasoning capabilities on standard benchmarks such as SuperGLUE; detailed results are provided in Appendix 11.

4.1 Factual Reading Comprehension↩︎

Our first set of experiments aims to assess whether our knowledgeless models are able to perform knowledge-intensive tasks including question answering, fact checking and hallucination detection when the necessary factual knowledge is provided as context. 1 presents the performance of KLLM and SLM models across these benchmarks. On the factual reading tasks (LAMA, SQuAD, and NQ), KLLM training consistently improves performance over the baselines, with gains that grow larger at scale. These results confirm that removing entity-specific cues during pretraining does not weaken factual reasoning; rather, it encourages models to leverage the input context more effectively.

Strikingly, the improvements are even more pronounced on fact-checking and hallucination detection. On FEVER, KLLM models achieve up to +7.8 F1 over their baselines, while on HaluBench the gap reaches as high as +10.4% accuracy at the 1.7B scale. Together, these findings demonstrate that KLLMs not only retain robust factual reasoning under context but also provide an advantage in maintaining faithfulness and reliability.

To ensure that the observed improvements are not due to generic corruption or regularization effects, we compare KLLM against several alternative pretraining perturbations, including random token masking, span corruption, and entity shuffling. These baselines yield only marginal improvements over SLM, while KLLM consistently outperforms them across all evaluated settings. Full results are provided in Appendix 14.

4.2 Retrieval and Tool-Grounded Evaluation↩︎

To assess performance in a typical real-world practical setting, we evaluate both models under retrieval-grounded inference, where answers must be derived from externally retrieved evidence. We consider three retrieval paradigms: (i) sparse retrieval using BM25 over Wikipedia; (ii) learned dense retrieval in the spirit of Dense Passage Retrieval (DPR); (iii) retrieval exposed as an external search tool that the model may invoke at inference time. Unless stated otherwise, results in this section correspond to the 10B-token pretraining setting using SmolLM-1.7B (1.7B). Results for additional scales (2.5B and 20B tokens) are provided in Appendix 10.

4.2.0.1 Sparse Retrieval (BM25)

We first evaluate a standard BM25 pipeline. For each query, we retrieve the top-\(k\) passages and provide them as context to the model. As shown in Table ¿tbl:tab:retrieval95results?, KLLMs consistently outperform SLMs across all datasets, with relative improvements of 20–25%. In addition, KLLMs exhibit higher abstention rates under insufficient evidence, suggesting improved alignment with evidence quality.

4.2.0.2 Learned Dense Retrieval (DPR-style)

In this setup, the retriever is a separate dual-encoder model trained with a standard contrastive objective on question–passage pairs. We train both SLM-based and KLLM-based retrievers, while keeping the reader model fixed. As shown in Table ¿tbl:tab:dpr95results?, KLLM-based retrievers achieve higher retrieval recall and consistently improve downstream QA performance across all datasets. These results demonstrate that the benefits of knowledge-suppressed pretraining extend beyond generation to retrieval itself.

4.2.0.3 Tool-Augmented Retrieval

Finally, we evaluate a controlled tool-use setting, in which retrieval is exposed to the model as an external search tool. Given a query, the model first decides whether to issue a search request or answer directly. If a search is triggered, the model generates a query, and a fixed retriever (BM25 over Wikipedia) returns top-\(k\) passages. The model is then required to answer using only the retrieved evidence or abstain if the evidence is insufficient. The same retrieval system, prompts, and decoding settings are used for SLMs and KLLMs, ensuring that the comparison isolates differences in the underlying model rather than the search tool.

Table 2 shows that KLLMs exhibit improved behavior in the tool-augmented setting. They achieve higher answer accuracy while reducing the rate of unsupported answers, and tend to invoke the search tool more appropriately when evidence is required. This indicates that knowledge-suppressed pretraining improves not only how models use retrieved evidence, but also when they choose to rely on external tools.

4.3 Commonsense Reasoning↩︎

Furthermore, we evaluate our KLLMs on three widely-used commonsense reasoning benchmarks. CommonsenseQA [21] tests the model’s ability to answer multiple-choice questions that require broad everyday knowledge and commonsense inference. StrategyQA [22] challenges models to reason over implicit multi-step processes to answer yes/no questions, emphasizing reasoning rather than memorized facts. PIQA [23] focuses on physical commonsense, assessing the model’s understanding of everyday interactions and the principles of physical reality. In all cases, we provide the necessary context or task-specific information to the models, ensuring that the evaluation reflects their reasoning capability rather than reliance on parametric knowledge.

[tbl:tab:commonsense95results] summarizes the results of our SmolLM-based KLLMs and SLM baselines. KLLMs consistenctly outperform the baselines across all three tasks across all model scales, with improvements ranging from modest gains for smaller models to substantial margins for the 1.7B architecture (+5.0 on CommonsenseQA, +4.9 on StrategyQA, and +5.7 on PIQA). These results suggest that anonymization during pretraining does not impede the models’ ability to capture commonsense patterns; instead, it encourages reliance on contextual reasoning rather than memorized associations. Combined with the reductions in hallucination observed in generation, this provides strong evidence that knowledgeless pretraining improves both factual robustness and commonsense generalization, particularly as the model capacity increases.

4.4 Closed-Book QA↩︎

In order to assess that our knowledgeless models are not acquiring parametric knowledge despite anonymization, we additionally evaluate our KLLM models on the LAMA and SQuAD datasets in a closed-book setting, where no supporting context is provided. ¿tbl:tab:closedbook95results? reports the results. The closed-book evaluation clearly shows that KLLM models retain only a minimal amount of factual knowledge compared to their non-anonymized baselines.

4.5 Calibration and Factuality↩︎

4.5.0.1 Confidence–Correctness Alignment

We first evaluate how well model confidence aligns with prediction correctness. Following prior work, we use the probability assigned to the first generated token as a proxy for model confidence. We report standard calibration metrics, including Expected Calibration Error (ECE), Brier score, and AUROC, which measure the alignment between predicted confidence and empirical correctness.

[tbl:tab:calibration95metrics] reports these metrics for both SLM and KLLM across pretraining scales. KLLMs consistently outperform their corresponding baselines, achieving lower ECE and Brier scores and higher AUROC. Notably, these improvements persist at larger scale (20B tokens), indicating that knowledge-suppressed pretraining leads to intrinsically more informative confidence signals rather than relying on post-hoc correction.

4.5.0.2 Selective Prediction and Abstention

We next evaluate calibration in a decision-making setting via selective prediction. A decision threshold is chosen on a held-out validation set to maximize each model’s F1 score, trading off precision (correctness of attempted answers) and recall (coverage). Results are reported in ¿tbl:tab:calibration95results?.

KLLMs consistently outperform the baselines across all model sizes and metrics. While recall remains comparable, KLLMs achieve substantially higher precision and improved F1 scores, with gains becoming more pronounced at larger scales. On average, KLLMs surpass the baseline by +17.4 points in precision, +3.1 points in recall, and +6.6 points in F1. These results indicate that KLLMs provide a better separation between correct and incorrect predictions, enabling more reliable abstention decisions.

4.5.0.3 Abstention Tuning

We additionally employ a factuality-oriented fine-tuning strategy aimed at encouraging models to abstain when uncertain rather than producing incorrect answers [24], [25]. Specifically, we partition the training data into two subsets. The model is first fine-tuned in the standard way on the first subset. It is then evaluated on the second subset, and any instance where the model outputs an incorrect prediction is relabeled with the abstention output “I don’t know the answer". A second fine-tuning stage is then performed on this modified data, providing the model with explicit abstention supervision.

As shown in Table [tbl:tab:hallu95training95results], this procedure yields consistent improvements in calibration and reliability. On average, KLLM models reach an F1 score of \(29.0\), compared to \(23.2\) for their baselines. Taken together, these results demonstrate that KLLMs improve both the accuracy and the reliability of predictions. Importantly, the gains in calibration are largest in settings where evidence is incomplete or misleading, suggesting that knowledge-suppressed pretraining enhances the model’s ability to recognize uncertainty and avoid unsupported answers.

Importantly, the improvements observed for KLLMs cannot be explained by easier optimization or undertraining effects. Across all scales, KLLMs consistently exhibit higher pretraining loss than their SLM counterparts (Appendix 9), reflecting the increased difficulty of predicting anonymized sequences without direct entity supervision. Despite this, KLLMs achieve stronger performance on contextual, retrieval-grounded, and calibration-oriented evaluations. This indicates that the observed gains arise from a shift in the learned inductive bias toward evidence-grounded reasoning, rather than from improved optimization or memorization efficiency.

5 Related Work↩︎

The primary motivation of this work is to develop models that are more robust by relying on externally provided, validated context rather than on their own parametric knowledge. As demonstrated both theoretically and empirically by [26], hallucinations in LLMs are inevitable, since no model can fully encode the entirety of existing factual mappings. Additionally, the fact that traditional training and evaluation reward guessing more than uncertainty acknowledgment makes it natural for LLMs to hallucinate [27]. This topic has been studied from many different perspectives [5], [28], [29]. This is also related to the setting of selective prediction, where models can abstain from answering a query [30], [31].

Another complementary direction to reducing hallucinations is improving model calibration [32], i.e., aligning the model’s confidence with the actual likelihood of correctness. This is particularly relevant to our work, as KLLMs are designed to abstain more readily from misinformation and benefit from mechanisms that quantify uncertainty. Prior approaches to calibration often operate at the logit level through post-hoc transformations [33], [34], or rely on uncertainty estimation methods [35]. More recent research has explored leveraging language models themselves for calibration, either by fine-tuning on correctness-labeled data [25], [36], prompting or in-context learning strategies [24], [37], or zero-shot instruction-oriented setups [38]–[40], as well as through consistency sampling [41]. Other approaches go further by exploiting internal representations for uncertainty classification [42], introducing explicit tokens for abstention or uncertainty [43], [44], or curating specialized datasets to train models to refuse unanswerable queries [45], [46].

6 Conclusion↩︎

We introduced KnowledgeLess Language Models (KLLMs), a simple pretraining intervention that suppresses entity-linked supervision and induces a strong reduction in reliance on parametric factual knowledge without architectural changes. This leads to a near-complete collapse of closed-book factual recall, while preserving—and often improving—performance on contextual question answering, fact verification, and commonsense reasoning.

Across a wide range of settings, KLLMs consistently outperform standard language models in evidence-grounded regimes, including retrieval-augmented and retrieval-trained setups. The gains are particularly pronounced under imperfect or incomplete evidence, where KLLMs exhibit higher robustness and more reliable abstention behavior. In parallel, they achieve improved calibration across multiple metrics, indicating better alignment between confidence and correctness.

These results suggest that pretraining objectives play a central role in shaping how models use knowledge: suppressing parametric supervision encourages reliance on external evidence and improves decision reliability. KLLMs therefore integrate particularly well in retrieval-augmented and tool-based setups, serving as base models that are more sensitive to evidence quality and less prone to hallucination.

The observed effects persist across scaling and ablation analyses. In future work, we plan to extend this paradigm by investigating broader forms of knowledge suppression on increasingly large amounts of training data, and extend our tool-usage experiments to investigate how KLLMs can be deployed in complex agentic workflows that may involve multi-step retrieval, tool chaining, and interaction. Overall, KLLMs provide a concrete step toward language models that are not only capable, but reliably grounded and controllable.

7 Limitations and Future Work↩︎

While our results demonstrate consistent benefits of knowledge-suppressed pretraining, several avenues for additional research remain.

7.0.0.1 Pretraining Scale.

Our study is conducted at pretraining scales up to 20B tokens. Although the observed trends are consistent across scales, it remains an open question whether the same effects persist at trillion-token regimes, where parametric knowledge and memorization effects may behave differently.

7.0.0.2 Scope of Knowledge Suppression.

Our approach focuses on anonymizing named entities as a proxy for suppressing factual supervision. While effective, this does not fully eliminate all forms of parametric knowledge, as models may still learn factual associations through non-entity cues, descriptive phrases, or incomplete anonymization due to NER errors. Extending this approach to broader forms of knowledge (e.g., events, relations, or implicit facts) remains an open direction.

7.0.0.3 Dependence on External Context.

By design, KLLMs rely more strongly on provided evidence. While this improves robustness in grounded settings, it may also lead to degraded performance in scenarios where high-quality context is unavailable or retrieval fails entirely. In such cases, standard models with stronger parametric knowledge may retain an advantage.

7.0.0.4 Retrieval and Tooling Assumptions.

Our evaluation considers controlled retrieval pipelines (BM25, DPR-style retrievers, and tool-based search) with fixed interfaces. In future work, we hope to investigate more complex interactions, including multi-step retrieval, tool chaining, or noisy external sources. Further evaluation in fully agentic or interactive environments is needed to assess how KLLMs behave under these conditions.

7.0.0.5 Mechanistic Understanding.

While our empirical results suggest that anonymization shifts models toward evidence-grounded reasoning, the underlying representational changes are not fully understood. In particular, it remains unclear how internal representations of uncertainty, factuality, and evidence are reorganized under knowledge-suppressed pretraining. Further analysis of model internals could help clarify these mechanisms.

7.0.0.6 Practical Trade-offs.

Finally, KLLMs introduce a trade-off between reduced parametric knowledge and increased reliance on external information. While desirable in many settings, this trade-off may not be optimal for applications requiring strong recall of stable, well-established facts. Designing hybrid approaches that dynamically balance parametric and retrieved knowledge remains an important direction for future work.

8 Anonymization Details↩︎

8.0.0.1 Entity Recognition.

Table 3 lists the entity types detected in the entity recognition phase, among which we only consider genuine named entities for anonymization, while retaining numeric values in the text.

8.0.0.2 Adding New Identification Tokens to the Vocabulary.

In order to identify each of the specific entities within a document, we assign a specific identity token additionally to the entity type description (2). Ideally, the generated tokens should exhibit maximal randomness to ensure minimal semantic overlap with any existing tokens in the language. To achieve this, we generate 100 novel tokens, each consisting of a randomly constructed string of 10 characters. Each character is sampled uniformly from a set comprising uppercase letters and digits. The choice of 100 tokens reflects a balance between distinctiveness—ensuring that each entity in a document can be uniquely identified—and frequency, such that each token occurs sufficiently often for the model’s learned semantics to remain effectively random.

9 Pretraining Loss↩︎

For further analysis, in Figures 2 and [figure:pretraining95eval95loss95curve], we plot the loss curves of our KLLM pretraining using Llama-3.2-1B.

During training, we observe that the non-anonymized baseline consistently achieves lower loss values compared to the anonymized models, with convergence occurring after less than one epoch. This behavior is expected, as explicit entity references provide stronger predictive cues than anonymized placeholders. For instance, predicting the continuation of “Barack Obama was born in ...” is considerably easier than for the anonymized variant “Person74 was born in ...”. The same holds for “The iPhone was developed by ...” in comparison with “Product42 was developed by ...”. A second factor contributing to this gap is that descriptive phrases referring to entities (e.g., “the 44th President of the United States”) are notably easier for the baseline model than for the knowledge-less version, which lacks entity-specific grounding. Finally, some loss discrepancies stem from limitations of our anonymization procedure itself, as the entity recognizer occasionally fails to anonymize certain mentions, leaving them as direct predictive cues.

Importantly, Table 4 shows that KLLMs maintain consistently higher training and evaluation losses despite achieving substantially stronger downstream performance in retrieval-grounded settings. This indicates that the improvements observed throughout the paper are not explained by easier optimization or generic undertraining effects. Instead, anonymization introduces a more challenging language modeling objective while simultaneously encouraging representations that rely more strongly on contextual evidence rather than direct parametric recall.

Together, these findings explain both the lower loss values and the earlier stagnation of the non-anonymized baseline, while supporting our interpretation that KLLMs induce a qualitatively different epistemic behavior rather than simply improving optimization efficiency.

10 Retrieval-Grounded Evaluation Details↩︎

In this section, we report retrieval-grounded evaluation results across all pretraining scales (2.5B, 10B, and 20B tokens). The 2.5B setting uses Llama-3.2-3B, while the 10B and 20B settings use SmolLM-1.7B (1.7B). To complement the contextual QA setting used in the main paper, we evaluate both models in a retrieval-augmented pipeline using BM25 over Wikipedia. For each query, we retrieve the top-\(k\) passages and provide them as context to the model, using identical prompts and decoding settings across models.

10.0.0.1 Sparse Retrieval (BM25).

Across all datasets, KLLMs consistently outperform SLMs, with improvements remaining stable across scales. Gains tend to increase slightly with larger training data, suggesting that knowledge-suppressed pretraining scales favorably.

10.0.0.2 Effect of Retrieval Quality.

As shown in Table 6, KLLMs achieve stronger improvements when gold evidence is absent, indicating improved robustness under imperfect retrieval conditions.

10.0.0.3 Dense Retrieval (DPR-style).

Consistent with the main text, KLLM-based retrievers achieve higher recall and improve downstream QA performance across all scales, indicating that the benefits of knowledge-suppressed pretraining extend to representation learning for retrieval.

11 SuperGLUE Results↩︎

3pt

First, we employ the SuperGLUE benchmark [48]. SuperGLUE is a suite of ten challenging language understanding tasks, covering areas such as question answering, textual entailment, co-reference resolution, and word sense disambiguation.

Table 8 presents the performance comparison between our KLLM, based on Llama-3.2-1B, against the baseline standard language model (SLM). The KLLM achieves comparable or slightly improved results on most tasks, with notable gains on CB (+3.9%), COPA (+5.9%), and WSC (+3.7%), confirming that linguistic competence is preserved.

12 Transfer from Anonymized to Natural Test Data↩︎

KLLMs are trained and evaluated primarily under anonymized inputs, reflecting their intended operating regime where entity-specific surface forms are removed during pretraining. A fundamental question for practical deployment is whether the linguistic abstractions learned under anonymized supervision transfer robustly to natural (de-anonymized) test inputs. To directly assess this, we evaluate the same KLLM model under open-book conditions on both anonymized and natural versions of the test sets.

We focus on reasoning- and factuality-oriented benchmarks that require multi-hop, causal, or evidence-based reasoning: FEVER, CommonsenseQA (CSQA), StrategyQA, PIQA, and HaluBench.

12.1 Open-Book QA: Anonymized vs.Natural Inputs (KLLM Only)↩︎

Table 9 reports KLLM performance under open-book evaluation on anonymized test inputs and on their corresponding de-anonymized natural counterparts.

12.2 Structural Transfer Analysis↩︎

Across all benchmarks, KLLM exhibits stable performance under de-anonymization, with only minor variations between anonymized and natural inputs. For knowledge-intensive verification (FEVER), performance decreases moderately (93.8 → 89.6), while for commonsense reasoning (CSQA) performance slightly increases (69.2 → 70.5). StrategyQA and PIQA show similarly small shifts, and HaluBench remains effectively unchanged.

These trends indicate that KLLM does not depend on surface-level entity placeholders for reasoning. Instead, its behavior is governed by structural, relational, and semantic representations that transfer robustly to real-world text. This confirms that anonymization-based pretraining does not introduce brittle reliance on artificial symbols and remains fully compatible with natural language inputs at inference time.

13 Ablation Studies↩︎

13.0.0.1 Continued Pretraining.

We evaluate whether the KLLM objective is effective only when training from scratch or also when applied on top of a standard pretrained model. To this end, we initialize from an SLM checkpoint and continue training under the KLLM anonymization objective.

6pt

As shown in Table 10, continued pretraining improves contextual factual QA from 43.1 to 46.8 and commonsense reasoning from 55.6 to 58.9 relative to the SLM initialization. At the same time, closed-book accuracy drops from 34.1 to 28.5, indicating partial suppression of parametric factual recall even without full retraining from scratch. Crucially, full training under the KLLM objective further reduces closed-book accuracy to 2.9, confirming that parametric factual knowledge is maximally suppressed only when anonymization is applied throughout pretraining.

13.0.0.2 Inference-Time Anonymization.

We also ablate the effect of removing anonymization at inference by evaluating KLLM on the de-anonymized (natural) test inputs. As shown in Table 10, factual QA on FEVER drops from 93.8 to 89.6 (–4.2 points), while the average commonsense score over CSQA, StrategyQA, and PIQA decreases slightly from 70.7 to 70.3. This modest degradation indicates that matching the train-time and test-time input distributions still benefits KLLMs, as test-time de-anonymization reintroduces surface-level entity forms that can re-activate weaker parametric priors and slightly misalign the model with the provided context.

13.0.0.3 Anonymization Robustness: Masking Ratio.

We evaluate the sensitivity of KLLMs to the strength of anonymization by varying the fraction \(p \in \{0.5, 0.75, 1.0\}\) of NER-identified entity mentions that are replaced with placeholders during pretraining, while keeping the SmolLM-1.7B architecture and all optimization settings fixed.

Table 11 shows a clear monotonic trade-off as \(p\) increases. Relative to the non-anonymized SLM (\(p=0.0\)), partial anonymization already yields modest gains in contextual reasoning (Factual QA: 43.1 \(\rightarrow\) 44.5 \(\rightarrow\) 47.3) while substantially reducing closed-book accuracy (34.1 \(\rightarrow\) 20.9 \(\rightarrow\) 9.0). Full anonymization (\(p=1.0\)) further improves both factual QA (48.1) and commonsense reasoning (60.8), while driving closed-book performance down to near-random levels (2.9). These results indicate that increasing the anonymization ratio progressively suppresses parametric knowledge, with full anonymization providing the strongest separation between contextual reasoning ability and memorized facts.

13.0.0.4 Robustness to Imperfect Context.

To evaluate sensitivity to evidence quality, we construct perturbed evaluation settings in which the input context is systematically degraded. Specifically, we consider (i) truncated context, where only a prefix of the retrieved evidence is provided, and (ii) perturbed context, where irrelevant or misleading sentences are injected into the evidence. We also report abstention rates when models are allowed to refrain from answering.

The results are summarized in Table 12. Under full context, both models achieve comparable performance. However, as the quality of the evidence deteriorates, KLLMs exhibit smaller performance drops compared to SLMs (e.g., under truncation and perturbation), while consistently maintaining higher abstention rates.

These findings indicate that KLLMs rely more strongly on the provided evidence and are less prone to fallback on spurious parametric associations when context becomes unreliable. Consequently, KLLMs demonstrate improved robustness to imperfect inputs and better alignment between prediction behavior and evidence quality.

14 Corruption Baselines↩︎

To isolate the effect of entity anonymization from generic data corruption or regularization, we evaluate several alternative pretraining perturbations applied to the same corpus and training setup.

14.0.0.1 Baselines.

We consider the following variants: (i) Random masking, where tokens are replaced with a placeholder at a rate matched to the proportion of anonymized entities; (ii) Span corruption, following a T5-style objective; (iii) Entity shuffling, where entity mentions are replaced with randomly sampled entities of the same type, preserving surface statistics while breaking identity.

All models are trained with identical architectures, tokenizers, and optimization settings.

As shown in Table 13, all corruption-based approaches provide only minor improvements over the SLM baseline, indicating limited benefits from generic noise injection or regularization. In contrast, KLLM achieves substantially larger gains across all metrics, suggesting that the improvements are not driven by corruption alone, but by the specific structure of entity anonymization, which removes entity-linked supervision while preserving relational and compositional structure.

15 Calibration Metrics Beyond Precision/Recall↩︎

In addition to the threshold-based calibration analysis in the main paper, we evaluate standard calibration metrics, including Expected Calibration Error (ECE), Brier score, and AUROC for correctness prediction.

Table 14 shows that KLLMs consistently achieve lower ECE and Brier scores, together with higher AUROC, across both 2.5B and 20B training regimes. These results confirm that the improvements observed under threshold-based evaluation are consistent with standard calibration measures.

16 Data Efficiency Analysis↩︎

We further analyze the number of training tokens required to reach comparable performance levels.

As shown in Table 15, KLLMs reach comparable contextual QA performance using substantially fewer training tokens, suggesting improved data efficiency due to reduced entity-specific redundancy.

17 Random Masking vs.Structured Anonymization↩︎

To verify that the observed improvements are not due to generic regularization, we compare anonymization to random token masking applied at the same rate.

Table 16 shows that random masking does not reproduce the strong gains in contextual reasoning nor the collapse of closed-book performance observed for KLLMs, indicating that the effect is specific to structured entity anonymization.

18 Representation-Level Analysis↩︎

We probe the extent to which models encode entity-specific information by training linear classifiers to predict entity identity from hidden states.

As shown in Table 17, KLLMs exhibit substantially lower probe accuracy, indicating reduced encoding of entity-specific information in internal representations.

19 Qualitative SLM–KLLM Comparisons↩︎

To provide a more concrete view of how KLLMs differ from standard language models in their reasoning behavior, we include a small set of qualitative examples drawn from FEVER, CommonsenseQA, StrategyQA, and HaluBench. In each case, the model is evaluated in the open-book setting with access to the relevant context.

20 Implementation Details↩︎

20.0.0.1 Pretraining Setup.

All SLM and KLLM models are trained from scratch using the same architecture, tokenizer configuration, optimizer, batch size, sequence length, and number of training tokens. The only difference between each matched pair is whether the pretraining corpus is anonymized. We train autoregressive Transformer language models with the standard next-token prediction objective. For the 2.5B-token experiments, we train models on the concatenation of CNN/DailyMail and Wikipedia. For the 10B- and 20B-token experiments, we use sampled shards from the HuggingFaceTB/smollm-corpus mixture, applying the same preprocessing and anonymization pipeline.

20.0.0.2 Tokenization.

We train BPE tokenizers on the corresponding pretraining corpus with a vocabulary size of 30k. Entity placeholder tokens are added as reserved tokens so that anonymized entity identifiers are not split into subword pieces. The same tokenizer configuration is used for the matched SLM and KLLM pairs at each scale.

20.0.0.3 Optimization.

Models are trained using AdamW with weight decay. Learning rates, warmup schedules, batch sizes, and training durations are matched between SLM and KLLM pairs. Hyperparameters are selected once based on stable pretraining loss and are not tuned separately for KLLM. This ensures that differences between SLM and KLLM reflect the anonymization-based supervision signal rather than optimization-specific choices.

20.0.0.4 Fine-Tuning.

For each downstream benchmark, we fine-tune models using the official training split when available. KLLMs are fine-tuned on anonymized versions of the training data, while SLMs are fine-tuned on the original data. We use the same fine-tuning hyperparameters, number of epochs, and validation protocol for each matched pair. Thresholds used for selective prediction and abstention are selected on held-out validation data only.

20.0.0.5 Inference and De-anonymization.

At inference time, KLLM inputs are anonymized using the same NER-based pipeline used during pretraining. Model outputs are then de-anonymized by mapping generated placeholders back to the original entity strings when possible. Evaluation is always performed against the original gold labels or answers.

20.0.0.6 Sparse Retrieval Setup.

For sparse retrieval, we use a fixed BM25 retriever implemented over a Wikipedia passage index. For each query or claim, we retrieve the top-\(k\) passages with \(k=5\) and provide these passages as context to the reader model. The same retrieved passages are provided to SLMs and KLLMs, with identical prompts and decoding settings. This ensures that differences in retrieval-grounded performance reflect the reader model rather than differences in the retrieval system.

20.0.0.7 Dense Retrieval Setup.

For DPR-style retrieval, the retriever is a separate dual-encoder model trained with a standard contrastive objective on question–passage pairs. We train SLM-based and KLLM-based retrievers using the same training examples, negative sampling strategy, optimizer, and number of updates. At inference time, each retriever retrieves the top-\(k=5\) passages from the same Wikipedia passage index. Retrieved passages are then provided to the corresponding reader model using the same prompting and decoding configuration as in the BM25 setup.

20.0.0.8 Tool-Augmented Setup.

For tool-augmented evaluation, retrieval is exposed as a discrete search action. The model may either answer directly, abstain, or issue a search query. Search queries are executed using the same fixed BM25 Wikipedia search tool used in the sparse retrieval experiments, returning the top-\(k=5\) passages. The returned passages are appended to the input context, after which the model must answer using the evidence or abstain if the evidence is insufficient. Both SLMs and KLLMs interact with the same tool interface and are evaluated with identical prompting, parsing, and decoding settings.

20.0.0.9 Compute.

Pretraining runs are conducted on 8 NVIDIA A100 GPUs. Fine-tuning and evaluation runs are conducted on a single NVIDIA A100 GPU. Pretraining is the dominant computational cost, while fine-tuning and evaluation are comparatively lightweight. We will release scripts and configuration files specifying the exact model sizes, training tokens, batch sizes, learning rates, and hardware assumptions needed to reproduce the experiments.

References↩︎

---

1.  Work done while at the African Institute for Mathematical Sciences. Now at Polytechnique Montréal.↩︎

2. https://huggingface.co/flair/ner-english-ontonotes-large↩︎

3. See 8 for a list of the detected entity types.↩︎

4. Based on a manual inspection of 50 randomly sampled documents from our pretraining corpus, we estimate an NER recall of approximately 87%.↩︎

5. Our pretrained KLLMs, preprocessing and evaluation scripts will be released upon publication.↩︎