RGB109,196,233 RGB 76,144,186 RGB 43,194,194 RGB32,149,130 RGB247,212,140 RGB17, 202, 247 RGB107, 250, 169 RGB224, 91, 91 RGB247, 237, 236 RGB254,153, 41 RGB229,120,114 RGB240, 59, 32

RGB 32,119,180 RGB248,118, 0 RGB 43,153, 43 RGB207, 37, 37

RGB 34, 94,168 RGB254,217,142 RGB254,153, 41 RGB240, 59, 32

RGB201, 245, 252 RGB252, 210, 131

1 Introduction↩︎

Tabular data dominate enterprise applications and domains such as healthcare and finance [1]. Large tabular models (LTMs) [2]–[4], i.e., tabular foundation models, achieve state-of-the-art performance for tabular predictions by leveraging in-context learning (ICL) to perform zero-shot predictions on unseen tasks [5]. At inference, the model receives a context (e.g., historical example rows) alongside a query row with a missing target field to be predicted. While LTMs are typically pre-trained on synthetic data [3], [6], recent models incorporate real-world tabular corpora [4], [7], [8], e.g., by fine-tuning on diverse real-world task mixtures to improve performance [9]. While LLMs are known to unintentionally memorize training data [10]–[12], the memorization dynamics of LTMs remain largely unexplored. Rigorous memorization probing is crucial to inform potential mitigations for sensitive fine-tuning data, such as differential privacy and adaptation of data pre-processing and training regimes.

Problem Setting.

In the ICL paradigm, a model predicts a missing target value \(y_q\) given query’s input features \(x_q\), and a context \(C = \{(x_1, y_1), \dots, (x_n, y_n)\}\) of examples. With LTMs, the user provides both the context \(C\) and the query \(q\), as shown in Fig. 1. We assume an LTM\(M_\Theta\) is fine-tuned on sensitive \(D\) by a data owner who provides access to the fine-tuned LTM to users. This setting opens up the possibility to manipulate \(C\) arbitrarily, to probe differences in model behavior on context-query pairs derived from data included in or excluded from \(D\). In contrast, existing works on ICL privacy primarily assume the model provider defines a sensitive \(C\), which an attacker attempts to extract via membership inference attacks (MIA) by manipulating \(q\) [13], [14]. MIAs [15] aim to infer whether a specific record was part of the training data, and transferring MIAs to LTMs presents unique challenges. Unlike LLMs, which can generate arbitrary text on a vast output space to extract memorized tokens [16], LTMs are encoder-only architectures that strictly constrain predictions to the pre-defined candidate labels provided in \(C\) [3], [4]. More importantly, while masked-language modeling can induce memorization [17], LTMs operate as Prior-Data Fitted Networks (PFNs) [18]. They are trained to infer structural connections within the context rather than memorize explicit targets [18]. This raises a fundamental, yet unanswered question: What memorization profile do LTMs, inherently trained to perform ICL, have? To investigate memorization capabilities of LTMs, we present IclMem and apply MIA. Specifically, we test membership of a candidate set of query rows by manipulating \(C\) as an active probing mechanism (Fig. 1). We hypothesize that by carefully modifying \(C\) to eliminate discriminative information, it disrupts the model’s context-derived prediction capabilities. Forced to abandon in-context reasoning, the model falls back on its memorized parametric knowledge. For member rows in \(D\), we expect predictions to exhibit higher confidence and remain robust to context manipulations [19], [20]. We identify specific circumstances for memorization in LTMs (Sec. 4.2) and discuss protection measures (Sec. 6).

2 Related Work↩︎

While LTMs[3], [4], [6], [7] rapidly advance, investigations into their privacy properties remain limited [21]. Existing LTM privacy literature primarily targets test-time evasion and adversarial robustness rather than memorization [22]–[24]. Conversely, ICL privacy research largely focuses on LLMs, probing either the leakage of hidden context examples [13], [14] or table memorization within text corpora [25], [26]. To assess memorization, state-of-the-art MIAs typically rely on shadow models [27], [28]. However, training shadow models for large models is computationally prohibitive [11] and yields unreliable false-positive bounds when the pre-training distribution is unknown [29]. Towards privacy assessment of LTMs, we adapt techniques from robustness-assessing MIA [19] and attribute inference (AIA) [30]–[32]. While AIA aims to infer an unknown sensitive value given a partial record, our goal is inferring membership of a record. To assess memorization in LTMs, we adapt AIA’s multiple-choice testing and label randomization and combine zero-information context manipulations with difficulty calibration via the pre-trained base model [33].

3 Contributions↩︎

We provide the first systematic assessment of memorization in LTMs. Our contributions are:

• We introduce IclMem, a probing framework designed to separate context-based predictions from parametric memorization. Our zero-information multiple-choice context strips away contextual patterns forcing the model to fall back on parametric memorization.

• We avoid common MIA false positives by accounting for distribution shifts (via data pre-processing), feature contamination (via data deduplication), base-rate confounding (via label randomization), and intrinsic low-complexity samples (via difficulty calibration).

• We perform an extensive evaluation on LTM ConTextTab across 1,128 fine-tuning configurations on 10 diverse classification and regression tasks from CARTE.

• We detect moderate memorization in 8 of the 10 tasks (with \(\text{AUC} > 0.5\) and \(\text{TPR}_{@r\%} > 0.1\) for \(r \in \{1, 10\}\)) in our controlled setup with single-task fine-tuning and fixed context-query pairs. The signal is higher for low-cardinality and binary tasks with small training query sizes (\(Q=50\)), but largely vanishes under practical setups (e.g., larger \(Q=512\), multi-dataset training, random context-query sampling) (App. 10).

Overall, our findings show memorization capabilities in LTMs which help to inform appropriate protection measures. Next, we introduce our methodology and setup (Sec. 4), detail our evaluation (Sec. 5), and conclude with limitations, mitigations, and future works (Sec. 6).

4 Methodology and Setup↩︎

We introduce IclMem, a framework designed to probe memorization in LTMs and mitigate common MIA evaluation pitfalls (Tab. 4, App. 7.1). First, we outline unique challenges of assessing LTMs. Then, we discuss the two stages of IclMem(Fig. 3, App. 7). Namely, controlled fine-tuning with careful data pre-processing and fixed query-context pairs across epochs (Sec. 4.1) and memorization probing with a suite of context manipulations calibrated against the pre-trained base model as reference (Sec. 4.2).

Challenges in Assessing LTMs. LTMs rely on ICL to infer structural feature-target correlations from the context, a mechanism that intuitively acts as a barrier to accessing the model’s parametric memory. Furthermore, unlike LLMs with massive output vocabularies, LTMs are encoder-only architectures strictly constrained to outputting probabilities over the specific candidate labels present in the context. Our context manipulations address these challenges via zero-information multiple-choice protocol (Sec. 4.2).

4.1 Establishing a Memorization Ground Truth↩︎

IclMem pre-processes data to eliminate artifacts, and performs controlled fine-tuning to induce memorization.

Data Pre-Processing. To avoid data artifacts being interpreted as memorization signal, IclMem carefully pre-processes data. First, we deduplicate records to prevent feature contamination, i.e., identical records appearing in both member and non-member sets, which inflate the memorization signal. Second, we ensure distributional closeness between disjoint member and non-member splits to preclude distributional shifts. Specifically, for target label distributions via total variation distance \(\text{TVD} < 0.05\) [34] for classification and the Kolmogorov–Smirnov statistic \(\text{KS}<0.05\) [35] for regression (i.e., difference between empirical cumulative density functions) (Tab. 6, App. 7.2). To mitigate the base-rate fallacy, i.e., model exploits label distribution to predict likely outcomes, we apply two measures. First, we enforce a uniform label distribution in the training context, preventing the model from minimizing loss by collapsing to majority-class predictions. Second, following [31], we apply label randomization, i.e., scrambling target labels while preserving marginal distributions, forcing the model to learn statistically improbable mappings.

Controlled Fine-Tuning. To induce memorization, we fine-tune on fixed context-query pairs for up to 500 epochs. Specifically, each training step processes a static context set \(C'\) with a batch of query rows of size \(Q\). We fine-tune across varying learning rates \(\eta \in \{10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}\}\) and query sizes \(Q \in \{50, 512\}\). We vary \(Q\) in training to test its influence on memorization. Smaller \(Q\) should amplify the contribution of individual query rows, while larger \(Q\) should dilute them. Our evaluation supports this dynamic, showing that larger \(Q=512\) hinders memorization (Sec. 5).

4.2 Probing Memorization with IclMem↩︎

The probing stage aims to detect memorization signals by combining zero-information context manipulations with difficulty calibration via the base model.

Multiple-Choice Protocol. To separate expected context-based model predictions from unintended parametric memorization, we propose a zero-information strategy mimicking a multiple-choice question. For each query row \(q=(x_q, y_q)\) and candidate label set \(Y\), we construct a probing context \(C_{base}\) containing exactly \(|Y|\) copies of the query’s features \(x_q\), each paired with a distinct candidate label \(y_i \in Y\), thereby providing the model with all possible candidate targets. Our hypothesis is that if we strip away all valid contextual patterns, the model will be forced to abandon in-context inference and default to its parametric memory, selecting the target value observed during fine-tuning.

Context Manipulations. To assess robustness of predictions across multiple runs, we subject \(C_{base}\) to a suite of 48 context manipulations \(\Pi\), including positional biasing, distractor injection, and distributional shifts by, e.g., increasing the counts of true target labels, or non-true labels (detailed in App. 7.3). We compute the probe loss \(L_k(q) = \mathcal{L}(M_{\theta}(x_q \mid C_k), y_q)\) for each manipulated context \(C_k \in \Pi\). The robustness score is defined as the average loss across all manipulations \(S(q) = \frac{1}{|\Pi|} \sum_{\pi_k \in \Pi} L_k(q)\), which is the basis for our memorization metric.

Difficulty Calibration. An inherently easy query (e.g., with obvious target from its features) has a low loss, regardless of its membership in \(D\). To account for this, we follow the difficulty calibration approach of Watson et al. [33] and compare robustness score from the fine-tuned model, \(S(q)\), to that of the pre-trained base model, \(S_{ref}(q)\), and compute its delta as \(\Delta S(q) = S(q) - S_{ref}(q)\). Notably, we compute \(S(q)\) not only using the loss but also using distributional metrics, i.e., prediction entropy and target confidence, which remain robust discriminators even with fine-tuning-induced artifacts (e.g., loss inversion, Sec. 5).

Memorization Metric. We compute AUC over the \(\Delta S\) scores (denoted \(\text{AUC}(\Delta S)\) or simply \(\text{AUC}\)), and consider \(\text{AUC} > 0.5\) as the threshold for detectable memorization. Unlike high AUC baselines for LLMs [14], LTMs operate over a constrained output space. Hence, we adopt this threshold to capture any calibrated signal above chance [36]. Furthermore, since global AUC averages across all records and can obscure localized memorization [27], we report the true positive rate at low false positive rates (\(\text{TPR}_{@r\%}\) for \(r \in \{1, 10\}\)) providing a granular view of more confident signals.

5 Empirical Evaluation↩︎

Next, we summarize our results of IclMem, and defer full details to App. 9 due to space constraints.

Setup. We evaluate IclMem on ConTextTab [4] on 10 tasks from CARTE [37]. We select these tasks as they were not used in pre-training and represent a wide variety of domains and data distributions (details in Tab. 5). We test for memorization across epochs to capture the evolution of the memorization over time resulting in 1128 probing runs (configurations in Tab. 2).

Isolating Memorization Outcomes. Our evaluation yields five distinct outcomes (summarized in Tab. 3).

0.35em

Impractical fine-tuning configurations (e.g., large learning rate) can induce data artifacts mimicking memorization. The most prominent is representational collapse (affecting \({\approx}55\%\) of runs at \(Q=50\)), where aggressive learning rates (\(\eta\ge 10^{-2}\)) destroy predictive capabilities, yielding near-constant outputs that artificially inflate loss. Difficulty calibration identifies these false positives: if \(M_{\text{ref}}\) achieves identical zero-shot separation, the signal reflects pre-existing dataset bias, not memorization. A secondary artifact is loss inversion (\({\approx}19\%\) of runs), where pre-trained structural biases drive member loss higher than non-member loss. A detailed breakdown of all outcomes, including context overfitting, is deferred to App. 8.2, with results for all datasets in Tabs. 10–11. After accounting for these fine-tuning artifacts, we isolate memorization signal in \(12.0\%\) of runs at \(Q=50\) (Tab. [fig:aggregate95outcomes]), and 8 of the 10 evaluated tasks in total under typical fine-tuning configurations (\(\eta\le 10^{-3}\)), exemplified in Tab. 1. More granularly, the TPR at low FPR metric reveals localized memorization. On beer_ratings at \(Q=512\), IclMem achieves only \(\text{AUC} = 0.59\) but identifies 49% of member records at 10% FPR; this dataset retains detectable memorization at \(Q=512\) due to its high target cardinality and small size, as analyzed below. On buy_buy_baby, we reach \(\text{TPR}_{@1\%} = 0.12\), indicating that in some instances fine-tuning records can be confidently distinguished.

Drivers of Memorization. We find that memorization depends on the interaction between query size \(Q\), target cardinality \(k\), and dataset size \(n\). Firstly, expanding \(Q\) mitigates memorization in our evaluation: at \(Q=512\), the overall detectable rate drops to \(1.8\%\) (a \(6.7\times\) reduction versus \(Q=50\)), as larger queries stabilize batch variance and enforce stronger zero-shot inductive biases. Secondly, at low query sizes (\(Q=50\)), memorization is driven by label density (\(n/k\), the average records per class). Binary tasks (\(k=2\), e.g., chocolate_bar_ratings) concentrate the gradient signal for specific classes per step, yielding the highest vulnerability (up to \(55\%\) detectable runs). As \(k\) grows, this concentration dissolves. Finally, high-cardinality tasks require two conditions to trigger memorization: high query coverage (\(Q/n\), the dataset fraction seen per step) and a target that is not structurally deducible. If a target cannot be logically inferred from its features alone (e.g., exact prices or subjective float ratings in beer_ratings), the model is forced to fall back on parametric memory, provided \(Q\) covers enough of the dataset. Conversely, if a target can be deduced via ICL (e.g., text-statistic formulas in clear_corpus), the model relies on in-context reasoning and yields no memorization signal, regardless of \(Q\).

Temporal Evolution of Memorization. Across fine-tuning epochs (Fig. 2), the \(Q=50\) signal emerges at epoch 1 (\(4.2\%\) of configurations) and grows to \(20\%\) by epoch 250. At \(Q=512\), memorization is largely suppressed: it peaks at \(5.5\%\) around epoch 50 and vanishes after, showing that larger queries reduce memorization and accelerate its decay.

6 Discussion and Conclusion↩︎

Next, we discuss limitations, mitigations, and future work.

Limitations. First, while IclMem detects memorization, the signal is moderate: only 20% of tasks show \(\text{TPR}_{@1\%} > 0.1\). Second, our probe uses an artificial setup (single task, fixed context-query pairs) to induce memorization. In contrast, realistic LTM regimes pre-train on large task mixtures for only 2–5 epochs [4], which dilutes memorization. We empirically verify that under joint multi-task training with random context-query sampling at \(Q=512\), detectable tasks drop from 5 to 1, with the signal appearing only at epoch 10, i.e., beyond realistic budgets (App. 10). Finally, while our evaluation is broad, our scope is currently limited (1 model, 10 tasks) and to be expanded.

Mitigations. To protect sensitive data, appropriate measures must be taken. Differential privacy (e.g., DP-SGD [38]) provides rigorous privacy guarantees that requires careful privacy-utility trade-offs but also shows promising utility when fine-tuning a public base model [39], [40]. Additionally, our findings show LTM memorization in certain training conditions that can be avoided (e.g., small query size with fixed query-context pairs over many epochs on single task, App. 10) or inform data filtering of memorization-prone samples (albeit with potential for privacy onion effects [41]).

Conclusion and Future Work. We initiated the systematic study of memorization in LTMs. In our controlled setup IclMem detects moderate memorization signal in 8 out of 10 evaluated tasks (\(\text{AUC}\) up to \(0.67\) and TPR at \(1\%\) FPR \(>0.1\)). Our results demonstrate that, under specific conditions, tabular foundation models can memorize data, underscoring the need for appropriate protective measures when sensitive data is involved. In future work, we aim to expand the evaluation to other LTMs [3], [6] and datasets [42]. We also aim to design data canaries (i.e., crafted samples with unique fingerprints or improbable feature-label mappings [43]) to amplify memorization signal and bound the worst-case detection limits of IclMem.

7 Detailed Probing Methodology and Pipeline↩︎

Figure 3 illustrates the IclMem pipeline, which consists of two main stages. (I) Establishing a memorization ground truth via data pre-processing and controlled fine-tuning. Here, we create a verified member/non-member split, optionally randomize labels as a null-hypothesis control, and log all context-query pairs for ground-truth verification. (II) Probing memorization using a multiple-choice protocol with adversarial context manipulation. Here, we build a zero-information context, apply a suite of 48 context manipulations, perform model inference under both the vanilla and manipulated contexts, aggregate scores, and compute delta scores against a pre-trained reference model for difficulty calibration.

Tab. 2 lists all fine-tuning and probing hyperparameters used in our evaluation. Tab. 3 defines the classification criteria for the five evaluation outcomes.

7.1 Addressing Common Pitfalls↩︎

Tab. 4 summarizes the four common MIA evaluation pitfalls identified by [44] and the corresponding countermeasures implemented in IclMem.

7.2 Dataset Details↩︎

Table 5 lists the 10 tasks evaluated in our experiments, all drawn from the CARTE benchmark [37]. We selected these datasets to cover a heterogeneous mix of real-world classification and regression domains. Crucially, all evaluated datasets are strictly excluded from the ConTextTab pre-training corpus to ensure a valid zero-shot baseline.

Member/Non-Member Split Validation. Tab. 6 reports distributional similarity between the member and non-member splits for all 10 datasets at both \(Q=50\) and \(Q=512\). For regression, we calculate the Kolmogorov-Smirnov (KS) statistic [35] on the continuous target values, i.e., the maximum difference between the empirical cumulative distribution functions, to bound the distance between splits. For classification, we compute the Total Variation Distance (TVD) [34] over the label distribution. All KS statistics are below \(0.05\) (maximum: \(0.035\) for buy_buy_baby at \(Q=512\)). Furthermore, all KS \(p\)-values are above the \(0.05\) significance threshold (minimum: \(0.530\) for buy_buy_baby at \(Q=512\)), meaning we fail to reject the null hypothesis that the member and non-member splits are drawn from the same distribution. This confirms that no statistically significant distributional shift exists between our data partitions. Similarly, TVD values for classification tasks are small (\(\leq 0.056\)), indicating near-identical label distributions across splits. We additionally report the Wasserstein distance for completeness [45]; note that this metric is scale-dependent and therefore not directly comparable across datasets.

0.4em

7.3 Context Manipulations↩︎

The probing stage subjects each multiple-choice context to a structured suite of 48 manipulations \(\Pi\), designed to degrade context utility and expose any reliance on parametric memory. Tab. 7 enumerates all configurations across 11 categories of manipulation types. Below we describe the 4 main categories.

1. Positional Shuffling: Alters the row index of the true target within the context (random, first, or last position). LTMs are designed to be permutation-invariant to row order; a model relying solely on ICL should produce identical predictions regardless of ordering. However, fine-tuning on static, fixed-order context-query pairs can introduce positional biases if optimization exploits ordering shortcuts. This manipulation tests whether any membership signal is feature-encoded in the model’s weights or an artifact of positional over-indexing.

2. Distribution Skewing: Skews the marginal label distribution in the context. Skew true over-represents the true target (amplification factor \(\in \{2,3,5\}\)); Skew incorrect over-represents \(k \in \{1,2,3\}\) incorrect targets (factor \(\in \{2,3,5\}\)); Mixed skewing adjusts both simultaneously. Uniform and random scaling amplify all candidate frequencies uniformly or randomly (factor \(\in \{2,5,10\}\)).

3. Feature Alteration: Feature reduction drops 2 valid feature columns to obscure pattern recognition; Distractor injection appends 1–2 entirely random columns to introduce spurious signal. Both test whether the memorization signal survives changes to feature space.

4. Combined Stress Tests: 2-way combinations pair positional shuffling with a single distributional or structural manipulation. 3-way stress tests simultaneously apply spatial, distributional, and structural degradation.

The count column in Tab. 7 specifies the number of distinct parameter configurations per category; the combinatorial categories (2-way, 3-way combinations) compose multiple single-axis manipulations to stress-test signal robustness simultaneously. Non-members rely entirely on the context for generalization and should exhibit clear loss spikes under strong degradation. Members, whose target mapping is encoded in the model’s weights, should maintain a stable probe loss across all manipulations.

Perturbation Effectiveness. We evaluate perturbation effectiveness on the detectable memorization configurations, measuring the mean loss-based \(\text{AUC}(\Delta S)\) per perturbation. Distribution skewing is the most effective single perturbation: amplifying the true target label by \(2\times\) yields AUC \(=0.557\) (\(\Delta{+}0.015\)), and simultaneously amplifying all labels by \(5\times\) yields the highest observed AUC of \(0.559\) (\(\Delta{+}0.017\)). This aligns with the design rationale: a memorized member resists context pressure because the correct label is encoded in its weights, whereas a non-member simply follows the over-represented label. Also, as expected, Positional shuffling is entirely ineffective (AUC \(\approx 0.543\), \(\Delta\approx 0\)), confirming that memorization is stored as weight-level feature-label associations rather than positional heuristics, and that the model is indeed permutation-invariant to row order. Feature reduction and distractor injection produce negligible gains (AUC \(0.543\)–\(0.546\)). Combining positional shuffling with distribution skewing (shuffle_and_skewed_2: AUC \(=0.557\)) does not improve over skewing alone, and heavily over-representing incorrect labels (\(5\times\)) slightly suppresses the signal (AUC \(=0.539\)), likely by reducing prediction confidence across all labels including the true one. No individual perturbation amplifies the signal by more than \(\Delta\text{AUC}\approx 0.017\); the practical value of the full perturbation suite lies in aggregating these stable but moderate individual signals into a more robust membership discriminator. Future work could explore more aggressive perturbations (e.g., based on label cardinality or feature importance) to further amplify signal.

7.4 The Necessity of Difficulty Calibration↩︎

Raw, absolute probing metrics are unreliable for auditing memorization [33]. Pre-trained models already encode strong semantic priors and structural logic, achieving high prediction confidence on inherently easy datasets without task-specific fine-tuning. Relying on absolute metrics confounds this pre-existing structural bias with training-induced memorization.

Impact on True Positive Rates (TPR). Across our \(530\) structurally healthy (non-collapsed) configurations, raw loss evaluation flagged \(241\) runs with uncalibrated \(\text{TPR}_{@10\%} \ge 0.10\). Difficulty calibration against \(M_{ref}\) revealed that \(230\) of these cases (\(95\%\)) were purely data artifacts: the pre-trained model naturally extracted these records prior to fine-tuning due to inherent sample easiness. Applying the calibrated margin (\(\Delta\text{TPR}_{@10\%} \ge 0.10\)) reduces this to \(11\) configurations with detectable memorization.

Impact on AUC-ROC. Global distributional metrics exhibit the same bias. If a dataset’s features correlate strongly with a target label under the model’s pre-training priors, \(M_{ref}\) may yield a raw AUC of \(0.85\) through zero-shot inference alone. An uncalibrated audit would flag a subsequent fine-tuned AUC of \(0.87\) as a large signal. Computing delta scores \(\Delta S(q) = S(q) - S_{ref}(q)\) and then \(\text{AUC}(\Delta S)\) correctly identifies this as a structural artifact: since the fine-tuning contributes minimal additional discriminative power, \(\text{AUC}(\Delta S) \approx 0.5\), indistinguishable from random guessing. Instance-level calibration via \(\Delta S(q)\) is therefore necessary to isolate parametric memorization.

8 Extended Evaluation Analysis↩︎

While Section 5 in the main text summarizes the five observed outcomes and their aggregate rates, this appendix provides a deeper analytical breakdown. We first detail the dominant artifact outcomes (representational collapse and loss inversion), which account for the majority of configurations, then analyze the secondary edge-case outcomes (context overfitting and zero-signal runs), track the temporal evolution of the memorization signal across fine-tuning epochs, and explore the disconnect between global and localized memorization.

8.1 Artifact Outcomes: Collapse and Inversion↩︎

Outcome I: Representational Collapse. Affecting over half of our configurations (\(55.2\%\) at \(Q=50\); \(50.9\%\) at \(Q=512\)), representational collapse is the dominant fine-tuning artifact. At aggressive learning rates (\(\eta\ge 10^{-2}\)), LTMs lose their in-context reasoning capability and output near-constant predictions. While this yields high uncalibrated AUCs (\(> 0.7\)), difficulty calibration confirms these are pre-existing dataset biases rather than induced memorization. We formally flag collapsed models using an empirically derived threshold: \(\mathcal{L}_{FT} > 5 \times \mathcal{L}_{Base}\). Across our 1,128 runs, the loss ratio (\(\mathcal{L}_{FT} / \mathcal{L}_{Base}\)) follows a strict bimodal distribution. Stable configurations (\(\eta\le 10^{-4}\)) tightly cluster between \(1.0\)–\(2.0\times\), whereas aggressive rates cause rapid degradation, pushing median ratios above \(8.0\times\) (and frequently exceeding \(10^4\)). This \(5\times\) boundary robustly separates structurally broken models from true parametric memorization.

Outcome II: Loss Inversion. In \(19.3\%\) of configurations at \(Q=50\), there is a further artifact: loss inversion, where non-members yield lower mean probe loss than members after fine-tuning. This occurs when training loss is already inverted in the pre-trained model, or when the collapsed model’s static predictions align better with non-members. Our framework addresses this structural bias by evaluating distributional metrics (i.e., entropy and confidence) which remain discriminative even when loss is inverted.

8.2 Secondary Dynamics and Edge Cases↩︎

Our methodology isolates two additional outcomes beyond detectable memorization and representational collapse.

Outcome IV: Context Overfitting. In 5.5% of runs at \(Q=50\), the model learns statistical properties of the dataset schema rather than individual record membership. To classify this outcome, we define \(g_k = \bar{\ell}^{M}_{\text{atk}}(M_\theta, \pi_k) - \bar{\ell}^{NM}_{\text{atk}}(M_\theta, \pi_k)\) as the mean member minus non-member probe loss gap under perturbation \(\pi_k\), and let \(\mu_{\text{gap}}\) and \(\sigma_{\text{gap}}\) denote its mean and standard deviation across all 48 manipulations in \(\Pi\). The mean probe loss gap (\(\mu_{\text{gap}}\)) is near zero, but \(\sigma_{\text{gap}}\) is minimal across all 48 context manipulations: the zero is stable on every perturbation, not noisy. This indicates the model treats members and non-members identically under all context manipulations, having overfitted to the table format rather than individual rows.

Outcome V: No Signal. This outcome captures configurations (8.0% at \(Q=50\); 20.8% at \(Q=512\)) where IclMem produces no systematic pattern. Noisy and inconsistent per-manipulation probe loss gaps (\(g_k\)) indicate that fine-tuning improved task performance without inducing a detectable row-level membership shift.

8.3 Overall memorization (AUC) vs confident instance signal (TPR at low FPR)↩︎

Our calibrated audit detected memorization (\(\text{AUC}(\Delta S) > 0.5\)) in 8 of the 10 evaluated tasks under specific hyperparameter regimes, despite representational collapse and chance predictions being the predominant outcomes. Tab. 8 reports the peak vulnerability configuration for each of these 8 tasks: learning rate, epoch, context size (\(Q\)), and the label randomization condition where the peak signal was detected. Normal indicates the true-label condition; Shuffled (M) indicates member-only label scrambling; Shuffled (All) indicates full-dataset scrambling. We report the calibrated \(\text{AUC}(\Delta S)\) to capture global memorization trends across all records, complemented by \(\text{TPR}\) at low \(\text{FPR}\) (\(\text{TPR}_{@10\%}\) and \(\text{TPR}_{@1\%}\)) to measure localized signal of individual samples. We report \(AUC\) and \(TPR\) for the metric (loss or entropy) yielding the maximum signal.

Global vs.Localized Memorization Signal. Several configurations in Tab. 8 yield a positive global memorization signal (\(\text{AUC}(\Delta S) > 0.5\)) while simultaneously exhibiting low TPR at strict FPR thresholds. Fine-tuning can induce a global distributional shift that separates members from non-members on average (AUC) without concentrating that signal at the individual-record level: \(M_{ref}\) may achieve higher extreme-tail confidence on a subset of records purely through zero-shot structural bias. A detectable global signal therefore does not imply a strong localized one; row-level memorization is confined to a smaller subset of configurations under specific hyperparameter regimes.

8.4 Data Characteristic Analysis↩︎

Fig. 6 plots all 10 evaluated datasets in the target-cardinality–majority-class-frequency space, colored by memorization outcome category. Tab. 9 reports the underlying statistics and per-\(Q\) detectable memorization run counts.

4pt

Feature Composition. CARTE datasets are mixed with both categorical and numerical features: beer_ratings has 14 of 19 features as numeric, i.e., the highest ratio among all memorized datasets, producing near-unique feature vectors that ICL cannot generalize from, contributing to its parametric memorization at \(Q=512\). In contrast, clear_corpus (15/26 numeric) has similar richness but its continuous readability targets are feature-predictable, suppressing memorization. For predominantly categorical datasets, feature type does not discriminate memorized from non-memorized tasks; target cardinality and class concentration remain the primary factors.

9 Exhaustive Hyperparameter Results↩︎

Tabs. 10 and 11 report outcome metrics for \(Q=50\) and \(Q=512\) respectively, covering all 8 CARTE datasets and all evaluated learning rates. For each query size, we report the epoch yielding the highest signal of the reported outcome per learning rate \(\eta\), selected independently for each \(Q\).

0.2em

0.2em

10 Memorization under Realistic Training Regimes↩︎

Memorization Signal under Realistic Regimes (\(Q{=}512\)). Tab. 12 shows how memorization degrades as the training regime progressively approaches practical conditions. Under single-task fine-tuning on fixed context-query pairs (worst case), 5 of 10 tasks are detectable. By fine-tuning a single model jointly on all 10 tasks with fixed context-query pairs, i.e., 10 gradient steps per epoch, the signal appears only for 2 tasks (babies_r_us: AUC 0.668\(\to\)​0.618; bikedekho: AUC 0.579\(\to\)​0.558). We intentionally fixed the context-query pairs across epochs, however, in practice the pairs are randomly sampled. With random context-query sampling and fine-tuning on all 10 tasks, the signal collapses to a single task: bikewale (AUC 0.575). Critically, this residual signal appears only at Epoch 10, already beyond the 2–5 epoch budget of real LTMs pre-training [4]; within that practical budget, no task would be detectable. The pattern confirms that memorization under IclMem requires two structural conditions absent from practical pre-training: (i) per-task gradient specialization and (ii) repeated exposure to fixed context-query pairs.

References↩︎