1 Introduction↩︎

TFMs enable accurate predictions on structured data via ICL: a small labeled dataset is placed in the model context and used to predict labels for new queries, without task-specific training [1], [2]. This makes TFMs appealing in domains such as healthcare and finance, where tabular data is common and labeled data is often scarce [3], [4]. However, the context records may contain private information.

We first verify that this concern is not merely hypothetical. In ICL, membership inference asks whether a target record was included in the private demonstration context. Adapting standard MIAs [5], [6] to tabular ICL, we find that prediction behavior reveals non-trivial information about context membership. Even a passive grey-box adversary obtains \(9.9\%\) TPR at \(1\%\) FPR, while stronger active attacks can reach substantially higher leakage. These results motivate mechanisms that protect the in-context data itself, rather than only the pretraining data of the foundation model.

DP [7] provides a formal way to limit the influence of any individual private record. Yet standard DP training methods are not directly suited to tabular ICL. DPSGD [8] requires gradients and parameter updates, whereas TFMs are used as frozen inference-time models. PATE [9], [10], in contrast, is naturally compatible with ICL as private data can be split across teacher contexts, whose predictions are aggregated with calibrated noise. This idea underlies PromptPATE [11] in the language domain. Like all prior PATE-style methods, PromptPATE requires unlabeled public data from the target distribution for knowledge transfer. In sensitive tabular domains, such public in-distribution data is often unavailable.

We introduce TabPATE, a central-DP framework for tabular ICL that removes this public-data requirement. TabPATE partitions the private context across teacher TFMs, queries them on synthetic tabular records, privately aggregates their votes using Confident-GNMax algorithm [10], and releases the resulting labeled synthetic records as a student context for privacy-preserving deployment. The key observation is that tabular domains differ from images and text: feature spaces are bounded, typed, and comparatively low-dimensional. Thus, useful teacher queries can be generated without public examples, either from a data-independent bounds-based prior or from a small DP release of marginal statistics. This yields a PATE-style defense for tabular ICL with no public in-distribution data.

We evaluate our TabPATE on standard tabular benchmarks and compare against non-private ICL, PromptPATE, query-time aggregation, and DP synthetic-data baselines. TabPATE preserves competitive utility at moderate privacy budgets while reducing MIA success close to random guessing. Our results suggest that public-data-free PATE-style knowledge transfer is a practical route to private deployment of TFMs. In summary, we make the following contributions:

1. We adapt membership inference to tabular ICL and show that private context records leak through model predictions, motivating formal protection.

2. We propose TabPATE, a central-DP PATE-style framework for tabular ICL that replaces public in-distribution transfer data with synthetic tabular queries.

3. We evaluate TabPATE across tabular benchmarks, showing competitive utility at moderate privacy budgets and near-random membership inference success.

2 Related Work↩︎

2.0.0.1 Tabular foundation models.

TFMs perform prediction on structured data via transformer-based ICL [12], [13]. Given labeled demonstrations in the context, TFMs predict labels for new queries without updating model parameters. TabPFN [1], [2] is a prominent example, pretrained on synthetic tasks and competitive with strong baselines. Recent models such as TabDPT [14] and TabICL [15], [16] further improve tabular ICL performance, with benchmarks tracking rapid progress [17], and applications to adjacent fields [18], [19]. We study the privacy of this inference-time learning paradigm, where sensitive records are placed directly in the model context.

2.0.0.2 Membership inference for ICL.

MIAs [5] are the standard empirical tool for measuring whether individual records influence model behavior. Modern attacks include calibrated likelihood-ratio attacks such as LiRA [6], label-only perturbation attacks [20], and poisoning-based amplifications such as Truth Serum [21]. For text ICL, Brainwash-style attacks show that context examples can leak through resistance to adversarial relabeling [22]. We adapt these attack ideas to tabular ICL to motivate the need for formal protection.

2.0.0.3 Differential privacy and PATE.

DP [7] bounds the effect of any single private record on a released computation. DPSGD [8] achieves this during training via gradient clipping and noise, but is not directly applicable to frozen ICL models. PATE [9], [10] instead partitions private data among teachers and transfers knowledge through noisy vote aggregation, making it a natural fit for ICL. PromptPATE [11] applies this idea to language-model prompting, but like standard PATE, relies on public in-distribution data for teacher queries. Other DP-ICL methods either answer each query privately at inference time [23] or target autoregressive text generation [24]. TabPATE follows the PATE-style student-release approach, but removes the public-data requirement by generating synthetic tabular queries.

2.0.0.4 DP tabular synthetic data.

DP synthetic data methods such as AIM [25] and MST [26] privately estimate low-dimensional marginals and post-process them with Private-PGM [27] to generate synthetic tables. We use such methods as baselines, but TabPATE differs in goal: it does not aim to release a synthetic version of the private dataset, but only to generate useful unlabeled queries for private PATE-style knowledge transfer.

3 Membership Inference in Tabular ICL↩︎

Before introducing TabPATE, we verify that private in-context records can leak through the predictions of a TFM. In standard training-based MIAs, membership asks whether a record was used for training [5]. In tabular ICL, the pretrained model is fixed; MIAs instead ask whether a target record was included in the private demonstration context used at inference time.

We consider adversaries along two axes: output access, where the model returns either labels only (black-box) or prediction probabilities (grey-box), and context manipulation, where the adversary is either passive or can inject additional examples into the context. This yields four threat models, detailed in 7.1. We adapt representative attacks to each setting: label-only perturbation [20], LiRA [6], Brainwash-style context manipulation [22], and LiRA with Truth Serum-style poisoning [21].

All attacks use the same LiRA-style calibration. For each target \(x_q\), we sample shadow demonstration sets, half including and half excluding \(x_q\), and extract an attack-specific signal \(s(x_q)\) from the model output. We fit per-sample Gaussian IN and OUT distributions and score membership by the log-likelihood ratio \[\mathrm{score}(x_q)= \log \frac{p_{\mathrm{IN}}(s(x_q))}{p_{\mathrm{OUT}}(s(x_q))}. \label{eq:lira95score}\tag{1}\] The attacks differ only in the signal: prediction stability under perturbations, logit-scaled confidence, number of mislabeled context copies required to flip the prediction, or confidence after poisoning.

1 shows that tabular ICL leaks membership information across all threat models. Even passive grey-box access yields a clear signal (\(0.73\) AUC, \(9.9\%\) TPR at \(1\%\) FPR). Active context manipulation substantially amplifies leakage: LiRA with poisoning reaches \(0.96\) AUC and \(75.4\%\) TPR at \(1\%\) FPR on average. More results are presented in 7. These results establish that using sensitive records directly as demonstrations creates measurable privacy risk, motivating the DP defense we introduce next.

4 Our TabPATE: Private Tabular ICL↩︎

TabPATE is a PATE-style mechanism for releasing a private ICL context for a TFM. Given a sensitive labeled dataset \(\mathcal{D}\), our goal is to produce a new labeled student context \(\widetilde{\mathcal{D}}\) that can be used for downstream predictions without further access to \(\mathcal{D}\). TabPATE follows the teacher–student structure of PromptPATE [11], but removes its requirement for unlabeled public in-distribution data.

We first partition \(\mathcal{D}\) into \(K\) disjoint subsets. Each subset defines an ICL teacher: the same frozen TFM prompted with that subset as its demonstration context. We then query all teachers on unlabeled synthetic tabular records \(\tilde{x}\) and aggregate their votes with Confident-GNMax [10]. If the teachers agree sufficiently, the mechanism releases a noisy plurality label \(\tilde{y}\); otherwise it abstains. The resulting pairs \((\tilde{x},\tilde{y})\) form the student context \(\widetilde{\mathcal{D}}\), which is released and used with the frozen TFM. Since downstream predictions depend only on this released context, they incur no additional privacy cost.

The main departure from PromptPATE is how the unlabeled teacher queries are obtained. Standard PATE-style methods assume public samples from the target distribution. TabPATE instead exploits the structured nature of tabular data: features have known types and bounded ranges, so candidate records can be generated from a schema rather than a public dataset. We use a parameter \(\alpha \in [0,1]\) to split the privacy budget \(\varepsilon\). When \(\alpha=0\), queries are sampled from a fixed, data-independent prior over the feature ranges, incurring no privacy cost before PATE aggregation. When \(\alpha>0\), TabPATE spends \(\alpha\varepsilon\) to release simple DP marginal statistics of the private context, and samples queries from the resulting product distribution; the remaining \((1-\alpha)\varepsilon\) is used for Confident-GNMax labeling. We empirically observe that in balanced classification setups, \(\alpha=0\) is sufficient, but that more complex problems, such as classification of imbalanced datasets or regression (see 5.4) benefit from setting a small positive \(\alpha>0\). The resulting method provides central \((\varepsilon,\delta)\)-DP for the released student context. The full algorithm and privacy proof are given in 8.

5 Experiments↩︎

We evaluate whether TabPATE provides useful private ICL without public in-distribution data. We focus on four questions: (i) how much utility TabPATE preserves under DP, (ii) how it compares to other DP baselines, (iii) whether it reduces the MIA risks from §3, and (iv) whether it is able to extend from pure classification to regression setups, another core application of TFMs.

5.1 Setup↩︎

For the main experiments, we evaluate on five OpenML tabular classification datasets used in prior tabular ICL work: German Credit (Credit-G) (\(n{=}1000\)), Blood Transfusion (\(n{=}748\)), Diabetes (\(n{=}768\)), Phoneme (\(n{=}5404\)), and Wilt (\(n{=}4839\)). We use TabPFN as the TFM [1], [2]. Unless stated otherwise, we use \(\alpha=0\) and report the results as averaged over five random seeds. We report balanced accuracy as the main utility metric, and evaluate privacy at \(\varepsilon\in\{1,10\}\) with \(\delta=10^{-5}\).

5.1.0.1 Baselines.

We compare TabPATE against five DP baselines covering complementary privacy strategies. As a non-ICL reference, we train a task-specific RealMLP [28] with DP-SGD. For releasable ICL contexts without public data, DP-Synthetic generates a private synthetic dataset from the full sensitive data using AIM [25] or MST [26], while DP-TabICL [29] releases private prototypes; we evaluate its central-DP variant since the local-DP variant scales exponentially with the number of binary features. We also include PromptPATE [11], which is closest to TabPATE but assumes access to unlabeled public in-distribution data for the PATE knowledge transfer. Finally, Query-Time is a sample-and-aggregate baseline inspired by prior private query answering [23], [30]: it partitions the private context among teachers and answers each downstream query through noisy ensemble aggregation, incurring additional privacy cost per query. A full overview of the baselines is provided in 9.

5.2 Utility↩︎

6pt

\(^\dagger\)Requires public data from target distribution; we use 30% held-out training data.

2 reports balanced accuracy averaged across the five datasets. At \(\varepsilon=10\), TabPATE achieves the best utility among the evaluated private methods, reaching \(71.2\%\) balanced accuracy without requiring public data, compared to \(68.6\%\) for PromptPATE, which uses held-out in-distribution data, and \(61.3\%\) for Query-Time. At the stricter budget \(\varepsilon=1\), PromptPATE benefits from real in-distribution queries, but TabPATE still outperforms the public-data-free baselines. Full standard-accuracy results and per-dataset breakdowns are given in 10.1.

5.3 Attack Mitigation↩︎

We next verify that our TabPATE mitigates the membership leakage observed for non-private tabular ICL. We attack the released student with the strongest passive grey-box attack, LiRA, and compare the resulting ROC curves in 1. Both TabPATE and PromptPATE substantially reduce attack success in the low-FPR regime: for the non-private TabPFN, LiRA reaches an AUC of about \(0.76\) and a TPR@1%FPR of \(10.9\%\), whereas TabPATE reduces the attack to near-random performance at both \(\varepsilon=1\) and \(\varepsilon=10\). PromptPATE shows a similar mitigation effect, but requires held-out in-distribution public data. Thus, TabPATE provides comparable empirical protection against MIAs while retaining the main practical advantage of requiring no public data. We present more results obtained on larger datasets in 10.3.

5.4 Regression↩︎

TFMs are also used for regression, so we finally evaluate whether TabPATE extends beyond classification. For regression, we replace the noisy teacher vote with a bounded noisy aggregation of teacher predictions. We first normalize the per-teacher regression target to \([0,1]\) and clip each teacher prediction to this range to bound sensitivity. For a synthetic query \(\tilde{x}\), the \(K\) teachers output clipped scalar predictions \(\hat{y}_1,\ldots,\hat{y}_K \in [0,1]\), which we aggregate by their mean and perturb with Gaussian noise. Since changing one private record can affect at most one teacher, the sensitivity of the averaged prediction is bounded by \(1/K\). The noisy label is then clipped to \([0,1]\), mapped back to the original target scale, and used as a demonstration for the student. As before, releasing the student context is post-processing of the private aggregation.

3 reports normalized RMSE (NRMSE; lower is better) on eight regression datasets. TabPATE remains effective without public data, though regression is more challenging than classification. PromptPATE obtains the best private utility because it uses real in-distribution public queries. Among public-data-free TabPATE settings, allocating a small budget share to marginal query generation improves over purely data-independent queries on almost all datasets, reducing mean NRMSE from \(0.806\) to \(0.759\). This supports the role of \(\alpha>0\) in settings where random queries insufficiently cover useful regions of the feature space.

2.5pt

\(^\dagger\)Requires public data from the target distribution.
\(^*\)All datasets use \(\alpha=0.05\), except Wine Quality (\(0.38\)) and Calif. Housing (\(0.21\)).

6 Conclusion↩︎

We showed that tabular ICL leaks sensitive context records through membership inference and introduced TabPATE, a PATE-style DP defense for tabular foundation models. TabPATE releases a private student context using synthetic queries from feature bounds or optional DP marginals, avoiding the need for public in-distribution data. Empirically, it preserves utility while reducing MIA success to near-random levels, suggesting a practical route to private tabular ICL.

7 Additional Membership Inference Details↩︎

7.1 Threat Models and Attack Signals↩︎

We evaluate four MIAs against tabular ICL, corresponding to the two output-access levels and two context-manipulation levels in 4. All attacks use the LiRA-style calibration from 1 .

7.1.0.1 Perturbation.

For each target \(x_q\), we add Gaussian noise scaled by feature standard deviation and measure the fraction of perturbed samples that preserve the original prediction: \[s(x_q)=\frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[f(x_q+\epsilon_i)=f(x_q)].\] Members tend to be more stable because their input-output mapping is present in the context.

7.1.0.2 LiRA.

For grey-box passive access, we use the logit-scaled confidence on the true label, \[s(x_q,y_q)= \log \frac{p_f(y_q\mid x_q)}{1-p_f(y_q\mid x_q)}.\] Context members tend to receive higher confidence.

7.1.0.3 Brainwash.

For black-box active access, the adversary repeatedly adds copies of \(x_q\) with an incorrect label and records the minimum number of copies required to flip the prediction. Members require more copies because the correct label is already reinforced in the context.

7.1.0.4 LiRA + Poisoning.

For grey-box active access, the adversary injects \(k\) mislabeled copies of \(x_q\) and then applies LiRA-style calibration to the poisoned signal. Poisoning separates member and non-member distributions because non-members have only the injected incorrect label in the context, while members also have the original correct record.

7.2 Per-Dataset Perturbation Results↩︎

Even under label-only passive access, the attack performs above random on all datasets, although the signal is modest (5). Vehicle is the most vulnerable in this setting, consistent with the broader trend that higher-dimensional and multi-class datasets provide richer prediction signals.

7.3 Poisoning Amplifies Membership Inference↩︎

Poisoning strongly amplifies leakage. In 6 with \(k=20\) mislabeled copies, the average AUC increases from \(0.73\) to \(0.96\), and several datasets become almost perfectly distinguishable. This corresponds to a small fraction of the context in our experiments, showing that active manipulation can expose membership even when passive attacks are weaker.

7.4 Comparison to Threshold-Based Attacks↩︎

We compare LiRA calibration to common threshold-based attacks using loss, confidence, prediction gap, entropy, correctness, and modified entropy in 7. LiRA is consistently strongest, with mean AUC \(0.73\) compared to \(0.55\) for the strongest threshold baseline. This supports using calibrated per-sample likelihood ratios as the main MIA evaluation.

7.5 Perturbation and Attack-Cost Ablations↩︎

The label-only perturbation attack is robust to its main hyperparameters. Varying the number of perturbations (¿tbl:tab:perturbation95count?) and perturbation scale ([tab:perturbation95scale]) changes AUC only marginally, remaining in the range \(0.52\)–\(0.55\) on German Credit. Thus, its limited success is due to the restrictive black-box passive setting rather than a poorly tuned perturbation configuration.

Because all attacks use the same number of shadow demonstration sets, their total cost is dominated by the number of target queries. LiRA and LiRA+Poisoning require one query per target after calibration, perturbation requires \(N\) queries, and Brainwash requires up to the maximum poison budget.

7.6 Comparison to Text-Based ICL↩︎

We also compare TabPFN to text-based tabular ICL using TabLLM-style serialization [31] in 8. Both models leak membership, although TabPFN is more vulnerable in our experiment. This suggests that membership leakage is not an artifact of one implementation, but a broader risk of using sensitive records as in-context demonstrations.

7.7 Feature Dimensionality↩︎

We observe a moderate positive correlation between feature count and MIA success in 2. Higher-dimensional datasets, such as Vehicle, tend to be more vulnerable, likely because model outputs encode richer per-record information. However, dimensionality alone does not fully explain leakage, which also depends on label type, dataset size, and local sample uniqueness.

8 TabPATE Algorithm and Privacy Analysis↩︎

8.0.0.1 Query generation.

For \(\alpha=0\), query generation is data-independent; after normalizing features to known ranges, we sample from a fixed prior such as a uniform distribution over the feature domain or a Gaussian centered at the midpoint of each range. This step has zero privacy cost. For \(\alpha>0\), we estimate per-feature means and variances with DP. Concretely, after clipping each feature vector to \(\ell_2\) norm at most \(C\), we release noisy versions of \[q_1(\mathcal{D})=|\mathcal{D}|,\qquad q_2(\mathcal{D})=\sum_{x\in\mathcal{D}} x,\qquad q_3(\mathcal{D})=\sum_{x\in\mathcal{D}} x\odot x .\] The sensitivities under add/remove adjacency are \(\Delta_1=1\), \(\Delta_2\le C\), and \(\Delta_3\le C^2\). We then compute \(\tilde{\mu}\) and \(\tilde{\sigma}\) from these noisy sufficient statistics. Sampling queries from these estimates is post-processing.

8.0.0.2 Privacy guarantee.

TabPATE operates in the central-DP model. We state the guarantee for the released student context \(\widetilde{\mathcal{D}}\).

9 Baselines↩︎

In the following, we present our baselines in more detail. We also provide an overview in 9.

DP-SGD (non-ICL baseline). As an additional reference point, we include a non-ICL baseline. We train a RealMLP [28] with DP-SGD.

DP-Synthetic Data. DP-Synthetic generates DP synthetic data from the full private dataset via AIM [25] or MST [26] and uses them as demonstrations for ICL of a student model.

DP-TabICL. DP-TabICL [29] releases \(k\) private prototypes from the private data. It splits the data into stratified splits. They propose an LDP and GDP variant. The former is limited to less than 14 binary features due to exponential scaling in the number of dimensions, hence we evaluate the latter, which computes DP per-feature means with the Laplace mechanism.

PromptPATE. PromptPATE [11] partitions the private data among ICL teachers and uses public in-distribution data for the private knowledge transfer via Confident-GNMax [10]. It requires access to unlabeled public data from the same distribution as the private data, which limits applicability in sensitive domains.

Query-Time. Query-Time, inspired by [23], [30], partitions the context data among teachers and directly answers queries at inference time with aggregated noisy ensemble predictions. Each query incurs additional privacy cost, requiring the system to stop once the budget is exhausted.

\(^\ddagger\)Non-ICL baseline: trains a task-specific model with DP-SGD instead of using ICL.

10 Additional Experimental Results↩︎

10.1 Extended Utility Results↩︎

This section reports the full utility results omitted from the main paper for space. 10 includes both balanced and standard accuracy averaged across datasets, while ¿tbl:tab:dp95results95per95dataset? provides the per-dataset balanced accuracy.

4pt

\(^\dagger\)Requires public data from target distribution; we use 30% held-out training data.

10.2 Effect of Query Generation↩︎

11 illustrates why marginal-based queries are useful in some settings. On Wilt, a highly imbalanced dataset, random uniform or Gaussian queries provide poor balanced accuracy, while DP marginal queries place more queries near the relevant data region and recover high utility.

10.3 Larger Classification Datasets↩︎

We evaluate TabPATE on larger datasets to test whether public-data-free PATE-style transfer remains useful beyond the small OpenML benchmarks [32]. 12 reports utility at \(\varepsilon=10\). We also measure passive grey-box MIA vulnerability of non-private tabular ICL on these datasets in 13. Non-private ICL remains vulnerable even at tens of thousands of samples, although leakage decreases on some larger datasets, consistent with the privacy onion effect [33] with potential fairness implications [34].

4pt

References↩︎