1 Introduction↩︎

Zeroth-order (ZO) optimization has emerged as a memory-tractable alternative to first-order LLM fine-tuning, replacing backpropagation with two forward passes per step [1]. When the training corpus is sensitive, trained weights are known to leak individual records through membership inference attacks (MIA) and verbatim extraction [2]–[6]. The dominant defense is differential privacy (DP), a quantitative guarantee that the influence of any single training record on the released model is provably bounded, parameterized by a parameter \(\varepsilon\). The workhorse instantiation, DP-SGD [7], clips per-record gradients to a fixed sensitivity budget and adds Gaussian noise tracked across steps via the Rényi accountant [8], [9]. This methodology extends to LLM fine-tuning [10], [11] with mature production tooling now in use [12]–[14]. Plugging it into ZO yields DP-ZO [15]–[17], which clips and noises the per-step zeroth-order scalar instead of the full gradient. Because the scalar is one-dimensional and the perturbation direction is data-independent, DP-ZO removes the explicit dimension dependence from the per-step noise. It does not, however, remove the impact on utility. Every published DP-ZO method inherits DP’s harsh utility-privacy trade-off: usable accuracy requires a loose privacy budget, and utility collapses as \(\varepsilon\) tightens, because the underlying mechanism (worst-case per-record sensitivity composed sequentially over \(T\) steps) is unchanged. This trade-off is a long-observed property of DP-SGD in iterative deep-learning training, where tight \(\varepsilon\) produces steep utility loss and disparate-impact effects [18]–[24]; reaching a meaningful high-privacy regime in LLM fine-tuning therefore requires a fundamentally different privacy accounting mechanism.

PAC Privacy [25] replaces DP’s worst-case sensitivity with a noise calibration based on the empirical covariance of the release, and bounds the mutual information (MI) between the secret and the release. The key advantage of PAC Privacy is that it exploits the inherent stability of the output and calibrates minimal noise for stable outputs. Recent work extends the framework to adaptive composition over inference-time response streams [26]. What remains open is the case relevant to private iterative fine-tuning, the same setting in which DP-SGD operates, where a per-step update is released to the adversary at every optimization step. At LLM scale (\(d\!\sim\!10^9\), \(T\!\sim\!10^3\)), however, optimal instance-based noise calibration derived from the output covariance is prohibitively expensive to achieve at this dimensionality, leaving iterative training out of reach of canonical PAC algorithms.

The proposed PACZero family closes both gaps at once, the high-privacy collapse of DP-SGD and the open problem of iterative-training PAC, by combining a zeroth-order substrate [1] with a per-step release that is sign-quantized. Concretely, PACZero aggregates per-sample two-point zeroth-order loss differences over \(M\!=\!128\) random subsets, sign-quantizes each subset mean to \(s_m\!\in\!\{-1,{+}1\}\), and releases one bit identifying the sign of the secret subset (Figure 1). When all \(M=128\) bits agree, this is unanimous and does not spend privacy budget under PAC privacy. The mechanism then instantiates as one of two variants depending on how the disagreement branch handles the released bit. The per-step release is now a Bernoulli with entropy \(\le\log 2\) regardless of \(d\), so the secret-to-release mutual information becomes a one-dimensional Gaussian integral that we evaluate exactly to calibrate the noise, replacing the variance-based Gaussian upper bound used in canonical PAC algorithms [25]–[27]. Two structural consequences follow from the quantized release. First, posterior-weighted unanimity becomes combinatorially detectable, when the posterior over the secret index puts all mass on subsets that agree on the sign, the released bit is a deterministic function of the public history and contributes zero conditional MI; this branch fires on \({\sim}34\)–\(45\%\) of training steps in our SST-2 cells. The base variant PACZero-MI calibrates the disagreement-branch Gaussian to a per-step MI budget \(\beta_t\). Second, the privacy budget can be driven exactly to zero by replacing the disagreement-branch Gaussian release with a uniform \(\mathcal{U}(\{-1,{+}1\})\) coin flip, and the resulting PACZero-ZPL variant satisfies \(I(S^*; Y_{1:T})\!=\!0\) for every \(T\), an MIA-resistance level matched in the DP framework only at \(\varepsilon\!=\!0\) (infinite per-step noise; see §2). Contributions are as follows:

• The PACZero family of PAC-private ZO mechanisms. We introduce PACZero², a family of mechanisms unified by sign quantization of subset-aggregated zeroth-order updates, with two concrete variants (PACZero-MI, PACZero-ZPL) that differ in how the disagreement branch is handled and span two points in the privacy-utility design space (§3). We evaluate both on SST-2 and SQuAD across OPT-1.3B and OPT-6.7B in both LoRA and full-parameter tracks (Table [tab:headline]).

• Exact binary-input Gaussian MI calibration We replace the variance-based Gaussian upper bound used by canonical PAC algorithms with the exact mutual information of the noised binary release, removing a calibration gap that grows unboundedly as the noise scale tightens (§3.1.0.1). PACZero-MI accuracy is flat across four decades of MI budget on SST-2 OPT-1.3B FT, reaching \(89.51\!\pm\!1.12\%\) at MI\(\,=\,0.33\) nats (matched-MIA DP \(\varepsilon\!\approx\!2\)), \(+2.9\)pp above the matched DPZero K\({=}1\) FT baseline [15] (Table 1).

• An MIA-resistance level no DP-ZO can match. The PACZero-ZPL variant guarantees \(I(S^*; Y_{1:T})\!=\!0\) for every \(T\) (MIA posterior at the prior), the first such guarantee for iterative LLM fine-tuning to our knowledge; matching it in DP requires \(\varepsilon\!=\!0\), to which every DP-SGD accountant assigns infinite noise (§2, §3.2).

Scope of the \(I\!=\!0\) guarantee. PACZero-ZPL protects the identity of the secret subset \(S^*\!\in\!\{S_1,\ldots,S_M\}\) sampled from \(\mathcal{D}\!=\!\mathrm{Unif}\{S_1,\ldots,S_M\}\) over a public candidate universe \(U\) (§2); under \(\mathcal{D}\) the prior MIA success rate per record is \(1/2\), and \(I(S^*;Y_{1:T})\!=\!0\) collapses the posterior to this prior. It is not a DP guarantee. DP at \(\varepsilon\!=\!0\) requires identical output distributions across all neighbors (and forces infinite per-step noise), which PACZero-ZPL does not satisfy. The matched-MIA-prior DP-\(\varepsilon\) annotations in our tables (e.g., “DP \(\varepsilon\!=\!0\)”) denote the DP value that yields the same MIA upper bound at prior \(1/2\), not a claim that PACZero satisfies \((\varepsilon,\delta)\)-DP (§6).

2 Background↩︎

Two-point zeroth-order gradient estimation. For a model with parameters \(\theta\in\mathbb{R}^d\) and a per-sample loss \(\ell_i(\theta)\), the two-point zeroth-order estimator [1], [28] samples a single direction \(z\sim\mathcal{N}(0,I_d)\) and computes the per-sample finite-difference scalar \[\hat{g}_i \;=\; \frac{\ell_i(\theta + \mu z) - \ell_i(\theta - \mu z)}{2\mu} \;=\; \langle \nabla \ell_i(\theta), z\rangle + O(\mu^2), \label{eq:zo-scalar}\tag{1}\] where \(\mu>0\) is the perturbation scale [1]. The parameter update is \(\theta \leftarrow \theta - \eta \cdot \hat{g} \cdot z\), where \(\hat{g}\) is obtained by clipping and averaging the \(\hat{g}_i\) over the batch [15], [17]. The crucial structural property, exploited by all DP-ZO methods [15]–[17], is that \(z\) is sampled independently of the data and is therefore public. The only data-dependent quantity per step is the scalar \(\hat{g}\). Privatizing the update therefore reduces to privatizing a scalar (not a \(d\)-vector), making the per-step noise scale dimension-independent.

PAC Privacy [25] provides an instance-based framework for measuring and bounding information leakage about a secret dataset from a mechanism’s output. We note that in Definition 1, privacy is established under the strong model of an informed adversary who knows \(\mathcal{D}\), and its protection over \(X\sim\mathcal{D}\) naturally extends to weaker adversaries without such knowledge. Given \(\mathcal{D}\) and \(\rho\), an informed adversary has a prior success rate even before observing any outputs, denoted as \(1-\delta_0 :=\max_{Q}\Pr_{X\sim \mathcal{D},\hat{X}\sim Q}[\rho(\hat{X},X)=1]\). A fundamental result of PAC Privacy [25] bounds the posterior advantage via mutual information (MI): \[\label{eq:pac95mi95bound} \mathrm{KL}(1-\delta_\mathcal{A}\,\|\,1-\delta_0)\leq I(X;\mathcal{M}(X)),\tag{2}\] where \(\mathrm{KL}(p \| q) := p\ln(p/q) + (1-p)\ln((1-p)/(1-q))\) is the binary KL divergence. We note that Inequality 2 holds simultaneously for every adversary and every attack criterion. Hence, one can specify and enforce a total MI budget, and derive concrete posterior success rate guarantees given the prior success rate of an informed adversary under \(\rho\).

When the mechanism interacts with a persistent secret over \(T\) rounds, releasing outputs \(Y_1, Y_2, \ldots, Y_T\), the chain rule of mutual information gives \[I(X;\, Y_{1:T}) \;=\; \sum_{t=1}^{T} I(X;\, Y_t \mid Y_{1:t-1}). \label{eq:chain-rule}\tag{3}\] To bound each conditional term, [26] proposes to maintain a posterior distribution \(P_{S \mid Y_{1:t-1}=y_{1:t-1}}\) over the secret space, which represents the strongest adversary’s optimal belief given the observed transcript so far. At step \(t\), the mechanism calibrates its randomness to ensure \(I(X; Y_t \mid Y_{1:t-1} = y_{1:t-1}) \leq \beta_t\) for every realization \(y_{1:t-1}\). This framework reduces privacy accounting to per-step MI control, which we exploit in § 3.

Instantiation and MIA guarantees. Following standard PAC Privacy instantiation [26], [27], [29], [30], given a universe of \(N\) training examples \(U=\{x_1,\ldots,x_N\}\), we construct \(M\) subsets \(S_1,\ldots,S_M\subset U\) such that each example in \(U\) appears in exactly \(M/2\) subsets. We then instantiate \(\mathcal{D}=\mathrm{Unif}\{S_1,\ldots,S_M\}\); equivalently, drawing a secret dataset from \(\mathcal{D}\) amounts to sampling a secret index \(j^*\sim\mathrm{Unif}[M]\) and setting the secret training set \(S^* = S_{j^*}\). This finite-support structure makes adaptive composition tractable: the posterior \(P(S^*=S_m \mid Y_{1:t-1}=y_{1:t-1})\) is a discrete distribution over \([M]\), maintained exactly in \(O(M)\) operations per step via Bayesian updates.

For membership inference attacks (MIA) targeting a record \(x_i\in U\), the adversary guesses whether \(x_i \in S^*\). Since each \(x_i\) appears in exactly \(M/2\) subsets and \(j^*\) is uniform, the prior MIA success rate is \(1-\delta_0 = 1/2\). Therefore, enforcing a total MI budget \(B\), numerically solving Eq. 2 yields a concrete upper bound on the posterior MIA success rate. For example, \(B=1/4\) nats limits MIA success to \(\approx 84\%\) and lowering \(B\) to \(1/128\) tightens the guarantee to around \(56\%\).

For comparison, \((\varepsilon,\delta)\)-differential privacy bounds MIA success at \(e^\varepsilon/(1+e^\varepsilon) + \delta\) [27]; concretely, \((1,10^{-5})\)-DP limits MIA success to \(73.11\%\), while \((0.1,10^{-5})\)-DP reduces it to \(52.50\%\). PAC Privacy and DP have fundamentally different semantics, and we do not claim equivalence between them. Throughout this paper we report PACZero’s privacy parameter in its native units (MI in nats), and where useful for context we annotate “the comparable DP \(\varepsilon\) that yields the same MIA upper bound at the prior \(1/2\)”. This DP-\(\varepsilon\) annotation is a numerical reference point only and does not transfer the DP guarantee to PACZero.

Finally, the bound 2 is attack-agnostic: for any attack criterion \(\rho\), one can evaluate the informed adversary’s prior success rate \(1-\delta_0\) under the subset construction, and derive concrete posterior success bounds by inverting the binary KL inequality at the enforced MI budget.

3 The PACZero Mechanism↩︎

The per-step procedure of PACZero is summarized in Algorithm 2. At step \(t\) we compute per-sample ZO scalars \(\hat{g}_i\) as in 1 , aggregate to subset means \(\bar g_m\) over the \(M\) subsets of §2, sign-quantize to \(s_m\!\in\!\{-1,+1\}\), and release a single bit \(Y_t\) that updates parameters via \(\theta_{t+1} = \theta_t - \eta_t Y_t z_t\). The release is deterministic on the unanimity branch (when the posterior \(p_t\!\in\!\Delta^{M-1}\) over the secret index \(j^*\) places all mass on subsets that agree on \(s_m\)) and noised on the disagreement branch otherwise. The mechanism instantiates as one of two variants (PACZero-MI, PACZero-ZPL) depending on how the disagreement branch is handled; §3.1–§3.2 treat each in turn.

3.1 PACZero-MI: adaptive MI allocation↩︎

3.1.0.1 Per-step MI of the noised binary release.

The building block of PACZero-MI is to privatize a binary-valued release given constrained mutual information. Let \(\xi\!\in\!\{-1,+1\}\) with \(P(\xi\!=\!+1) = q^+\) and \(\tilde{Y} = \xi + N\), \(N\!\sim\!\mathcal{N}(0,\sigma^2)\). The mutual information of \(\xi\) given \(\tilde{Y}\) is \[I(\xi;\tilde{Y}) \;=\; \sum_{s\in\{-1,+1\}} P(\xi{=}s) \int \phi_\sigma(y - s)\, \log \frac{\phi_\sigma(y-s)}{q^+\phi_\sigma(y-1) + (1-q^+)\phi_\sigma(y+1)} \,dy, \label{eq:binary-mi}\tag{4}\] where \(\phi_\sigma\) is the Gaussian density with standard deviation \(\sigma\). Given \((q_t^+, \beta_t)\), \(\sigma_t\) solving \(I(\xi;\tilde{Y})=\beta_t\) is recovered by quadrature plus bisection. We denote this process as \(\mathrm{BinaryMI}^{-1}(q_t^+,\beta_t)\) (Appendix 10). This evaluates the MI exactly for the binary-Gaussian channel, replacing the variance-based Gaussian upper bound used by canonical PAC-Privacy implementations [25]–[27].

Given the posterior \(p_t\) over the \(M\) candidate subsets and the \(M\) sign-quantized bits \(s_1,\ldots,s_M\), we calibrate noise via \(\mathrm{BinaryMI}^{-1}\) to control the per-step MI. Here, two cases arise:

• Unanimity: when all \(s_m\) in the support of \(p_t\) agree, the secret subset’s bit \(s_{j^*}\) is identical to every other candidate’s bit, so releasing it reveals nothing about which candidate is active—the MI is zero even without noise (\(\sigma_t\!=\!0\)). In this branch, the posterior need not be updated and the consumed budget is \(\beta_t^{\mathrm{used}}\!=\!0\). This is a direct consequence of PAC Privacy’s stability-based calibration: when the output is stable across all plausible secrets, no noise is required.

• Disagreement: when the support of \(p_t\) contains both signs (i.e., \(q^+\in (0,1)\)), we set \(\sigma_t = \mathrm{BinaryMI}^{-1}(q_t^+, \beta_t)\) and release the noised signal \(\tilde{Y}_t = s_{j^*} + \mathcal{N}(0,\sigma_t^2)\). The posterior is then updated via the likelihood of \(\tilde{Y}_t\). To avoid gradient explosion caused by the noise, we post-process the noisy release to \(Y_t = \mathrm{sign}(\tilde{Y}_t)\!\in\!\{-1,+1\}\). The MI between the secret index and the pre-quantized noisy release is exactly \(\beta_t^\mathrm{used}= \beta_t\) by construction; the MI between the secret index and the post-processed bit \(Y_t\) can only be smaller by the data-processing inequality.

This is formalized in the following lemma (proof in Appendix 9.1).

Adaptive MI allocation. Given a total MI budget \(\mathrm{MI}_{\mathrm{tot}}\) and total step count \(T\), a uniform allocation sets \(\beta_t = \mathrm{MI}_{\mathrm{tot}}/T\) [26]. This is suboptimal: when the unanimity branch fires at step \(t\), the per-step budget \(\beta_t\) is unspent, and uniform allocation cannot recover it later, resulting in a loose overall mutual information bound. PACZero-MI uses the adaptive allocation \[\beta_t \;=\; \max\!\left(0,\, \mathrm{MI}_{\mathrm{tot}} - \mathrm{MI}_{\mathrm{used}}^{(t-1)}\right) \,\big/\, (T - t + 1), \label{eq:adaptive-beta}\tag{6}\] where \(\mathrm{MI}_{\mathrm{used}}^{(t-1)} = \sum_{\tau<t} \beta_\tau^{\mathrm{used}}\) is the cumulative MI consumed strictly before step \(t\), with an additional feasibility cap \(\beta_t \!\leftarrow\! \min(\beta_t,\, 0.999\,h(q_t^+))\) to ensure the bisection targets a realizable MI (Appendix 10). The denominator ensures the remaining budget is divided over the remaining steps, so the run consumes at most \(\mathrm{MI}_{\mathrm{tot}}\) over its entirety. Crucially, \(\beta_t\) depends only on the public transcript (\(\mathrm{MI}_{\mathrm{used}}^{(t-1)}\) is a function of past \(\tilde{Y}_{<t}\) and \(\sigma_{<t}\), not \(S^*\)) and the public step index. By the adaptive composition framework of [26] applied to the chain rule 3 together with the per-step bound from §3.1.0.1, a complete PACZero-MI run satisfies \(I(S^*; Y_{1:T}) \le \sum_{t=1}^T \beta_t^{\mathrm{used}} \le \mathrm{MI}_{\mathrm{tot}}\).

Privacy guarantee. With the tight per-step accounting of conditional mutual information (cf. Lemma 1) and adaptive budget allocation, the end-to-end fine-tuning algorithm that iteratively applies Algorithm 2 satisfies the following mutual information guarantee (full proof is deferred to Appendix 9.2.):

3.2 PACZero-ZPL: zero privacy loss variant↩︎

We motivate PACZero-ZPL as the limit of Algorithm 2 as the per-step budget \(\beta_t\!\to\!0\). On unanimity nothing changes: the per-step MI is identically zero regardless of \(\beta_t\) (Lemma 1), and the unnoised sign \(s_{j^*}\) is released deterministically. On disagreement the calibration \(\sigma_t = \mathrm{BinaryMI}^{-1}(q_t^+, \beta_t)\) sends \(\sigma_t\!\to\!\infty\), and \(\mathrm{sign}(s_{j^*} + \mathcal{N}(0,\sigma_t^2))\) converges in distribution to \(\mathcal{U}(\{-1,+1\})\), independent of \(s_{j^*}\). PACZero-ZPL formalizes this limit by bypassing the calibration entirely and releasing the coin flip directly, \[Y_t \;\sim\; \mathcal{U}(\{-1,+1\}),\quad\text{independent of S^* and z_t,} \label{eq:zpl-coin}\tag{8}\] on every disagreement step. The posterior is left unchanged (\(p_{t+1}\!=\!p_t\)), since \(Y_t\) carries no signal about \(j^*\), and remains at the uniform prior \(p_0\) throughout. On unanimity the release is \(s_{j^*}\) as in PACZero-MI. The result is a one-line modification of Algorithm 2 (lines 11–13). Convergence depends on the unanimity rate not collapsing during training, an empirical property verified in § 4.

4 Experiments↩︎

We evaluate PACZero on the two LLM tasks reported by the strongest DP zeroth-order baseline, DP-AggZO [17]: SST-2 (binary sentiment classification) [31] and SQuAD (extractive QA, F1) [32]. Each task is run on OPT-1.3B and OPT-6.7B [33] across two parameter tracks, LoRA \(r{=}8\) [34] and full-parameter fine-tuning (FT). The section is structured around three claims: (i) PACZero-MI matches DP-ZO at the matched-MIA comparison points used by prior DP-ZO work, while PACZero-ZPL reaches \(I(S^*; Y_{1:T})\!=\!0\), with the strongest PACZero-ZPL cell on SST-2 6.7B FT at \(90.52\!\pm\!1.43\%\) (Table [tab:headline], §4.2); (ii) PACZero-MI test accuracy is flat across four decades of MI budget on the same recipe (Table 1); (iii) prior DP-ZO methods collapse below \(\varepsilon\!=\!1\) at a per-step DP noise floor that no step budget can overcome (Table 2).

4.1 Setup↩︎

Protocol. We follow the DP-AggZO protocol [17], using \(|N_{\mathrm{train}}|\!=\!1000\), \(|N_{\mathrm{dev}}|\!=\!500\), \(|N_{\mathrm{eval}}|\!=\!1000\), \(\delta\!=\!10^{-5}\), and weight decay \(\lambda\!=\!0\) [1]. Each run trains for up to \(T\) steps with dev-best checkpoint selection treated as post-processing under 3 , so reported MI accounts for the full \(T\)-step run. Headline budgets are total \(\mathrm{MI}\!\in\!\{0.33, 0.68\}\) nats, matching the MIA upper bound of DP \(\varepsilon\!\in\!\{2,6\}\) at \(\delta\!=\!10^{-5}\) under the conversion of [17], [26] (numerical reference only; §2). For PACZero-MI we use \(M\!=\!128\) subsets; PACZero-ZPL uses \(M\!=\!126\) throughout for implementation reasons (Appendix 19). Both tracks share the standard LoRA-\(r{=}8\)/\(\alpha{=}16\) configuration [15], [34]. FT step budgets follow DP-AggZO (\(T\!=\!1000\)); LoRA step budgets are dev-tuned per cell (\(T\!=\!2000\) for PACZero-MI, \(T\!=\!1000\) for PACZero-ZPL). All learning rates and clip values are dev-tuned per cell; LR sweeps (Appendix 14), clip ablations (Appendix 15), the LoRA rank ablation (Appendix 16), the canonical PACZero-ZPL configuration with \(T\)-ladders (Appendix 19), and per-seed values (Appendix 12) are in the appendix.

Baselines. Our primary comparators are DPZero [15] and DP-AggZO [17] at matched perturbation count \(K\!=\!1\), both directly aligned with PACZero’s \(K\!=\!1\) design. As a reference for parallel directional probing (\(K\!>\!1\)), we additionally report DP-AggZO at \(K\!=\!16\). Headline DP cells in Table [tab:headline] are taken from [17] Table 2 (parity check in Appendix 11); since [17] excludes DP-LoRA on OPT, all cross-method DP comparisons live in the FT block, and the LoRA block of Table [tab:headline] functions as a within-method ablation (track-robustness for PACZero-MI, track-robustness for PACZero-ZPL at \(I\!=\!0\)). Bao’s non-private MeZO cells are run to convergence over substantially more steps [17], we use the results reported as-is. We additionally reproduce the high-privacy regime in-house at \(K\!\in\!\{1, 64\}\) and \(\varepsilon\!\in\!\{0.2, 0.3, 0.5, 1.0, 2.0\}\) (Table 2). A \(K\!\in\!\{4, 16\}\) ablation (Appendix 20) confirms the \(K\!=\!1\) design is not a special case missing scale, and a non-private mechanism decomposition isolating sign quantization is in Appendix 18.

4.2 Headline results↩︎

4pt

Reading Table [tab:headline]. At the matched-MIA comparators \(\mathrm{MI}\!\in\!\{0.33, 0.68\}\), PACZero-MI on SST-2 1.3B FT exceeds the matched DPZero \(K\!=\!1\) comparator by \(+2.9\)pp at \(\mathrm{MI}\!=\!0.33\) and \(+1.0\)pp at \(\mathrm{MI}\!=\!0.68\), and at 6.7B FT sits within \(1.6\)pp of the non-private MeZO baseline of [17] (\(93.8\) FT). PACZero-ZPL satisfies \(I(S^*; Y_{1:T})\!=\!0\) for every \(T\) by construction (§3.2), and on SST-2 FT the PACZero-ZPL cells in Table [tab:headline] sit within \(2.1\)–\(3.3\)pp of the non-private FT ceilings (\(91.1\) at 1.3B, \(93.8\) at 6.7B). On SQuAD, a generative QA stress test for the one-bit release, PACZero-MI trails DPZero \(K\!=\!1\) by \(6\)–\(13\) F1 across the FT cells. These cells are not optimization-matched, since DPZero \(K\!=\!1\) runs at \(T\!=\!20{,}000\) [15] while ours follow the DP-AggZO protocol at \(T\!=\!1000\) [17], a \(20\times\) step gap that with the coarser binary release likely explains part of the deficit. PACZero-ZPL still obtains nontrivial F1 at zero MI. FT is the headline track in Table [tab:headline]; the LoRA-SQuAD cells trail their FT counterparts across both scales. This tracks the unanimity rate driving PACZero-ZPL, which falls from \(\sim\)​34–45% on SST-2 to \(\sim\)​25–33% on SQuAD-FT and \(\sim\)​8–11% on SQuAD-LoRA (Appendix 13).

Tight-MI plateau. Holding the lr/clip/\(T\) recipe fixed and sweeping the per-step MI budget across four decades on SST-2 FT OPT-1.3B (Table 1), PACZero-MI test accuracy spans \(86.47\)–\(89.56\%\) (\(3.09\)pp) and dev accuracy spans \(85.80\)–\(88.80\%\) (\(3.00\)pp). The unanimity fraction \(f\!=\!n_{\mathrm{free}}/T\) is stable at \(37.6\)–\(40.7\%\) across the entire range, indicating that the bulk of optimization signal originates from zero-cost unanimity steps regardless of how tight the MI budget is set on the disagreement branch (For full unanimity range, see Appendix 13). The flat shape across four decades implies the PACZero-MI headline cells are robust to MI budget choice within \(10^{-4}\)–\(0.68\) nats, a robustness DP-ZO does not exhibit across its \(\varepsilon\) sweep (Table 2); the matching LoRA plateau is in Appendix 17.

2.5pt

The DP-ZO cliff: a privacy regime DP-ZO cannot enter. DPZero \(K\!=\!1\) FT remains at chance accuracy (\(\sim\!50\%\)) for every \(\varepsilon<\!1\) at \(T\!=\!20{,}000\) (Table 2). DP-AggZO \(K\!=\!64\) FT exhibits a sharp cliff in \((0.3, 1.0]\), first lifting above chance only at \(\varepsilon\!=\!0.5\) (\(65.94\%\)) and reaching usable accuracy at \(\varepsilon\!=\!1.0\) (\(86.24\%\)). Doubling the step budget to \(T\!=\!2000\) at \(\varepsilon\!=\!0.2\) leaves DP-AggZO at chance (\(48.74\%\)), confirming the cliff is set by per-step DP noise rather than undertraining. On the same FT track, PACZero-ZPL at \(I\!=\!0\) matches the strongest in-house DP cell (\(88.42\%\)) by \(+0.57\)pp (Table [tab:headline]). PACZero thus produces the only usable cells in a privacy regime where every published DP-ZO accountant [15]–[17] assigns infinite per-step noise.

6pt

5 Related Works↩︎

DP-SGD [7] clips per-record gradients to a sensitivity budget and adds Gaussian noise tracked by Rényi accounting [8], [9]. A mature line adapts this template to LLM fine-tuning, with refinements to clipping and adapter integration [10], [11], [35]–[39]. The zeroth-order branch privatizes the per-step finite-difference scalar instead of the full gradient and removes explicit dimension dependence from the per-step noise. [15] (DPZero) prove nearly dimension-independent rates with a Gaussian scalar mechanism, the DP-ZO [16] adopts the same scalar-privatization insight empirically with Gaussian and Laplace mechanisms (the latter enabling pure \(\varepsilon\)-DP), and [17] (DP-AggZO) aggregate \(K\) finite-difference coefficients into a vector before clipping, reporting the strongest results among the DP-ZO baselines on OPT-1.3B/6.7B. All of these inherit a per-record (or per-aggregate) sensitivity bound, additive noise scaled to it, and sequential composition over \(T\) steps, which is the source of the well-documented utility collapse as \(\varepsilon\!\to\!0\) [18], [20], [21]. PACZero retires per-record sensitivity: subset-aggregated sign quantization makes the per-step release a single bit, calibrated by the exact mutual information of the noised binary release rather than by a clipping norm, with per-sample clipping retained only as a utility stabilizer. The PACZero-ZPL variant releases data-independent coins on disagreement steps to reach \(I\!=\!0\), a regime whose matched-MIA DP equivalent (\(\varepsilon\!=\!0\)) would require unbounded per-step noise under any Gaussian or Laplace DP-ZO accountant. The single-bit release shares structure with signSGD [40], [41] but is calibrated for privacy rather than compression. Detailed discussion of DP and its relaxations is deferred to § 6.

6 Discussion and Limitations↩︎

Privacy semantics: PAC Privacy vs DP. PAC Privacy and DP answer different questions and we do not claim equivalence (§2). DP bounds the worst-case log-likelihood ratio between output distributions on neighboring datasets; PAC Privacy bounds the mutual information between the output and the secret drawn from a specified distribution. The core practical distinction is what noise is calibrated to: DP calibrates to worst-case per-record sensitivity, while PAC Privacy calibrates to the empirical stability of the output under the data-generating distribution \(\mathcal{D}\). After sign quantization and subset aggregation, this stability can be much smaller than sensitivity—and on unanimity steps it is exactly zero, allowing directed updates at no privacy cost.

Our construction \(\mathcal{D}=\mathrm{Unif}\{S_1,\ldots,S_M\}\) with balanced membership directly models the standard MIA game [2], [42]: a universe of candidates is subsampled, and the adversary who knows all candidates determines whether a target record was included. In particular, PACZero-ZPL’s \(I(S^*;Y_{1:T})\!=\!0\) is a guarantee under our explicit candidate-set distribution \(\mathcal{D}\!=\!\mathrm{Unif}\{S_1,\ldots,S_M\}\), not a worst-case-adjacency DP guarantee, and the matched-MIA-prior DP-\(\varepsilon\) annotations in our tables (e.g., “DP \(\varepsilon\!=\!2\)”) are numerical references calibrated to the same posterior MIA bound under \(\mathcal{D}\), not transfers of DP semantics to PACZero. We also note that while we have focused on resistance against balanced MIA games, PAC Privacy provides concrete guarantees for any inference attack on the secret dataset, such as reconstruction, group MIA, and positive identification [29].

Several DP relaxations such as smooth sensitivity [43], per-instance DP [44], individual privacy accounting [45], [46], Pufferfish [47], and Bayesian DP [48] reduce noise below the global-sensitivity worst case also by incorporating instance-specific or distributional information. However, all remain calibrated to some form of per-record divergence (local sensitivity, per-instance \(\varepsilon_i\), or pairwise discriminative distributions), so they charge a positive cost whenever any single record’s inclusion changes the output distribution. PAC Privacy’s distributional MI formulation avoids this: it asks whether the output helps identify which subset is active, not whether it distinguishes specific record pairs. When the answer is collectively “no” (unanimity), the cost is zero even if individual records’ pairwise sensitivities are positive. This structural difference is what enables useful signal at \(\mathrm{MI}\!=\!0\).

Future work. The most immediate next step is a hybrid budget-then-ZPL schedule: spend the MI budget with PACZero-MI so the posterior \(p_t\) concentrates over \(S^*\), then switch to PACZero-ZPL and train to dev-best. Total MI stays capped because the zero-cost tail contributes no further conditional MI, and the inherited posterior should raise the unanimity rate above a from-scratch ZPL run. On the theory side, the unanimous-update-plus-random-jiggle dynamics call for a formal convergence analysis. The unanimity rate driving PACZero-ZPL and the agreement statistics governing PACZero-MI’s binary-channel MI are properties of the loss landscape rather than invariants of the mechanism, so whether our observed rates persist on architectures with sharper optimization geometry (Llama, Gemma, Mistral), at scales above OPT-6.7B, and on token-level generation tasks is an open question. Pre-training is a separate regime: the \(M\)-subset construction with \(|U|\!=\!1000\) fits fine-tuning corpora and would need rethinking at billion-token scale.

7 Conclusion↩︎

We introduced PACZero, a family of PAC-private zeroth-order mechanisms for fine-tuning large language models, built around a single primitive: sign-quantize subset-aggregated ZO updates and release one bit identifying the sign of the secret subset. The binary release admits exact mutual-information calibration as a one-dimensional Gaussian integral, replacing the variance-based upper bound used by canonical PAC algorithms. It also exposes posterior-weighted unanimity, which contributes zero conditional MI on \({\sim}34\)–\(45\%\) of SST-2 training steps. The same primitive instantiates two variants that span the privacy-utility frontier: PACZero-MI (budgeted MI) and PACZero-ZPL (\(I(S^*;Y_{1:T})\!=\!0\) via a uniform release on disagreement steps). On SST-2 OPT-1.3B full fine-tuning, PACZero-ZPL reaches \(\boldsymbol{88.99\!\pm\!0.91\%}\) at \(I\!=\!0\), within \(2.1\)pp of the non-private FT baseline of [17]. Quantizing before calibrating noise opens an MIA-resistance level, \(I(S^*;Y_{1:T})\!=\!0\) (matched-MIA DP \(\varepsilon\!=\!0\)), that no published DP-ZO accountant can reach at finite per-step noise; PACZero attains it while maintaining competitive utility.

Acknowledgments↩︎

The contribution of Marten van Dijk and Murat Bilgehan Ertan to this publication is part of the project CiCS of the research program Gravitation which is (partly) financed by the Dutch Research Council (NWO) under the grant 024.006.037. We acknowledge the use of the DAS-6 High-Performance Computing cluster at Vrije Universiteit Amsterdam for GPU-based experiments [49].

8 Appendix Organization↩︎

Appendix is organized as follows.

• Appendix 9 contains deferred full proofs of Lemma 1 (Appendix 9.1) and Theorem 1 (Appendix 9.2).

• Appendix 10 collects PACZero implementation details: the role of the stability clip, numerical evaluation of the binary-channel MI integral, the sign convention at zero, the unanimity tolerance, and the entropy-ceiling cap on \(\beta_t\).

• Appendix 11 reports the parity check between our in-house DP-AggZO implementation and the cells of [17] cited in Table [tab:headline].

• Appendix 12 gives per-seed test values for every multi-seed cell of Table [tab:headline].

• Appendix 13 reports the unanimity fraction \(f\!=\!n_{\mathrm{free}}/T\) for every cell of Table [tab:headline].

• Appendix 14 reports per-cell headline recipes (Table 6) and the PACZero-ZPL learning-rate sweep that selects them (Table 7).

• Appendix 15 reports the 6.7B LoRA clip-norm ablation.

• Appendix 16 reports the 6.7B LoRA rank ablation.

• Appendix 17 reports the LoRA analogue of the FT plateau in Table 1, spanning four decades of nominal MI.

• Appendix 18 decomposes the PACZero per-step release into deterministic surrogates and reports the random-sign negative control.

• Appendix 19 characterises PACZero-ZPL in the canonical \(M\!=\!128\) dev-tuned configuration and reports three \(T\)-ladder studies on post-convergence drift, including a 6.7B FT multi-seed extension.

• Appendix 20 extends the per-step release to \(K\!\in\!\{4,16\}\) following the DP-AggZO-style aggregation pattern at both 1.3B and 6.7B SST-2.

• Appendix 21 reports per-cell GPU-hours and infrastructure (Tables 19–20).

9 Deferred full proofs↩︎

9.1 Proof to Lemma 1↩︎

9.2 Proof to Theorem 1↩︎

10 PACZero methodology: implementation details↩︎

10.0.0.1 The role of the stability clip.

Per-sample clipping at magnitude \(c\) before subset aggregation is a utility stabiliser, not a privacy device. It bounds any single sample’s leverage on the subset mean, so a hard example cannot single-handedly flip a subset’s sign. Because the released quantity is a single bit and the binary-channel MI accounts for all leakage of \(s_{j^*}\), the magnitude of \(\hat{g}_i\) does not enter the privacy proof; clipping only affects the empirical sign distribution. § 4 confirms it is utility-relevant (\(\sim\!2\)pp on LoRA) but no calibration to a record-level sensitivity is required.

10.0.0.2 Numerical evaluation of the binary-channel MI integral.

We evaluate 4 via 60-node Gauss–Hermite quadrature, with log-sum-exp stabilisation to handle the \(\sigma\!\downarrow\!0\) regime cleanly. Given \((q^+, \beta_t)\), \(\sigma_t\) solving \(I(\xi;\tilde{Y})=\beta_t\) is recovered by bisection in \(\log\sigma\), since \(I(\xi;\tilde{Y})\) is monotonically decreasing in \(\sigma\) by data-processing.

10.0.0.3 Sign convention at zero.

In Algorithm 2 we adopt the convention \(\mathrm{sign}(0):=+1\); i.e.exact zeros in either \(\bar g_m\) or the noised disagreement-branch release \(\tilde{Y}_t\) are mapped to \(+1\). This preserves the binary support \(\{-1,+1\}\) assumed by Eq. 4 and Lemma 1. The event \(\bar g_m\!=\!0\) exactly is not observed in any of our runs.

10.0.0.4 Unanimity tolerance.

Algorithm 2 writes the unanimity test as \(q_t^+\!\in\!\{0,1\}\). Our implementation evaluates this with a small tolerance, \(q_t^+\!\le\!\tau\) or \(q_t^+\!\ge\!1\!-\!\tau\) with \(\tau\!=\!10^{-12}\), so floating-point roundoff in the posterior weights does not silently push a near-unanimity step onto the noised branch. Under this tolerance, the residual conditional MI on a tolerance-detected unanimity step is bounded above by the binary entropy at the tolerance, \(h(\tau)\!\approx\!\tau\log(1/\tau)\), which at \(\tau\!=\!10^{-12}\) is \(\sim\!2.8\!\times\!10^{-11}\) nats per step. Summed over the headline \(T\!=\!2000\) this is \(\le\!5.5\!\times\!10^{-8}\) nats, six orders of magnitude below the tightest reported budget; the chain-rule bound of Theorem 1 is preserved up to this tolerance term, which we treat as zero throughout.

10.0.0.5 Entropy-ceiling cap.

A single bit cannot leak more MI than the binary entropy of the secret distribution, \(h(q_t^+) = -q_t^+\log q_t^+ - (1-q_t^+)\log(1-q_t^+)\). When the posterior has concentrated, \(h(q_t^+)\) shrinks below \(\beta_t\) and the bisection in step 11 of Algorithm 2 would target an infeasible MI. Before invoking \(\mathrm{BinaryMI}^{-1}\) we therefore cap \(\beta_t \leftarrow \min(\beta_t, 0.999\!\cdot\! h(q_t^+))\). The cap is a function of \(p_t\) only (not \(S^*\)), so admissibility under adaptive composition is preserved.

11 DP-AggZO parity check↩︎

To verify that our in-house DP-AggZO implementation matches the published cells of [17], we reran a representative subset of the \(K\!=\!64\) FT cells at single seed and compared against [17] Table 2. The mean absolute deviation across the parity-check subset is \(|\Delta|\!=\!0.9\)pp, with the largest deviation \(2.4\)pp at \(\varepsilon\!=\!2\) (single-seed; the published cell is single-seed as well, so seed variance is the most likely source). This level of parity is consistent with the codebase being faithful to the published recipe; the residual gap is attributable to seed variance and minor accountant calibration rounding. This justifies treating the published FT \(K\!=\!1\) and \(K\!=\!16\) cells of [17] as direct comparators in Table [tab:headline].

12 Per-seed values for multi-seed cells↩︎

For each multi-seed cell reported in the headline Table [tab:headline], we give the per-seed test values used to compute the mean\(\,\pm\,\)std (sample standard deviation, \(n\!-\!1\) denominator).

12.0.0.1 SST-2 multi-seed cells (Table [tab:headline]).

Tables 3 and 4 report per-seed test accuracy % for the eight SST-2 multi-seed cells.

4pt

4pt

12.0.0.2 SQuAD multi-seed cell (Table [tab:headline]).

The 1.3B SQuAD FT PACZero-ZPL cell in Table [tab:headline] pools three seeds at the locked recipe lr\(\!=\!10^{-4}\), \(c\!=\!1000\), \(T\!=\!1000\), \(M\!=\!126\), polynomial scheduler, --pac_load_best_dev. Per-seed test F1: \(62.36\) (s0), \(63.53\) (s1), \(60.87\) (s2) \(\Rightarrow\) \(n\!=\!3\) mean \(\boldsymbol{62.25\!\pm\!1.33}\) (test F1), dev F1 \(57.73\!\pm\!1.87\). The remaining SQuAD cells in Table [tab:headline] are reported single-seed at the per-cell compute cost of one \(T\)-step trajectory at OPT-6.7B SQuAD scale.

12.0.0.3 Canonical-configuration PACZero-ZPL multi-seed values (Appendix 19).

6.7B LoRA canonical-configuration per-seed peak-rung test % (Table 14): \(90.14\) (seed \(0\), \(T\!=\!500\)), \(87.16\) (seed \(1\), \(T\!=\!500\)), \(86.35\) (seed \(2\), \(T\!=\!500\)); \(n\!=\!3\) mean \(\boldsymbol{87.88\!\pm\!2.00}\).

13 Unanimity rates per headline cell↩︎

Table 5 reports the unanimity fraction \(f\!=\!n_{\mathrm{free}}/T\) for every cell of Table [tab:headline], broken out by variant. SST-2 cells fire the unanimity branch on \(34\)–\(45\%\) of steps; SQuAD-FT drops to \(25\)–\(33\%\); SQuAD-LoRA collapses to \(8\)–\(11\%\). Within each (track, scale, task) cell the rate is broadly stable across PAC-MI at \(\mathrm{MI}\!=\!0.33\), PAC-MI at \(\mathrm{MI}\!=\!0.68\), and PACZero-ZPL, consistent with unanimity being a property of the loss landscape rather than the variant. The SQuAD-LoRA collapse tracks the largest F1 gaps in Table [tab:headline]: PACZero-ZPL draws all signal from unanimity steps, and that rate falls fastest exactly where the F1 gap is widest.

6pt

14 Per-cell headline recipes and learning-rate sweeps↩︎

14.0.0.1 Per-cell headline recipes.

Table 6 consolidates the dev-tuned learning-rate, clip, \(T\), and \(M\) values used for every cell of the headline Table [tab:headline]. PACZero-MI recipes are dev-best winners on a per-cell single-seed sweep; the PACZero-ZPL recipes are dev-tuned per (track, scale). Sweeps that produced the chosen lr values are reported below (this section, Table 7); clip sweeps in Appendix 15.

6pt

14.0.0.2 PACZero-ZPL learning-rate sweep.

Single-seed sweeps over the chosen LR ranges select the headline LRs used by the canonical PACZero-ZPL configuration (Appendix 19). The dev-winner selections are reported in Table 7. Note that the FT track has two dev-winner rows: at clip\(\!=\!1000\), lr\(\!=\!10^{-4}\) wins (\(88.76\%\) test, single seed); at no-clip, lr\(\!=\!10^{-4}\) wins (\(89.56\%\) test, single seed). The \(\mathrm{\small PACZero-ZPL}{}\) headline FT recipe in Table 6 uses clip\(\!=\!1000\) (matching the PACZero-MI FT recipe) because the \(M\!=\!126\) multi-seed extension was completed at clip\(\!=\!1000\); a corresponding \(M\!=\!126\) no-clip multi-seed pool was not run within the compute budget. The canonical \(M\!=\!128\) no-clip multi-seed pool (\(87.96\!\pm\!0.80\), Table 12) is reported as the dev-winner-recipe reference.

6pt

15 Clip ablations↩︎

15.0.0.1 6.7B LoRA clip-norm ablation.

Table 8 reports the clip ablation for SST-2 OPT-6.7B LoRA. The dev-winner is \(c\!=\!10\), which is the clip used by the headline 6.7B LoRA cells in Table [tab:headline]. Multi-seed entries are mean\(\,\pm\,\)std over \(n\!=\!3\) seeds; single-seed entries are seed 0. Unanimity values are sampled from one representative seed per cell. The clip-norm sweep at MI\(\!=\!0.68\) is reported only at the dev-winner \(c\!=\!10\). At \(c\!=\!25\) the MI\(\!=\!0.33\) and MI\(\!=\!0.68\) single-seed cells (seed 0) share the same public-PRG perturbation directions, both cells consume their full MI budget (per-trajectory \(\mathrm{cum\_mi}\!=\!0.3289\) and \(0.6800\) respectively, matching the nominal budget to numerical precision), and both select checkpoint \(600\) as dev-best with \(\mathrm{eval\_loss}\!=\!0.3035\) to four decimals—producing identical \(\{\mathrm{dev}, \mathrm{test}\}\!=\!\{88.20, 88.76\}\%\). This is consistent with the tight-MI plateau (Table 1, Appendix 17): in the MI=\(0.33\)–\(0.68\) range, the same-cell test accuracy is approximately invariant in the MI budget. The MI\(\!=\!0.68\) cell at \(c\!=\!25\) is therefore not an independent MI\(\!=\!0.68\) datapoint, and we omit it from the table.

6pt

16 LoRA rank ablation↩︎

The dev-winner rank for 6.7B LoRA is \(r\!=\!8\), which is the rank used by the headline cells. The ablation runs at clip \(c\!=\!25\) (the matched convention at the time the rank cells were produced); the headline 6.7B LoRA cells in Table [tab:headline] use clip \(c\!=\!10\) at \(r\!=\!8\).

8pt

17 LoRA plateau on SST-2 OPT-1.3B↩︎

The LoRA analogue of the FT plateau in Table 1 is reported in Table 10. Test accuracy spans \(89.45\)–\(90.37\%\) across \(4\) decades of nominal MI (\(0.92\)pp), and dev accuracy spans \(89.60\)–\(90.60\%\) (\(1.00\)pp). Per-step \(\mathrm{cum\_mi}\) matches nominal MI to numerical precision across all \(15\) cells. The unanimity fraction \(f\!=\!n_{\mathrm{free}}/T\) ranges from \(40\%\) at the tightest budget to \(47\%\) at \(\mathrm{MI}\!\approx\!0.2\)–\(0.5\), consistent with the binary-release argument: tighter MI raises \(\sigma_t\), the posterior concentrates more slowly, and fewer steps reach the unanimity branch. The headline observation is the flat shape across \(4\) decades. Compared to the FT plateau in the main paper, the LoRA plateau spans a tighter test-accuracy range (\(0.92\)pp vs \(3.09\)pp) at the cost of a higher step budget (\(T\!=\!2000\) vs \(T\!=\!1000\)).

2.5pt

18 Mechanism decomposition↩︎

Table 11 decomposes the PACZero per-step release into deterministic surrogates (run with --no_privacy). Each row replaces the full release with one surrogate, and random_sign is the negative control (uncorrelated \(\pm 1\) at every step). Quantized variants are \(T\)-sensitive. At \(T\!=\!1000\) they lag, and at \(T\!=\!2000\) they recover. The negative control collapses to chance (\(53.10\%\)), confirming that the optimization signal is necessary. quant_full (sign of the full-batch mean) outperforms its raw counterpart by \(+2.3\)pp at \(T\!=\!2000\), locating the gain primarily in the sign-quantization step rather than in the subset-aggregation step.

4pt

19 Canonical PACZero-ZPL configuration and \(T\)-ladders↩︎

The PACZero-ZPL cells in the headline Table [tab:headline] use \(M\!=\!126\) throughout. This reflects an early-implementation choice for the PACZero-ZPL sweep that was held fixed once multi-seed pools were under way, to keep the cells internally comparable across the PACZero-ZPL row; the privacy and utility guarantees of §3.2 are not specific to a particular value of \(M\), and Theorem 2 gives \(I(S^*; Y_{1:T})\!=\!0\) for any \(M\!\geq\!2\). To verify that the headline cells are not unfairly weak choices for the PACZero-ZPL row, we additionally characterise PACZero-ZPL in the canonical dev-tuned configuration with \(M\!=\!128\) throughout, with FT no-clip as the dev-winner of Appendix 14. We report the canonical PACZero-ZPL cells in Table 12; these are the dev-tuned single-recipe cells used as a reference against the \(M\!=\!126\) headline cells. The corresponding \(M\!=\!128\) pool was statistically indistinguishable.

6pt

19.0.0.1 Post-convergence drift: \(T\)-ladder studies.

PACZero-ZPL releases a uniform coin flip on disagreement steps. Once the model has converged, additional coin-flip releases drive parameters off the dev-best minimum, producing post-convergence drift visible as \(T\) grows past the dev-best rung. We report three ladders for the canonical configuration: 1.3B FT (Table 13), 6.7B LoRA across three seeds (Table 14), and 6.7B FT single-seed (Table 15).

6pt

6pt

6pt

19.0.0.2 6.7B FT PACZero-ZPL multi-seed extension at \(T\!=\!1500\).

The single-seed ladder above documents the seed-0 trajectory; we additionally trained two new seeds (\(s_1, s_2\)) at the same canonical recipe (lr\(\!=\!10^{-4}\), no-clip, \(T\!=\!1500\)) to extend the cell to multi-seed at the final-\(T\) rung (Table 16). Both new seeds’ final-\(T\) checkpoints coincide with their per-trajectory dev-best (eval-loss decreases monotonically across the ladder for both seeds), so the final-\(T\) test accuracy is the per-seed dev-best test accuracy by construction. Pooling at the per-seed dev-best rung with the seed-0 trajectory’s dev-best rung (\(T\!=\!1000\), test \(89.33\%\)), the \(n\!=\!3\) test accuracy is \(90.14\!\pm\!0.76\%\). Restricting to the new \(T\!=\!1500\) pair only (\(s_1, s_2\)) gives \(90.54\!\pm\!0.41\%\) at \(n\!=\!2\). The 6.7B LoRA canonical-configuration \(n\!=\!3\) pool sits at \(87.88\!\pm\!2.00\%\) (Table 14), so the FT track exceeds the LoRA track by \(+2.26\)pp at \(n\!=\!3\) on the canonical PACZero-ZPL configuration at this scale.

6pt

20 K-aggregation ablation (DP-AggZO-style)↩︎

The DP-AggZO mechanism of [17] aggregates \(K\) independent zeroth-order direction estimates per training step before the privacy release; the cells we cite from [17] Table 2 in our Table [tab:headline] are at \(K\!=\!16\), and we reproduce \(K\!=\!64\) cells in-house in the high-privacy regime (Table 2). PACZero as defined in §3 releases a single per-step bit (\(K\!=\!1\)). To verify that this design choice does not artificially understate PACZero’s achievable utility, we extend the per-step release to \(K\!\in\!\{4, 16\}\) following the DP-AggZO-style aggregation pattern: \(K\) independent ZO directions are sampled per step, each consumes \(\mathrm{per\_step\_mi}/K\) nats, the released \(K\) bits \(Y_{t,1},\ldots,Y_{t,K}\in\{-1,+1\}\) are averaged, and the parameter update is \(\theta_{t+1}\!=\!\theta_t - \eta_t (1/K)\!\sum_{k} Y_{t,k}\!\cdot\!z_k\). The PAC-MI chain-rule bound is preserved per step (\(\sum_k \beta_{\text{per},t,k}\!=\!\mathrm{per\_step\_mi}\)), and the \(K\!=\!1\) reduction is bit-exact equivalent to the original PACZero trainer (verified empirically).

Table 17 reports the \(K\)-ablation on SST-2 OPT-1.3B at \(\mathrm{MI}\!=\!0.33\), single seed. Both tracks show \(K\)-aggregation producing deltas that are well within seed noise of the \(K\!=\!1\) baseline. The ft track has \(\Delta\!\in\![-1.15, -0.80]\)pp at \(K\!\in\!\{4,16\}\) versus the \(K\!=\!1\) baseline, while the LoRA track has \(\Delta\!\in\![-1.55, +0.97]\)pp (\(-1.55\)pp at \(K\!=\!4\) and \(+0.97\)pp at \(K\!=\!16\), both within the \(K\!=\!1\) baseline’s \(\pm 1.53\)pp std at \(n\!=\!4\)). The headline interpretation is that PAC-MI’s per-step release is robust across \(K\), and the \(K\!=\!1\) design used throughout the paper is not a special case that misses scale. The small per-track magnitude of the deltas suggests that, at this MI budget, the per-release noise penalty from splitting the budget across \(K\) releases is approximately balanced by the \(K\)-averaging variance reduction on the parameter update; the sign of the residual is task- and recipe-dependent.

6pt

20.0.0.1 Scale extension to OPT-6.7B SST-2.

To verify that the \(K\)-agnostic finding at 1.3B is not specific to the smaller scale, we extended the same ablation to OPT-6.7B SST-2 at \(\mathrm{MI}\!=\!0.33\), single seed (Table 18). The ft track is \(K\)-agnostic at 6.7B as well: \(K\!=\!4\) reaches \(92.66\%\) and \(K\!=\!16\) reaches \(92.89\%\) versus the \(K\!=\!1\) multi-seed baseline of \(92.20\!\pm\!0.45\%\) (\(n\!=\!4\)); both deltas (\(+0.46\)pp, \(+0.69\)pp) sit inside the baseline’s \(\pm 0.45\)pp std band. The LoRA track shows a different pattern: \(K\!=\!4\) and \(K\!=\!16\) both reach \(93.92\%\) versus the \(K\!=\!1\) multi-seed baseline of \(90.21\!\pm\!0.43\%\) (\(n\!=\!3\)), a saturated \(+3.71\)pp delta multiple standard deviations above the baseline. The two LoRA cells are distinct trajectories with different dev-best checkpoints (dev \(91.0\%\) at \(K\!=\!4\) vs \(90.0\%\) at \(K\!=\!16\)) that coincidentally land on the same test count (\(819/872\!=\!93.92\%\)). Two readings are consistent with this single-seed observation: (i) the LoRA-rank-\(8\) subspace at 6.7B is small relative to the full FT space, so the per-release noise penalty from splitting the per-step MI budget across \(K\) releases is partially absorbed by the \(K\)-averaging variance reduction on a low-dimensional update direction, producing a net positive delta that saturates by \(K\!=\!4\); or (ii) the cell is sampling the high tail of the \(K\!=\!1\) seed distribution, with the saturation across \(K\!\in\!\{4,16\}\) inheriting the same draw. We do not separate these readings at single-seed; the more robust statement is that scaling to 6.7B does not invert the \(K\)-agnostic \(\Delta\)-direction on the ft track and produces a saturating positive \(\Delta\) on the LoRA track, neither pattern undermining the \(K\!=\!1\) design used throughout the paper.

6pt

21 Per-cell compute and infrastructure↩︎

21.0.0.1 Hardware.

All experiments run as single-GPU jobs (one process per GPU). OPT-1.3B cells use a mix of NVIDIA A4000 (16 GB), A5000 (24 GB), A6000 (48 GB), A100 (40 GB), and H100 (80 GB); OPT-6.7B cells use exclusively A100 (40 GB) and H100 (80 GB) for memory reasons.

21.0.0.2 Totals.

Summing across the rows of Tables 19–20 (every cell appearing in this paper’s main and appendix tables), the paper-cited compute is approximately \(1{,}500\) GPU-hours, dominated by the OPT-6.7B SQuAD cells of Table [tab:headline] (\(\sim\!930\) h). Including exploratory and discarded experiments not reported in this paper (additional learning-rate and clip sweeps, smoke tests, retries from preempted runs, mechanism variants outside Table 11, and undertrained runs at canonical learning rates that we then re-tuned), the project consumed approximately \(3{,}500\) GPU-hours.

6pt

6pt

References↩︎

---

1. Affiliated with Vrije Universiteit Amsterdam.↩︎

2. Code: https://github.com/bilgehanertan/paczero/↩︎