Note. The methods described in this paper are the subject of a pending patent application. This manuscript is intended for public disclosure after the filing date.

1 Introduction↩︎

Extreme low-bit quantization—in particular binary (\(1\)-bit) and sub-2-bit representations—is the most aggressive form of LLM weight compression: a 1.125-bit group-binary representation retains only \(7.0\%\) of the FP16 weight footprint. The dominant PTQ paradigm, exemplified by GPTQ [1] and its successors, quantizes one layer (or one block [2]) at a time by minimizing a local output-reconstruction error. This layer-local view has a well-known blind spot: the error committed by layer \(l\) perturbs the input of layer \(l+1\), and these perturbations compound with depth. Recent work has begun to address error accumulation—QEP [3] propagates quantization errors into the calibration of subsequent layers, YAQA [4] sketches network-level Hessians to guide per-layer rounding, and TurboBoA [5] and ResComp [6] refine the compensation rules of the GPTQ family. All of these methods, however, are sequential-commit procedures: each layer’s codes are finalized once, in order, and information available only after later layers are quantized never flows back to earlier decisions.

This paper develops and empirically validates a different design point: a joint objective over the discrete variables of all layers, organized around an explicit state variable—the accumulated error—that is updated by a linear recursion and can be driven toward cancellation by the optimizer. Concretely, our contributions are:

1. A recursive accumulated-error objective (Section 3.2). We define a propagation operator \(A_l\) from the layer’s input differential, a local quantization residual \(q_l\) evaluated at teacher features, and the recursion \(e_{l+1}=A_l e_l+q_l\). We prove an exactness lemma: with the finite-difference instantiation of \(A_l\) on the quantized network, the recursion equals the true feature deviation for arbitrary nonlinear layers, not merely to first order (Lemma 1). This licenses an efficient implementation that never forms a Jacobian.

2. Finite-sample statistics matching under relative normalization (Section 3.3). We match means, random-projected covariances, and centered empirical kernels between teacher and quantized features, normalized by the teacher’s own statistics so that layers of different scales contribute comparably, and we monitor the same quantity (\(E_{\mathrm{GE}}\)) as an equivalence metric at evaluation time.

3. A mirror-descent latent parameterization for binary codes (Section 3.4). Binary codes are produced by \(u=\tanh(\beta z)\) with an annealed \(\beta\) and a hard-forward finishing phase; we show this is exactly mirror descent under a binary-entropy mirror map and give the initialization that reproduces the teacher’s signs and magnitudes.

4. Controlled ablations with pre-registered seeds (Section 4). On a 1.5B-parameter model we isolate each mechanism at 1.125 bits and at 4.125 bits, evaluate in-domain and out-of-domain, and report all results, including the negative finding that statistics matching is not a quality driver.

2 Related Work↩︎

2.0.0.1 Layer-wise PTQ and error compensation.

GPTQ [1] minimizes per-layer reconstruction error with second-order column updates; BRECQ [2] extends the unit to blocks; AdaRound [7] learns up/down rounding within a layer. QEP [3] explicitly propagates quantization errors and compensates for accumulated errors during sequential layer-wise PTQ, and reports the largest gains in the extremely low-bit regime; TurboBoA [5] adds a correction for errors propagated from preceding layers without backpropagation; ResComp [6] identifies a compensation-aware error term inside the GPTQ update; YAQA [4] uses Kronecker-factored sketches of layer Hessians taken with respect to the network output. Our method differs structurally from this entire family: the codes of all layers remain free variables of a single objective, the accumulated error is an explicit state of a recursion rather than an adjustment to calibration inputs, and later-layer information flows back into earlier-layer updates.

2.0.0.2 Binary and near-binary LLMs.

BinaryConnect [8] and XNOR-Net [9] established latent full-precision weights with sign forward passes and scale factors (\(s=\mathrm{mean}|W|\)). BitNet [10] and BitNet b1.58 [11] train 1-bit/ternary LLMs from scratch; OneBit [12] converts pretrained models with a sign–value decomposition and distillation; BiLLM [13] performs salience-aware binary PTQ; FBI-LLM [14] trains fully binarized LLMs from scratch via autoregressive distillation while keeping the embedding and the causal head in full precision. We use the standard latent-weight machinery but embed it in the joint cross-layer objective above; our experiments quantize all transformer-block linear layers and keep embedding/LM head in FP16, matching the common setting of the conversion-based literature.

2.0.0.3 Feature and relation distillation for quantization.

FAQD [15] matches feature affinities on label-free data; ZeroQ [16] matches batch-norm statistics to synthesize calibration data; classical KD [17] matches logits. Our statistics term differs in its finite-sample, relative-normalized construction and, more importantly, in the role our experiments assign to it: it is an equivalence-preserving regularizer rather than a quality driver.

2.0.0.4 Discrete optimization for quantization.

ProxQuant [18] uses proximal operators toward quantized sets; Ajanthan et al. [19] formulate quantization as mirror descent and derive \(\tanh\)-type maps; QuIP# [20] and SpinQuant [21] precondition weights by Hadamard/learned rotations. Our contribution here is not the mirror map itself but its integration with a group-wise log-scale, an annealing schedule with sign-fixing criteria, and the joint cross-layer objective.

3 Method↩︎

3.1 Group-wise discrete parameterization↩︎

Let \(W\in\mathbb{R}^{m\times n}\) be a weight matrix. We partition each row into groups of \(G\) consecutive entries and represent \[\widehat{W} \;=\; s \odot q, \qquad q\in\mathcal{D}^{m\times n}, \label{eq:param}\tag{1}\] where \(\mathcal{D}\) is a finite value set and \(s>0\) is one scale per group, broadcast within the group. For binary quantization \(\mathcal{D}=\{-1,+1\}\) and the storage cost is \(1+16/G\) bits per weight (\(1.125\) bits at \(G=128\) with FP16 scales); for INT4, \(\mathcal{D}=\{-8,\dots,7\}\) and the cost is \(4.125\) bits. All derivations below are agnostic to \(\mathcal{D}\).

3.2 Cross-layer error dynamics↩︎

Write the full-precision (teacher) network as a composition of layer maps \(h_{l+1}=F_l(h_l)\) and the quantized network as \(\widehat{h}_{l+1}=\widehat{F}_l(\widehat{h}_l)\), with shared input \(h_0=\widehat{h}_0=x\). Define the deviation \(e_l:=\widehat{h}_l-h_l\). Two quantities localize how error is created and transported: \[\begin{align} q_l &\;:=\; \widehat{F}_l(h_l)-F_l(h_l) &&\text{(local quantization residual, at teacher features)}, \tag{2}\\ A_l &\;:=\; \frac{\partial F_l}{\partial h}\Big|_{h_l} &&\text{(propagation operator, input differential)}. \tag{3} \end{align}\] Expanding \(\widehat{h}_{l+1}-h_{l+1}=\widehat{F}_l(h_l+e_l)-F_l(h_l)\) to first order in \(e_l\) gives the recursion \[e_{l+1}\;=\;A_l\,e_l\;+\;q_l,\qquad e_0=0. \label{eq:recursion}\tag{4}\] Equation 4 exposes the design principle: the optimizer can choose later-layer residuals \(q_l\) that cancel the propagated term \(A_l e_l\), which is impossible if each \(q_l\) is minimized in isolation. The operator \(A_l\) never needs to be materialized: Jacobian–vector products, low-rank sketches, or the finite differences below all suffice.

In practice we do not run the recursion with the teacher Jacobian. The following lemma shows that a specific instantiation makes 4 exact and reduces it to a forward difference.

The corollary matters for implementation: the exact accumulated error of a \(1.5\)B-parameter model can be obtained with one extra forward pass, and the resulting loss \[L_E \;=\; \mathbb{E}_{x}\!\left[\frac{\left\lVert e_L(x) \right\rVert_2^2}{\left\lVert h_L(x) \right\rVert_2^2+\epsilon}\right] \label{eq:le}\tag{5}\] (relative normalization; identity readout) trains all layers’ codes and scales jointly through automatic differentiation. We verified Lemma 1 numerically on random linear networks to \(10^{-10}\) absolute tolerance (Section 4.5).

3.3 Finite-sample feature-statistics matching↩︎

Let \(H,\widehat H\in\mathbb{R}^{n\times d}\) collect \(n\) token features of a chosen layer from the teacher and the quantized network on the same inputs, with means \(\mu,\widehat\mu\). With a fixed random projection \(R\in\mathbb{R}^{d\times k}\) (\(k{=}64\), entries \(\mathcal{N}(0,1/k)\)), define centered projections \(P=(H-\mu)R\), \(\widehat P=(\widehat H-\widehat\mu)R\) and the centered empirical kernel \(K=\frac{1}{d}(H-\mu)(H-\mu)^{\!\top}\). The statistics loss on one layer is \[D \;=\; \underbrace{\frac{\left\lVert \widehat\mu-\mu \right\rVert_2^2}{\left\lVert \mu \right\rVert_2^2+\epsilon}}_{D_\mu} +\underbrace{\frac{\left\lVert \widehat C-C \right\rVert_F^2}{\left\lVert C \right\rVert_F^2+\epsilon}}_{D_C} +\underbrace{\frac{\left\lVert \widehat K-K \right\rVert_F^2}{\left\lVert K \right\rVert_F^2+\epsilon}}_{D_K}, \qquad C=\tfrac{1}{n-1}P^{\!\top}P, \label{eq:stats}\tag{6}\] and \(L_{\mathrm{GE}}\) averages \(D\) over a layer set \(S\) (every \(L/4\) blocks plus the final block). The relative normalization makes deep and shallow layers commensurable and keeps the loss dimensionless. The same quantity evaluated on held-out data, denoted \(E_{\mathrm{GE}}\), serves as a statistical-equivalence metric: \(E_{\mathrm{GE}}\!\approx\!0\) certifies that first- and second-order feature statistics and token-relation structure of the teacher survive quantization. We emphasize a scope statement: 6 is a finite-sample, functional criterion, not a claim of mathematical equivalence of the two networks. For completeness we also report centered linear CKA [22] in the code base; note that independent \(d\)-dimensional representations exhibit a positive CKA bias of order \(r/n\), which our unit tests reproduce (Appendix 9).

3.4 Latent binary codes as mirror descent↩︎

Each binary group is parameterized by latent variables \(z\) and a log-scale \(r=\log s\): \[\widehat{W}=s\,u,\qquad u=\tanh(\beta z), \label{eq:latent}\tag{7}\] with initialization chosen to reproduce the teacher exactly at \(\beta=\beta_0\): \[s_0=\mathrm{mean}\,|W_{\mathrm{group}}|,\qquad z_0=\frac{1}{\beta_0}\,\mathrm{artanh}\!\big(\mathrm{clip}(W/s_0,\pm\rho)\big), \quad \rho=0.95 . \label{eq:init}\tag{8}\]

3.4.0.1 Mirror-descent interpretation.

Let \(\psi_\beta(u)=\frac{1}{\beta}\left[\frac{1+u}{2}\log\frac{1+u}{2} +\frac{1-u}{2}\log\frac{1-u}{2}\right]\) be the (scaled) binary-entropy mirror map on \((-1,1)\). Then \(\nabla\psi_\beta(u)=\mathrm{artanh}(u)/\beta\) and its convex conjugate satisfies \(\nabla\psi_\beta^*(z)=\tanh(\beta z)\) (Appendix 7). Gradient descent on \(z\) with the reparameterization 7 is therefore exactly mirror descent on the primal \(u\in(-1,1)\): the dual variable accumulates gradients, and the mirror map squashes the iterate back into the box. Increasing \(\beta\) sharpens the map so that the primal iterates approach the binary vertices \(\{\pm1\}\); we anneal \(\beta:1\to16\) exponentially over training. In the final \(20\%\) of steps the forward pass uses \(\mathrm{sign}(z)\) (hard-forward) while gradients flow through the \(\tanh\) surrogate, and the final codes are \(B=\mathrm{sign}(z)\). Two diagnostics govern sign freezing: the flip rate \(R_{\mathrm{flip}}\) (fraction of sign changes per interval) and the softness \(R_{\mathrm{soft}}=\mathbb{E}|u-\mathrm{sign}(z)|\); measured terminal values in our runs were \(R_{\mathrm{flip}}\!\approx\!3\times10^{-4}\) and \(R_{\mathrm{soft}}\!\approx\!0.028\). The \(\ell_\infty\) box view is equivalent: \(su\) lives in \([-s,+s]\) per weight, the vertex-inducing pressure comes from the sharpened map, and a positive lower bound on \(s\) (enforced by the log parameterization) prevents scale collapse.

3.5 Total objective and ablation switches↩︎

The training objective is \[J=\lambda_E L_E+\lambda_{\mathrm{GE}} L_{\mathrm{GE}}+\lambda_D L_{\mathrm{KD}}, \label{eq:total}\tag{9}\] where \(L_{\mathrm{KD}}\) is a temperature-\(\tau\) KL on the teacher’s top-\(k\) logits (\(k{=}256\), \(\tau{=}2\)), used only in distillation conditions. Setting subsets of coefficients to zero yields the ablation conditions of Section 4; e.g., error compensation alone is \(\lambda_{\mathrm{GE}}=\lambda_D=0\).

4 Experiments↩︎

4.1 Setup↩︎

Model and quantization scope. Qwen2.5-1.5B [23]; all \(196\) linear matrices of the transformer blocks (\(\approx1.31\times10^{9}\) parameters) are quantized; embedding and LM head remain FP16. Group size \(G{=}128\) (1.125 bits/weight; quantized-part capacity \(184.2\) MB, \(7.0\%\) of FP16). Data. Calibration: WikiText-2 [24] train, \(0.13\)M tokens. Training: \(1{,}000\) steps \(\times\) batch \(8\) \(\times\) length \(512\) (\(4.1\)M tokens). Evaluation: WikiText-2 test (\(33\)K tokens); OOD evaluation uses C4 [25] (validation) and CNN/DailyMail (articles). Optimization. AdamW; learning rates \(5\times10^{-4}\) (\(z\)) and \(5\times10^{-3}\) (\(\log s\)); \(\beta:1\to16\); hard-forward for the last 20%. INT4 runs use \(300\) steps with learning rates \(2\times10^{-5}\) (latent weights) and \(5\times10^{-4}\) (\(\log s\)), reflecting the raw scale of INT4 latents. Protocol. Seeds \(\{1234,2025,7\}\) were fixed before any result was observed and never changed; every completed run is reported. Results are mean \(\pm\) standard deviation over seeds. Teacher perplexity: \(11.951\) (WikiText-2), \(16.18\) (C4), \(14.70\) (CNN/DailyMail). One binary condition-seed takes \(\approx9\) minutes on a single A100.

Conditions. (A) RTN: \(B=\mathrm{sign}(W)\), \(s=\mathrm{mean}|W|\), no training; (B) layer-local output reconstruction (each layer independently); (G) logit distillation only; (C) statistics matching only (\(\lambda_E{=}\lambda_D{=}0\)); (D) accumulated-error loss only (\(\lambda_{\mathrm{GE}}{=}\lambda_D{=}0\)); (E) combination of (C)+(D) with \(\lambda_{\mathrm{GE}}{=}0.1\).

4.2 Binary ablation (1.125 bits/weight)↩︎

Table 1 isolates each mechanism. Three observations. (i) Joint error compensation is the quality engine. (D) beats layer-local reconstruction (B) by more than two orders of magnitude and logit distillation (G) by \(32\%\) relative; the (D)–(G) gap exceeds \(8\sigma\) and the ordering is preserved in every seed. Since (B) is precisely the “minimize each \(q_l\) in isolation” strategy, the (B) vs.(D) gap is a direct measurement of the value of the cancellation structure in 4 . (ii) Statistics matching is not a quality driver. (C) alone collapses perplexity (ratio \(262\)) yet achieves the best \(E_{\mathrm{GE}}\) of all conditions—the two metrics dissociate. (iii) The combination inherits both. (E) sits near (D) in quality and near (C) in statistical equivalence; neither single mechanism achieves this pair. This is a functional interaction rather than an additive aggregation: the direction that preserves statistics is, by (C), catastrophic for quality on its own, and only the joint optimization finds solutions good under both criteria.

4.3 Bit-independence: INT4 (4.125 bits/weight)↩︎

Table 2 repeats the ablation with \(\mathcal{D}=\{-8,\dots,7\}\): (d) and (e) beat layer-local (b) by \(\approx9\sigma\). The identical objective, with no bit-specific modification, improves both the 1-bit and 4-bit regimes, supporting the claim that the compensation mechanism is a property of the objective, not of the value set.

4.4 Out-of-domain evaluation↩︎

Table 3 evaluates the same checkpoints on domains never seen during calibration. Error compensation’s advantage grows off-domain: the (D)/(G) quality gap widens from \(1.45\times\) in-domain to \(1.60\times\) on C4 and \(1.62\times\) on CNN/DailyMail; the mechanism is not overfitting the calibration domain. Statistics matching preserves equivalence off-domain: conditions with \(L_{\mathrm{GE}}\) hold \(E_{\mathrm{GE}}\) at \(0.42\)–\(0.88\) off-domain while conditions without it sit at \(1.41\)–\(2.99\). An honest negative: the relative OOD degradation factor (OOD ratio \(\div\) in-domain ratio) of (E) is not better than (D)’s (\(5.43\) vs.\(4.79\) on C4); adding statistics matching does not buy extra perplexity robustness. Its measurable benefit off-domain is the preservation of feature statistics, i.e., of the teacher-equivalence certificate, at a \(\sim20\%\) quality cost. Applications that require downstream-representation stability (feature reuse, probing, cascaded systems) are the natural consumers of this trade.

4.5 Equation-level verification↩︎

All core equations are unit-tested in NumPy independently of the training code: (1) the initialization 8 reproduces \(\tanh(\beta_0 z_0)=\mathrm{clip}(W/s_0,\pm\rho)\) to \(10^{-12}\) and preserves teacher signs; (2) \(\beta{=}64\) drives \(|u-\mathrm{sign}(z)|\) below \(0.02\); (3) centered linear CKA equals \(1\) under orthogonal transforms and exhibits the \(r/n\) bias on independent representations; (4) the statistics loss 6 is zero iff distributions match and responds to mean and variance shifts; (5) the recursion of Lemma 1 equals the forward difference exactly on random linear networks (\(10^{-10}\)).

5 Limitations↩︎

We state the boundaries of the evidence plainly. (1) Scale: all measurements are at 1.5B parameters; behavior at 8B+ is extrapolation, and the literature suggests larger models quantize more gracefully, which could shrink (or grow) the reported gaps. (2) Scope of binarization: embedding and LM head remain FP16; fully binarized configurations (including tied embedding/output projections) are designed in the framework but not measured here. (3) Baselines: condition (B) is our own layer-local reconstruction implementation, a controlled proxy for the GPTQ family rather than a head-to-head comparison against tuned public implementations; QEP-style sequential propagation is not re-implemented. (4) Metrics: we report perplexity and \(E_{\mathrm{GE}}\) only; no downstream task suite. (5) Statistics: three seeds suffice for the \(>8\sigma\) headline gaps but not for fine distinctions (e.g., INT4 (d) vs.(e)). (6) Compute: the exact accumulated-error loss costs one extra teacher-parallel forward per step; layer-recursive implementations with explicit operator application cost \(\approx3\times\) condition (D).

6 Conclusion↩︎

Treating extreme low-bit quantization as a joint optimization over all layers’ discrete variables—with the accumulated error as an explicit, provably exact recursive state, and finite-sample statistics matching as an equivalence-preserving regularizer—yields large, seed-stable improvements over layer-local and distillation-only training at 1.125 bits, transfers unchanged to 4.125 bits, and degrades gracefully out of domain. The two mechanisms play complementary roles that neither plays alone: error compensation owns output quality; statistics matching owns the preservation of the teacher’s representational statistics. We believe the exactness lemma, which collapses the operator formalism into two forward passes, makes the approach practical at larger scales, and we plan fully binarized (embedding and LM-head inclusive) experiments as the next step.

Reproducibility↩︎

Pre-registered seeds \(\{1234,2025,7\}\); all completed runs reported; exact hyperparameters in Section 4; equation-level unit tests in Section 4.5. Notebooks reproducing Tables 1–3 are available from the author.

7 Derivation of the mirror map↩︎

Let \(\psi_\beta(u)=\frac{1}{\beta}\big[\frac{1+u}{2}\log\frac{1+u}{2} +\frac{1-u}{2}\log\frac{1-u}{2}\big]\) on \(u\in(-1,1)\). Differentiating, \[\nabla\psi_\beta(u) =\frac{1}{2\beta}\log\frac{1+u}{1-u} =\frac{1}{\beta}\,\mathrm{artanh}(u),\] so the dual (mirror) variable is \(z=\mathrm{artanh}(u)/\beta\) and the inverse map is \(u=\nabla\psi_\beta^{*}(z)=\tanh(\beta z)\). A gradient step \(z\leftarrow z-\eta\,\partial J/\partial z\) followed by the inverse map is mirror descent on \(u\) with distance-generating function \(\psi_\beta\). As \(\beta\to\infty\) the curvature of \(\psi_\beta\) at the boundary sharpens, and minimizers of linearized objectives over \([-1,1]\) concentrate on the vertices \(\{\pm1\}\); annealing \(\beta\) therefore interpolates from soft averaging to vertex selection. The initialization 8 follows by inverting \(u_0=\mathrm{clip}(W/s_0,\pm\rho)\), the clip guarding \(\mathrm{artanh}\)’s domain; signs of \(z_0\) equal signs of \(W\), so \(B_0=\mathrm{sign}(W)\) and RTN is recovered at initialization.

8 First-order form of the recursion↩︎

With the teacher-side operator \(A_l=\partial F_l/\partial h|_{h_l}\), \[e_{l+1}=\widehat{F}_l(h_l+e_l)-F_l(h_l) =\underbrace{\widehat{F}_l(h_l+e_l)-\widehat{F}_l(h_l)}_{\widehat{A}_l(e_l)} +\;q_l =A_l e_l+q_l+O(\left\lVert e_l \right\rVert^2)+\big(\widehat{A}_l-A_l\big)(e_l),\] where \((\widehat A_l-A_l)(e_l)\) is second order in the perturbations (product of the weight perturbation and \(e_l\)). Lemma 1 shows the quantized-side instantiation absorbs all higher-order terms exactly.

9 CKA bias on independent representations↩︎

For centered linear CKA between independent random \(n\times d\) representations with \(d\ll n\), the expected value concentrates near \(d/n\) rather than \(0\) (rank-driven positive bias). Our unit tests use \(n=400\), \(d=16\) and verify \(\mathrm{CKA}<5\,d/n\), plus exact invariance under orthogonal transforms. Reported CKA values should therefore be read against this floor.

References↩︎