1 Introduction↩︎

Ternary weights are attractive: matrix multiplication degenerates into additions and subtractions, and storage approaches \(\log_2 3 \approx 1.58\) bits per weight. BitNet [1], [2] shows that 1-bit and 1.58-bit LLMs can be trained from scratch; ternary weight networks [3] and trained ternary quantization [4] established the format earlier for vision models. But a single ternary plane carries at most 1.58 bits of information per weight, so post-training projection of a full-precision matrix onto one ternary plane loses too much: reasoning collapses [5].

PTQTP [5] takes the important step of using two trit-planes, a superposition of two ternary matrices with row-wise scales, and shows this restores much of the lost expressiveness without retraining. But two planes is still a fixed capacity budget with a ceiling it cannot get past, and there is nothing special about two, or about any fixed number of planes. ExTernD replaces a fixed count of full-size planes with a factored representation whose inner dimension \(k\) is a free parameter: \(A \approx B\,\operatorname{diag}(D)\,C\) with both factors ternary. Setting \(k = \mu\min(m,n)\) (typically \(\mu = 2\)–\(3.5\), but unbounded) gives the representation as much capacity as needed; each additional component fits the residual left by all previous ones, so capacity is spent exactly where the ternary constraint hurt and accuracy is not capped (Sec. 2.1, proof in Appendix 7). All of this is strictly post-training: the factorization sees one weight matrix at a time and never a gradient.

Mainstream post-training quantization (GPTQ [6], AWQ [7], llama.cpp k-quants) sits at a small number of discrete accuracy/cost points: 3, 4, 5 bits per weight. Because \(k\) is a real-valued rank rather than a bit-width, ExTernD’s accuracy/cost trade-off is instead continuous: the inner-rank multiplier \(\mu\) (storage and compute both scale linearly with it, Sec. 2.6) and a sparsity threshold \(\tau\) (which sets the fraction of zeros in \(B\) and \(C\), e.g.\(\tau{=}0.7 \to {\sim}41\%\) zeros, \(\tau{=}1.0 \to {\sim}57\%\), \(\tau{=}2.0 \to {\sim}87\%\)) can each be set per matrix to any real value, letting a target accuracy be hit exactly rather than rounded up to the next bit-width.

Contributions: (i) ExTernD and a greedy ALS algorithm for it, with a proof (Appendix 7) that its error falls below any \(\varepsilon > 0\) as \(\mu\) grows, making it to our knowledge the only ternary quantization that can provably reach any target accuracy, up to bf16, and hence beat any fixed-plane-count method at large enough \(\mu\); (ii) a batched block-ALS GPU algorithm with decorrelated targets, matching sequential quality at \({\sim}16\times\) speed; (iii) an importance-weighted variant using llama.cpp imatrix statistics, including a non-obvious correction to the V-step; (iv) an empirical study on Gemma-4-E2B, Qwen3.5-4B and Granite-4.0-h-tiny, including a full end-to-end model conversion.

2 Method↩︎

2.1 Expanded inner rank↩︎

Given \(A \in \mathbb{R}^{m\times n}\), we seek \[A \;\approx\; \hat{A} \;=\; B\,\operatorname{diag}(D)\,C, \qquad B \in \{-1,0,1\}^{m\times k},\; C \in \{-1,0,1\}^{k\times n},\; D \in \mathbb{R}^k .\] Quality is reported as energy preserved, \(E = 1 - \|A - \hat{A}\|_F^2 / \|A\|_F^2\).

With real factors, \(k = \min(m,n)\) suffices (SVD). With ternary factors each rank-one component \(d_i\, b_i c_i^\top\) is a crude, quantized object; at \(k = \min(m,n)\) the representation hits an information ceiling well short of full precision. The core idea is to keep adding components past full rank: component \(i\) is fitted to the residual \(R = A - \sum_{j<i} d_j b_j c_j^\top\), i.e.to the accumulated quantization error of its predecessors. The multiplier \(\mu = k/\min(m,n)\) becomes the accuracy dial. Crucially this only works through sequential deflation: solving for all \(k > \min(m,n)\) components jointly is underdetermined and fails completely (0–50% energy in our ablation, vs.–100% with deflation).

Because \(\mu\) is unbounded and the residual is monotone in it (Appendix 7), the accuracy ceiling that a fixed plane count imposes does not exist here: any target accuracy is reachable by construction rather than by luck of the fit.

2.2 Ternarization operator and continuous sparsity↩︎

For \(u \in \mathbb{R}^m\), the adaptive mean-threshold ternarization with scale \(\tau\) is \[T_\tau(u)_i = \operatorname{sign}(u_i)\cdot \mathbf{1}\!\Big[\,|u_i| > \tau \cdot \tfrac1m \textstyle\sum_j |u_j|\,\Big],\] keeping the single largest entry if the result would be all-zero. \(\tau\) controls factor sparsity continuously and almost matrix-independently. The optimal ternary projection \(\arg\min_{t,\alpha} \|u - \alpha t\|_2\) has a closed form (sort \(|u|\), keep the top-\(s^\star\) prefix maximizing \(\frac{1}{\sqrt s}\sum_{i\le s}|u|_{(i)}\)) but empirically ties \(T_{0.7}\), so we use the cheaper threshold form.

2.3 Greedy sequential ALS↩︎

With residual \(R \leftarrow A\), for \(i = 1,\dots,k\):

1. Alternating ternary fit (15 iterations): \(v \leftarrow T_\tau(R^\top u)\), \(u \leftarrow T_\tau(R v)\).

2. Optimal scale: \(d_i = \dfrac{u^\top R v}{\|u\|_2^2\,\|v\|_2^2}\), the least-squares minimizer of \(\|R - d\,uv^\top\|_F\).

3. Deflate: \(R \leftarrow R - d_i\,uv^\top\); store \(B_{:,i} = u\), \(C_{i,:} = v^\top\).

Step 2 guarantees \(\|R\|_F^2\) never increases (Appendix 7), which is what makes \(\mu\) a well-behaved dial: more components monotonically means less error, all the way to exact recovery.

2.4 Batched block ALS (GPU)↩︎

Extracting components in blocks of \(b\) (default 256, capped at \(\lfloor\min(m,n)/8\rfloor\)) turns per-component vector work into block matmuls. The naive batching, ternarizing \(R^\top U\) column-wise, collapses (99.5% \(\to\) 50% energy at \(b{=}256\)): every column chases the same dominant residual direction. The fix is to ternarize the jointly decorrelated least-squares target: \[V \leftarrow T_\tau\!\Big(\big[(U^\top U + \epsilon I)^{-1} U^\top R\big]^\top\Big), \qquad U \leftarrow T_\tau\!\Big(R\,V\,(V^\top V + \epsilon I)^{-1}\Big),\] so each component is fitted against the residual minus its block-mates’ contributions (implicit within-block deflation; an exact block solve in place of Gauss–Seidel). After the alternating iterations, all \(b\) scales are solved jointly and exactly: \[\big[(U^\top U)\odot(V^\top V)\big]\, d = \operatorname{diag}(U^\top R V),\] then \(R \leftarrow R - U\operatorname{diag}(d)V^\top\). Deflation across blocks stays fully sequential. Block width must respect \(b \lesssim \min(m,n)/8\) or the ternary Gram matrices become ill-conditioned.

Refinement sweeps. After extraction, revisiting each block (add its contribution back to \(R\), re-run the block fit, re-deflate) improves energy monotonically: \({\approx}{+}0.4\) points on hard matrices after 10 sweeps, and a 4–8% reduction in the multiplier needed for fixed energy (larger at high \(\tau\)).

2.5 Importance-weighted ALS↩︎

Frobenius error weights all input channels equally; activations do not. Given per-input-channel second moments \(h\) from a llama.cpp importance matrix (imatrix), we minimize \(\sum_{ij} \tilde{h}_j\,(A_{ij} - \hat{A}_{ij})^2\) with \(\tilde{h} = h/\max(h) + \lambda\). The U-step and the joint scale solve generalize directly (\(W = R \odot \tilde{h}\) as target, \(V^\top\operatorname{diag}(\tilde{h})V\) Gram matrices); they are the exact weighted normal equations. The V-step does not: solving it against \(W\) loses 0.3–1.8 points of weighted energy versus the unweighted algorithm, because the \(\tilde{h}\) rescaling starves low-importance channels of support under a threshold that spans the whole component. In a column-weighted objective \(\tilde{h}\) cancels in the V-step, so the correct V-target is the plain residual \(R\). \(\lambda\) interpolates continuously between the uniform (\(\lambda \to \infty\)) and pure-imatrix (\(\lambda = 0\)) objectives; \(\lambda = 0\) matches what llama.cpp imatrix quants optimize and is stable in practice.

2.6 Cost model↩︎

Inference computes \(y = B(\operatorname{diag}(D)(Cx))\): \(k(m+n)\) ternary add/subtracts plus \(k\) multiplies, versus \(mn\) multiply-adds, a ratio of \(\mu\,(m{+}n)/\max(m,n)\) that is linear (hence continuous) in \(\mu\). With sparse mask+sign packing (a 1-bit zero/nonzero mask per stored element plus one sign bit per nonzero, \(\mathrm{bpw} = 2 - \mathrm{sparsity}\)), storage per original weight is \[\mathrm{bpw}_{\mathrm{eff}} = \mu \cdot \frac{m+n}{\max(m,n)} \cdot (2 - \mathrm{sparsity}), \label{eq:bpw}\tag{1}\] again linear in \(\mu\) and continuous in \(\tau\) through the sparsity. This packing is near the entropy floor for unstructured supports (mask entropy \(H(0.43) \approx 0.99\) bits; signs incompressible).

3 Results↩︎

Setup: matrices from google/gemma-4-E2B, Qwen/Qwen3.5-4B, and IBM Granite-4.0-h-tiny. Every experiment in this paper (decompositions, multiplier searches, the full-model conversion, and all perplexity evaluations) was run on a single AMD MI50 32 GB GPU (torch/ROCm). The Q4_K baseline is a faithful torch port of llama.cpp’s quantize_row_q4_K_ref (4.5 bpw).

3.1 Energy vs.multiplier; batched = sequential↩︎

On Qwen3.5-4B at \(\mu{=}2.5\), \(\tau{=}0.7\), 10 sweeps: MLPs 99.21–99.26%, v_proj 99.52%, o_proj 99.78%. Same algorithm, easier model; achievable energy at fixed \(\mu\) is strongly model-dependent.

3.2 The two dials interact favorably↩︎

Sweeping \(\tau\) at a fixed 99%-energy target: sparsity rises continuously (41% at \(\tau{=}0.7\) to 87% at \(\tau{=}2.0\)) while the required \(\mu\) rises (up_proj: 2.82 \(\to\) 7.21). The nonzero budget \(\mu\times\)density falls monotonically (1.65 \(\to\) 0.87): sparser factors with more components need fewer total nonzeros for equal energy, up to a convergence cliff near \(\tau = 2.5\). Under mask+sign packing (Eq. 1 ), where zeros also cost a mask bit, the optimum is interior at \(\tau = 1.0\) (\(\sim\)​57% sparsity).

3.3 Effective bits at matched Q4_K accuracy↩︎

Q4_K achieves a strikingly uniform 99.4–99.5% energy on every matrix of both models. Choosing \(\mu\) per matrix to match it exactly:

Because the packing is near the entropy floor, closing this gap is a multiplier problem, not a coding problem: parity requires \({\sim}17\%\) lower iso-quality \(\mu\). Refinement sweeps give \({\sim}5\%\); importance weighting (next) gives another \({\sim}4\%\) independently.

3.4 Importance weighting↩︎

With the corrected V-step (Sec. 2.5) at \(\lambda = 0\), weighting helps most where the imatrix is skewed: on Granite-4.0-h-tiny, v_proj weighted energy 96.55% \(\to\) 98.95% (\(+2.40\)); flat-imatrix tensors are unchanged. On Qwen3.5-4B, the rank multiplier needed for 99.5% weighted energy drops 2–8% on MLP gate/up and v_proj, and 4.5–7.4% on gated-DeltaNet projections, the best territory, since their qkv/out input importance is strongly skewed. Redoing the iso-Q4_K accounting under the weighted metric: mean bpw\(_{\mathrm{eff}}\) 5.49 \(\to\) 5.26 (\(-4.2\%\)), best tensors 5.08–5.14, shrinking the Q4_K gap from \({\sim}22\%\) to \({\sim}17\%\) overall and 13–15% on the best tensors.

3.5 End-to-end model conversion↩︎

All 200 language-model linear layers of Qwen3.5-4B (96 MLP, 32 attention, 72 gated-DeltaNet) were decomposed with the weighted algorithm at a fixed \(\mu = 3\), \(\tau = 1.0\), \(\lambda = 0\), no sweeps, in 19.9 minutes total on one GPU. Per-tensor energy: plain min/mean 98.52/99.45%, weighted 99.34/99.65%.

The pure ladder also shows that the tuned Q4_K_M mix beats pure Q5_K despite fewer bits: per-tensor budget allocation is worth more than 0.6 uniform bpw for k-quants, and per-matrix \(\mu\) allocation, trivially expressible here because \(\mu\) is continuous, is the analogous untapped lever for the decomposition.

4 Deployment notes↩︎

The factors store today in llama.cpp’s TQ2_0 (2.06 bpw; its per-block fp16 scales absorb \(D\) into \(C\)’s row scales for free), and bitnet.cpp-style LUT kernels [8] apply directly to both.

Emerging multiplication-free ASICs stand to exploit this the most, and the inner-rank expansion is designed with them in mind: the \(3.4\)–\(6\times\) op-count growth reads as slowdown only on multiplier-based datapaths, where a ternary add buys nothing over a multiply–add.

5 Limitations↩︎

Everything here ran on a single AMD MI50 32 GB GPU, which bounds the evaluation. We have not tested above 4B or on matrices larger than \({\sim}9216\times2560\), so the behaviour of \(\mu\) at 30B–70B+ is unknown. Evaluation is perplexity-only: no downstream benchmarks, no KLD, no long-context checks. We compare only against k-quants, with no head-to-head against established ternary schemes [2], [5], which is the most valuable missing experiment: our accuracy claim against them is formal (Sec. 2.1), not yet measured. The known levers (sweeps \({\sim}5\%\), weighting \({\sim}4\%\), per-matrix rank allocation) are unstacked, and no fused kernels exist.

QAT is untried and is the obvious next lever: a short straight-through pass on an ExTernD initialization should recover accuracy, plausibly faster than QAT on a conventionally quantized model, since rank expansion hands the optimizer \(\mu\,(m{+}n)/\max(m,n)\) times as many trainable ternary entries per layer, already initialized near target. Whether STE gradients behave through two chained ternary factors is untested.

Matching Q4_K or Q5_K bit-for-bit is not the objective; those are a familiar yardstick. The objective is a viable ternary LLM, where accuracy is set by \(\mu\) rather than capped by the format, and this is best read as a step towards highly accurate ternary quantization rather than a finished recipe.

6 Conclusion↩︎

Expanding the inner rank turns ternary representation from a fixed, lossy format with a hard accuracy ceiling into a family that provably converges to full precision as \(k\) grows. This is what makes ternary LLMs viable: accuracy becomes a dial (\(\mu\), continuous, with \(\tau\) for sparsity) rather than a ceiling, so multiplication-free kernels and multiplication-free hardware can be pointed at any accuracy target, to our knowledge the only known way to get there. A simple greedy ALS already lands within \(\sim\)​17–20% of Q4_K’s bit efficiency at matched accuracy, with identified, unstacked levers of comparable total size, while keeping inference multiplication-free.

7 Monotone residual decrease and convergence↩︎

In practice the safeguard never activates, since the ALS decrease is far larger than the single-entry bound, and empirical convergence is much faster than geometric in \(mn\) (99% energy at \(\mu \approx 2.5\)–\(3\)).

References↩︎