1 Introduction↩︎

Building capable language models today requires substantial resources. DeepSeek-R1 [1] used thousands of GPUs to show that reinforcement learning alone can induce emergent reasoning. A.X-K1 [2] involved a large team to build a 519B-parameter Korean model, and GLM-5.2 [3] scales to 744B parameters. Scaling laws [4] suggest that frontier budgets are needed for frontier capabilities.

This paper explores a complementary direction: getting more out of a fixed budget. We study three techniques that, together, allow competitive models to be trained with substantially less compute.

The starting observation is that language-model supervision is often inefficient. In a typical instruction-response pair, only a minority of output tokens carry factual or reasoning content (cf.[5]); the rest is syntactic scaffolding that a pretrained model already produces well. Standard cross-entropy loss [6] weights all tokens equally, spreading the supervision signal across positions where it adds little.

We show that concentrating supervision on this semantic minority has three useful consequences.

1. A gradient-coupling (“wave propagation”) effect (Section 3.2): because transformer weights are shared across positions, a gradient step on the supervised tokens also improves the unsupervised ones. This is a token-level instance of auxiliary-task transfer [7]; we measure its magnitude and show it depends on linguistic coherence.

2. Selective supervision provides a signal strong enough to recover from aggressive depth compression (Section 3.3): 48 layers reduced to 6, with roughly 78% of the loss gap recovered in 500 steps.

3. Compressibility enables expert fusion (Section 3.4): several compressed models, assembled as mixture-of-experts [8] experts, improve over any single model at the same active-parameter budget.

We validate these techniques on CHERRY-1.8B (Section 5), a Korean foundation model whose every trainable parameter derives from our own training runs, with no dependence on any external model’s weights at inference time. We use sovereign in this specific, narrow sense throughout (Table 11).

1.0.0.1 Contributions.

• We characterise the wave-propagation effect as positive gradient coupling, prove that it guarantees improvement of unsupervised tokens when the coupling is positive (Theorem 1), and measure it on held-out data.

• We demonstrate depth compression with recurrent recovery (\(2.5\times\) fewer parameters at loss within noise of the dense baseline).

• We introduce MoEE, a fusion of compressed experts with selective distillation.

• We describe CHERRY-1.8B, which integrates these techniques.

• We study the induction of self-correction (“Atcha”) behaviour through minimal two-token supervision (97.6% retention at the 1.2B scale).

Research Questions↩︎

This paper is structured around ten research questions (RQs) derived from a single foundational hypothesis:

“If we train only on semantically decisive tokens (GT tokens), will the remaining semantically adjacent outputs also improve through a wave-like propagation effect in shared weights?”

Each RQ is experimentally verified with explicit pass/fail results. Table 1 provides the complete verification matrix.

2 Related Work↩︎

2.0.0.1 Token-level training objectives.

Standard language model training applies uniform cross-entropy across all output positions [6]. Rho-1 [5] and its conceptual parent RHO-LOSS [9] select high-excess-loss tokens via a reference model, reducing pretraining tokens by 5–10\(\times\). Like SGT, Rho-1 also operates at the token-within-sequence level and masks loss on unselected positions; the difference is the selection criterion—Rho-1 selects by excess loss relative to a reference model for corpus filtering, whereas SGT selects by semantic role (entities, answer keywords, reasoning/meta-cognitive pivots) for instruction tuning, paired with an explicit anchor term that prevents the collapse we observe at \(\alpha=1\) (Observation 3). Focal loss [10] downweights easy examples in classification but does not select which positions to supervise. Curriculum learning [11] orders examples by difficulty but treats all tokens within each example uniformly. The trivial precursor of span-selective supervision—answer-only loss masking, standard in instruction tuning—supervises only the response span; SGT generalises this to sub-response semantic selection.

2.0.0.2 Gradient coupling and auxiliary-task transfer.

The mechanism underlying wave propagation, namely a gradient step on one loss reducing another when their gradients are positively aligned, is the defining quantity of multi-task and auxiliary-task learning. Gradient surgery (PCGrad) [7] formalises constructive vs.conflicting gradients via exactly the gradient inner product that defines our coupling coefficient \(\gamma\) (Eq. ?? ), and the empirical neural tangent kernel [12] governs how a step on one example changes predictions on others. Our Theorem 1 is a token-level specialisation of this transfer to position-shared weights; our contribution is not the existence of the coupling but its measured magnitude (\(\bar\gamma{=}0.72\)), its dependence on linguistic coherence (Corollary 1), and its use as a training-efficiency principle.

2.0.0.3 Importance sampling and selective backpropagation.

Katharopoulos and Fleuret [13] weight training examples by estimated loss contribution; Jiang et al. [14] skip low-importance examples during backpropagation. Process reward models [15] assign token-level importance scores for reasoning supervision. These methods operate at the example or step level; SGT operates at the token-within-sequence level, maintaining the full causal context while concentrating supervision on semantically decisive positions. Masked language modelling (BERT [16], T5 [17]) also trains on token subsets, but in a bidirectional reconstruction objective incompatible with autoregressive generation.

2.0.0.4 Knowledge distillation.

Hinton et al. [18] established soft-target distillation for model compression. Subsequent work has explored layer-wise [19], attention-based [20], and task-specific distillation. Our SGT distillation restricts the KL objective to GT positions, exploiting wave propagation to transfer teacher knowledge to the full student distribution at \({\sim}5\times\) lower cost than full-sequence distillation.

2.0.0.5 Model compression and pruning.

Structured pruning removes entire layers [21], attention heads, or intermediate dimensions. LLM-Pruner [22] uses gradient-based importance scoring. Closest to our recipe, Gromov et al. [23] prune contiguous deeper layers and heal with light fine-tuning—the same merge-then-recover structure we use, but via deletion rather than averaging. Our adjacent-layer merging is complementary: rather than removing layers, we average adjacent pairs, preserving a smooth initialisation that recovers quickly under SGT training.

2.0.0.6 Mixture of Experts.

Shazeer et al. [8] introduced sparsely-gated MoE for language models. Switch Transformers [24] simplified routing to top-1 selection. Recent frontier models (DeepSeek-V3 [25], GLM-5.2 [3]) use MoE at trillion-parameter scale. Assembling independently obtained models is studied by Branch-Train-Merge [26] (parallel expert LMs combined by ensembling) and Model Soups [27] (weight averaging of fine-tuned models). Our MoEE differs by obtaining experts via compression of a shared backbone and assembling them with a learned, MTP-augmented router rather than by averaging weights (soups) or routing experts trained on divergent data (BTM).

2.0.0.7 Recurrent depth and parameter reuse.

Universal Transformers [28] loop a single layer with adaptive halting; ALBERT [29] ties one block’s weights across all depths. Geiping et al. [30] scale recurrent depth for test-time compute. Our RDT combines heal-after-merge compression (cf.[23]) with ALBERT-style weight tying realised as recurrent unrolling of the merged core, recovering effective depth from a compressed parameter set.

2.0.0.8 Speculative decoding.

Leviathan et al. [31] and Chen et al. [32] use a small draft model to propose tokens verified by the target model. Multi-token prediction heads [33] enable self-speculative decoding without a separate draft model. Our oracle MTP uses cross-attention prediction heads trained with future hidden states, enabling 1.6–2.1\(\times\) speedup without an auxiliary model.

3 Approach↩︎

3.1 Overview↩︎

The Cherry framework operates in four phases:

1. Selective training: Identify semantically decisive tokens (GT set \(\mathcal{G}\)) and train with a mixed loss \(\mathcal{L}_{\mathrm{SGT}}\) that upweights GT positions.

2. Compression: Reduce model depth via adjacent-layer merging.

3. Recurrence: Restore effective depth through learned loop unrolling of the compressed model’s middle layers.

4. Fusion: Assemble compressed models as MoE experts with SGT-guided distillation from frontier teachers.

Each phase builds on the previous: selective training provides the signal quality that helps compression recovery, compression provides the parameter efficiency that makes expert fusion practical, and fusion provides representational diversity that improves quality at a fixed active-parameter budget.

3.2 Selective Ground Truth Token Training (SGT)↩︎

3.2.1 GT token hierarchy↩︎

3.2.2 GT token identification↩︎

GT tokens are identified via a hybrid pipeline that respects the model’s chat template, reasoning format, and tool-calling conventions:

1. Chat-template-aware preprocessing. Training data is fully mapped to the target model’s chat template: system prompt, multi-turn history, reasoning and tool-calling special tokens (e.g.<|spartan_think|>; spartan is an organisational codename retained in the vocabulary), and structured-output delimiters. This ensures GT identification operates on the exact token sequence the model will encounter at inference.

2. Rule-based seeding. Named entities, numerical values, answer-final tokens, first tokens of structured output fields, closing delimiters (quotes, braces), and generic/property references in tool-call schemas are tagged using part-of-speech and NER heuristics.

3. Contrastive refinement. For each candidate token \(y_i\), we compute the activation difference \(\delta_i^\ell = \|\mathbf{h}(y_i^+) - \mathbf{h}(y_i^-)\|_2\) between a correct and corrupted response at layer \(\ell\). Tokens exceeding \(\delta_i > \mu + \sigma\) are promoted to \(\mathcal{G}\).

3.2.2.1 GT type taxonomy

We distinguish six functional categories of GT tokens, each addressing a distinct aspect of model behavior:

3.2.2.2 Partial causal formulation

For instruction-tuning data, SGT uses a partial causal setup: given the full chat context up to the GT region boundary, the model is asked to generate only the GT token span causally. Formally, for a GT span \([g_\text{start}, g_\text{end}] \subseteq \mathcal{G}\): \[\mathcal{L}_{\text{partial}} = -\sum_{t=g_\text{start}}^{g_\text{end}} \log p_\theta(y_t \mid y_{<t})\] where \(y_{<t}\) includes the full prefix (system prompt, multi-turn context, response up to \(g_\text{start} - 1\)). This is more efficient than full-sequence causal training: only \(|\mathcal{G}|/|\mathcal{F}| \approx 15\%\) of positions contribute to the loss, yet wave propagation (Theorem 1) ensures the remaining 85% improve.

3.2.2.3 Empirical statistics

The GT ratio on our training data is \(|\mathcal{G}|/|\mathcal{F}| = 0.153 \pm 0.021\) across 1,024 sampled examples. By category: Factual 41%, Structural 23%, Reasoning 18%, Self-correction 8%, Guardrail 6%, Verification 4%. When \(|\mathcal{G}| = 0\) for a given example (rare; \(<\)​0.3%), the SGT loss reduces to standard cross-entropy.

3.2.3 SGT loss↩︎

\[\mathcal{L}_{\mathrm{SGT}} = \alpha \cdot \frac{1}{|\mathcal{G}|} \sum_{i \in \mathcal{G}} \ell(f_\theta(\mathbf{y}_{<i}), y_i) + (1 - \alpha) \cdot \frac{1}{|\mathcal{F}|} \sum_{i \in \mathcal{F}} \ell(f_\theta(\mathbf{y}_{<i}), y_i) \label{eq:sgt}\tag{1}\]

The mixing coefficient \(\alpha \in [0,1]\) balances GT-focused and full-causal supervision. We find \(\alpha = 0.7\) optimal.

3.2.4 Wave propagation: theory↩︎

3.2.4.2 The \(\alpha = 0\) control: \(\mathcal{W} > 1\) is not created by SGT.

We confront the most important control directly. At \(\alpha = 0\) (ordinary full-sequence training, no GT selection at all), the 1.0B DUS model shows \(\mathcal{W} = 14.76\)—higher than every SGT configuration (Table 16). This is decisive for interpreting the wave factor: the fact that non-GT tokens improve more in absolute loss than GT tokens (\(\mathcal{W} > 1\)) is a generic property of training on natural language—semantically loaded GT tokens start at low loss with little room to fall, while syntactic non-GT tokens start high and fall far—and is not produced by SGT. We do not, therefore, present \(\mathcal{W} > 1\) as evidence for SGT. SGT’s contribution is efficiency: it achieves a comparable non-GT improvement while applying direct supervision to only 15% of tokens, a \(4.5\times\) per-supervised-token return (Eq. 2 , Table ¿tbl:tab:wave?). The genuine, SGT-specific phenomenon—non-GT improvement carried by positive gradient coupling rather than direct supervision—is isolated not by \(\mathcal{W} > 1\) but by the coherence contrast of Corollary 1 (\(\bar\gamma = 0.72\) on coherent text vs.\(\to 0\) on shuffled text) and by the collapse of pure-GT training when the anchor is removed (Observation 3).

3.2.5 “Atcha” moment induction via SSGT↩︎

A central hypothesis of this work is that meta-cognitive state transitions—the “Aha moment” observed emergently in DeepSeek-R1 [1]—can be induced rather than discovered. We term these transitions “Atcha” moments—the Korean equivalent of “Aha”—and demonstrate that explicitly training on the minimal token set that triggers self-correction is sufficient to install meta-cognitive capabilities.

3.2.5.1 Core hypothesis

DeepSeek-R1 [1] reported the “Aha moment” as an emergent phenomenon discovered during RL training. We hypothesize that such moments are not rare emergences but inducible through targeted supervision: training on the single pivot token that triggers self-correction, plus the corrected answer keyword, should propagate to the entire reasoning chain via wave propagation. We call these SSGT tokens (\(\mathcal{G}^{**}\)): the minimum viable supervision for inducing meta-cognitive state transitions.

3.2.5.2 “Atcha” moment taxonomy

We formalize three distinct meta-cognitive moments, each requiring different SSGT token configurations:

3.2.5.3 Experimental design

For each moment type, we construct training examples where:

• The full chat context up to the moment trigger is provided as input (partial causal formulation, Section 3.2.2).

• Only the SSGT tokens (2–5 tokens per example) are supervised via the SGT loss (\(\alpha = 0.8\)).

• The hypothesis: if wave propagation holds for SSGT, then training on \(<\)​0.5% of total tokens should reshape the entire reasoning trajectory.

3.2.5.4 Results

• Retention: 97.6% at 1.2B scale, 116.9% at 0.5B. The \(>\)​100% retention at 0.5B indicates that SSGT outperforms full-sequence training. We attribute this to an implicit regularisation effect: at small scales, full-sequence supervision overfits to syntactic tokens (95% confidence interval on 200-sample retention:\(\pm\)​5.4pp; the 116.9% effect is significant at \(p < 0.01\) vs.% null). Concentrating supervision on meta-cognitive pivots avoids this dilution.

• “Atcha” self-correction rate: fraction of hallucination-inducing prompts where the model spontaneously self-corrects increases from 12% (base) to 47% after SSGT training (\(3.9\times\) improvement from \(<\)​0.5% of tokens).

• “Jamkkan” verification rate: fraction of uncertain-context prompts where the model requests additional verification increases from 8% to 34% (\(4.3\times\)).

• Guardrail compliance: jailbreak success rate drops from 23% to 4% with SSGT training on breaking tokens alone.

3.2.5.5 Implication for scaling

If SSGT works at 1.8B, the same principle applies to trillion-parameter models: training on a handful of meta-cognitive pivot tokens per example—rather than the full output sequence—could dramatically reduce alignment training cost while achieving equivalent or superior self-correction capability. The key insight is that “Aha moments” are not discovered but prescribed: by explicitly teaching the model when and how to self-correct (Atcha), verify (Jamkkan), or refuse (guardrail), we install meta-cognitive capabilities through minimal supervision.

3.3 Recurrent Depth Compression↩︎

3.3.1 Adjacent-layer merging↩︎

Given a transformer with \(L\) layers, we compress to \(L/k\) layers by averaging all parameters of each group of \(k\) adjacent layers [35]. The compressed model starts at high loss (\({\sim}13\) for \(k \geq 2\)) but recovers rapidly under SGT training.

3.3.2 Recurrent unrolling↩︎

Inspired by Universal Transformers [28] and latent recurrence [30], the compressed model’s architecture is partitioned into prelude (first \(\lfloor L'/6 \rfloor\) layers), recurrent core (middle layers), and coda (final \(\lfloor L'/6 \rfloor\) layers). The recurrent core is unrolled \(n\) times, creating \(L' + n \cdot |core|\) effective layers with only \(L'\) unique parameter sets.

3.4 Mixture of Efficient Experts (MoEE)↩︎

3.4.1 Architecture↩︎

\(N\) compressed models serve as experts, each with recurrent unrolling. Following the sparsely-gated MoE paradigm [8], a learned linear gate selects top-\(k\) experts per token with soft weight combination: \[\mathbf{h}_{\mathrm{out}} = \sum_{j \in \mathrm{top\text{-}k}} w_j \cdot f_{\theta_j}(\mathbf{x})\] A load-balancing auxiliary loss prevents expert collapse.

3.4.2 Oracle multi-token prediction↩︎

Three MTP heads at horizons \(h \in \{1,2,3\}\) predict future tokens via cross-attention to the combined expert output: \(\mathcal{L}_{\mathrm{MTP}} = \sum_{h=1}^{3} 0.5^h \cdot \mathrm{CE}(\mathrm{MTP}_h(\mathbf{z}_t), y_{t+h})\). At inference, these heads enable self-speculative decoding [31]–[33] (1.6–2.1\(\times\) wall-clock speedup).

3.4.3 SGT distillation↩︎

A quantised frontier model serves as a frozen teacher [18]. SGT restricts KL-divergence to GT positions: \[\mathcal{L}_{\mathrm{distill}} = \alpha \sum_{t \in \mathcal{G}} \mathrm{KL}(p_{\mathrm{teacher}} \| p_\theta)_t + (1{-}\alpha)\,\mathcal{L}_{\mathrm{full}}\] Note: the KL term sums over \(|\mathcal{G}|\) positions without the \(1/|\mathcal{G}|\) normalization of Eq. 1 ; this is intentional—the unnormalized form strengthens the GT signal when \(|\mathcal{G}|\) is large, which we found empirically beneficial for distillation.

Wave propagation carries teacher knowledge from GT positions to the full student distribution, reducing distillation cost by \({\sim}5\times\).

3.4.4 Sovereign multi-teacher fusion↩︎

The MoEE paradigm extends naturally to frontier-scale models. Consider a landscape of publicly available frontier models: GLM-5.2 (744B/40B active) [3], DeepSeek-V4 (1.6T/49B active) [36], MiniMax-M3 (428B/22B active) [37], A.X-K1 (519B/33B active) [2], K-EXAONE-236B (236B/23B active) [38]. Each of these can be:

1. Compressed via adjacent-layer merging into an efficient sub-model (e.g., a 236B model reduced to a \({\sim}\)​4–6B expert that, at NVFP4 [39]—NVIDIA’s 4-bit floating-point format—fits in a few GB; Table 6).

2. Used as a teacher for SGT distillation, where only GT-position logits are matched (reducing distillation memory and compute by \(5\times\)).

3. Assembled as an expert in a MoEE architecture, where learned routing selects the best compressed teacher per token.

This triple-use paradigm—standalone model, distillation teacher, MoE expert—is what makes frontier knowledge accessible to sovereign AI development. The resulting student is parametrically independent of any teacher: a sovereign model that absorbs frontier capabilities without being a derivative.

4 Experiments↩︎

All experiments use CHERRY-1.8B (48 layers, 1.019B text backbone; total 1.829B including 385M embeddings, 343M MTP heads, 42M fusion layers, 12M gate parameters, and 28M norms/biases—all compression and SGT experiments operate on the 1.019B backbone unless otherwise stated) on a single workstation with 128 GB of unified memory, using AdamW [40] (\(\beta_1{=}0.9\), \(\beta_2{=}0.95\)), learning rate \(2{\times}10^{-5}\), BF16 mixed precision, and seed 42 unless otherwise noted. Training data: 12,800 curated Korean instruction-response pairs (avg. tokens/example, 6.55M total tokens), sourced from a mix of Korean knowledge QA, multi-turn dialogue, chain-of-thought reasoning, and safety-alignment examples. Examples were selected to maximise domain diversity and GT-token density; the relatively small dataset size is by design—SGT’s efficiency hypothesis predicts that concentrating supervision on GT tokens should extract disproportionate value from limited data. Generalisability to larger or English-dominant corpora remains to be validated (Limitations). Batch size: 4, sequence length: 2,048, gradient accumulation: 4 (effective batch: 16). Each of the 500 steps processes \(16 \times 2{,}048 = 32{,}768\) tokens; total training budget: 16.4M tokens (\({\sim}\)​0.016 epochs over a 1B-token corpus). GT tokens constitute 15.3% of response tokens on average.

4.1 Wave propagation validation↩︎

4.1.0.1 Comparison with full-sequence training

At matched step counts, full-sequence training (\(\alpha{=}0\)) achieves lower absolute eval loss (0.681 vs. at step 500) because it directly supervises all tokens. This is expected—SGT’s advantage is not lower loss at equal steps, but comparable loss with dramatically lower supervision cost. The key insight is per-token efficiency: SGT concentrates gradient signal on the 15% of tokens that carry semantic payload, and wave propagation (Theorem 1) transfers the improvement to the remaining 85%. Per supervised token, SGT achieves \(4.5\times\) higher loss reduction than full-sequence training.

In the deployment scenario that motivates this work—adapting a pretrained model with limited curated data—the relevant comparison is not loss-at-fixed-steps but annotation cost. Labelling GT tokens requires expert identification of semantic payload, which is \({\sim}6\times\) more cost-effective than labelling full sequences for equivalent downstream quality (Section 8).

4.1.0.2 Wave factor

Under SGT, GT loss drops 94% (\(0.925 \to 0.054\)). Non-GT loss drops 61% (\(2.742 \to 1.075\)). In absolute terms, non-GT improvement (\(\Delta = 1.667\)) is \(1.91\times\) the GT improvement (\(\Delta = 0.871\)). Per-token efficiency: at the step-100 eval optimum, training on 15% of tokens produces \({\sim}\)​67% of the total loss reduction, yielding \(4.5\times\) per-token ROI (defined as \(0.67/0.15\): the ratio of fractional loss reduction to fractional token supervision). At the step-500 checkpoint reported elsewhere for consistency the recovered fraction is higher; we cite the conservative step-100 figure. In perplexity terms (\(e^{\mathcal{L}}\)), eval total drops from \(e^{2.585} = 13.3\) (step 0) to \(e^{0.994} = 2.70\) (SGT step 500) vs.\(e^{0.681} = 1.98\) (full-sequence step 500). SGT’s higher final perplexity reflects the per-token efficiency trade-off: training on \(6.7\times\) fewer supervised tokens at \(4.5\times\) the per-token yield.

4.1.0.3 Why 500 steps: convergence analysis

We use 500 steps as the standard training budget throughout this paper. The choice is justified by three observations:

1. Eval loss plateau. Eval loss stabilises after step 100, oscillating within a \(\pm\)​0.12 band from steps 100–500 (range: 0.797–1.044; Appendix 10.2). Step-100 achieves the best eval loss (0.797); subsequent steps do not improve it.

2. Train loss convergence. Train loss drops below 0.1 by step 200 and remains there (\(< 0.05\) by step 400), indicating that the GT token loss has been effectively minimized.

3. Wave factor stability. \(\mathcal{W}\) stabilises at \(2.18 \pm 0.15\) from step 100 onward, confirming that the wave propagation effect is fully established within the first 100 steps and maintained through step 500.

The eval loss uptrend from step 100 (0.797) to step 500 (0.994) reflects overfitting on the 12,800-example training set—a 25% increase typical of small-dataset fine-tuning. We report step-500 results throughout for consistency across experiments; the best eval model is at step 100. Extending to 1,000 steps (verified in a separate run) produced \(<\)​0.01 additional improvement, confirming that 500 steps capture the full SGT learning dynamic at this scale. For production deployment, early stopping at step 100–200 with the best validation checkpoint is recommended.

4.2 Depth compression scaling↩︎

Key observations:

• 6L\(\times\)​8 (227M, loss 2.934) \(\approx\) 24L\(\times\)​2 (566M, loss 2.926). Parameter reduction: \(2.5\times\) at identical performance.

• All compressed models recover \(>\)​75% of the loss gap in 500 steps despite starting at loss \({\sim}\)​13.

• 8L\(\times\)​6 (264M) slightly outperforms 12L\(\times\)​4 (340M): more recurrence with fewer unique layers can be more effective.

4.3 MoEE validation↩︎

The central question for MoEE is whether routing across compressed experts yields gains that justify the additional parameters. We address this with two comparisons: total parameters and active parameters (the parameters used per token during inference).

4.3.0.1 Active-parameter fairness

MoEE with top-2 routing across 2 experts activates one expert (12 layers, \(\sim\)​512M parameters) per token, comparable to the single 24L\(\times\)​2 model (566M). At matched active parameters, MoEE achieves 2.789 vs.—a 4.7% reduction. This gain comes from routing diversity: the gate selects the expert whose compressed representation best matches each input, and the two experts develop complementary specialisations during training (Expert A converges toward factual/structural content, Expert B toward reasoning-heavy sequences; routing entropy 0.94 confirms balanced utilisation).

4.3.0.2 Versus uncompressed model

The 48L uncompressed model (1,019M, all active) achieves dramatically lower loss (0.994). MoEE does not compete with uncompressed models at matched total parameters—its value proposition is deployment under compression constraints: when a model must fit in limited memory (e.g., edge devices, multi-tenant serving), MoEE extracts more capability from compressed experts than any single compressed model of equal or greater active size.

4.3.0.3 Detailed MoEE ablation (RQ4)

To isolate the contribution of each component, we run an ablation study holding total parameters approximately constant (\({\sim}\)​1B):

4.3.0.4 Frontier compression feasibility (RQ4 extension)

We project MoEE to frontier-scale models using the compression ratios validated at 1B scale. The core claim: if adjacent-layer merging compresses a 48L model to 6L with 78% recovery, the same technique applied to frontier models yields workstation-deployable experts for MoEE assembly.

^*Recovery rate extrapolated from validated 1B-scale experiments (Table 4). Frontier-scale compression experiments are in progress; the projection is a lower bound, as larger models typically exhibit higher layer redundancy [21].

4.4 SGT distillation validation↩︎

We validate SGT distillation by using the full 48L model as teacher and a 6L compressed model as student.

SGT distillation achieves 19% lower loss than SGT alone (0.797 vs.) on a 6L student, with KL divergence converging from 0.43 to 0.19 over 500 steps. The teacher’s knowledge transfers through GT positions and propagates to the full distribution via the wave effect—confirming the theoretical prediction of Section 3.2.

4.4.0.1 Note on distillation vs.standalone loss

The 6L distilled student (0.797) achieves lower eval loss than the 48L model at step 500 (0.994) but not than the 48L at step 100 (0.797, Table ¿tbl:tab:wave?). Two factors explain this: (1) the 48L step-500 number reflects overfitting on 12,800 examples, while the 6L’s lower capacity provides implicit regularisation; (2) distillation adds the teacher’s soft-target distribution as a regulariser, smoothing the student’s output beyond what standalone SGT provides. The fair comparison is 6L+distill (0.797) vs.6L SGT-only (0.984): distillation improves the same architecture by 19%.

4.5 Multi-seed reproducibility↩︎

Wave propagation is reproducible: across three random seeds, \(\mathcal{W}\) ranges from 12.21 to 13.67 with standard deviation 0.62 (coefficient of variation: 4.8%). All seeds show stable training with \(\mathcal{W} \gg 1\).

4.5.0.1 Reconciling \(\mathcal{W}\) magnitudes.

The eval-set wave factor (Table ¿tbl:tab:wave?, \(\mathcal{W} \leq 2.18\)) and the multi-seed wave factor (Table 8, \(\mathcal{W} \approx 12.8\)) differ because they measure different quantities. Table ¿tbl:tab:wave? reports absolute loss deltas on held-out data where the baseline non-GT loss is already low (2.74), bounding the ratio. Table 8 uses the 1.0B DUS model on training-set losses where the non-GT baseline is \(>\)​13, producing larger absolute \(\Delta_{\bar{\mathcal{G}}}\) and hence larger \(\mathcal{W}\). Both confirm the core claim: non-GT tokens improve more than GT tokens under SGT.

4.6 SSGT “Atcha” moment verification↩︎

4.7 “Atcha” moment observation results↩︎

To verify that SSGT training actually installs meta-cognitive capabilities (RQ5, RQ6), we evaluate on a held-out set of 200 Korean prompts per moment type, measuring whether the model spontaneously exhibits the trained behavior without explicit instruction.

4.7.0.1 Evaluation protocol.

Self-correction rate is measured by an automated classifier: we prompt the model with factual questions where the correct answer requires revising an initial plausible-but-wrong continuation (seeded by few-shot examples that do not contain self-correction). A response is scored as “self-correcting” if it contains an explicit revision marker (e.g., “a, jamkkan” (oh, wait) / “dasi bomyeon” (on reflection) / “sasileun” (actually)) followed by a factually different answer from the initial continuation. Correction accuracy is then judged by exact-match against the ground-truth answer. Verification request rate and jailbreak success rate use analogous keyword+semantic classifiers. All classifiers were validated against 100 human-annotated examples per category (Cohen’s \(\kappa > 0.82\)); the 200-sample evaluation yields 95% CI of \(\pm\)​6.9pp on rate metrics, which we acknowledge as a limitation.

4.7.0.2 Key findings

1. Atcha moment is inducible, not emergent (RQ5 = ✔). Training on an average of 2.1 pivot tokens per example (\(<\)​0.4% of total tokens) increases the self-correction rate \(3.9\times\) and correction accuracy from 34% to 78%. The model learns when to self-correct and what to correct—both from the same minimal supervision.

2. Wave propagation extends to meta-cognitive tokens. The Atcha pivot token is GT; the subsequent correction reasoning chain (5–20 tokens) is non-GT. Yet the correction chain improves from 34% to 78% accuracy, confirming that wave propagation (Theorem 1) operates at the meta-cognitive level: training the pivot reshapes the entire downstream reasoning trajectory.

3. Jamkkan moment validates RAG-aware self-verification (RQ6 = ✔). The model learns to request verification before committing to uncertain claims. Post-verification accuracy improves \(+32\)pp, indicating the verification request is not performative but functionally useful.

4. Guardrail breaking tokens are sufficient for safety alignment (RQ6 = ✔). Training on 1.8 breaking tokens per jailbreak example reduces jailbreak success from 23% to 4% without increasing false refusals (\(+1\)pp). This is a \(>\)​5\(\times\) cost reduction versus full-sequence safety training.

5 CHERRY Architecture↩︎

CHERRY-1.8B is a sovereign Korean foundation model integrating all three innovations (Fig. 9).

5.1 Sovereign tokenizer↩︎

We train a custom BPE [41] tokenizer on a Korean-centric corpus with 24 governance-aware special tokens (<|spartan_think|>, <|spartan_tool|>, <|spartan_refuse|>, etc.—spartan is an organisational codename retained in the token vocabulary) that encode reasoning mode, tool invocation, and safety governance directly in the token vocabulary.

5.2 Text backbone: cross-model depth-up-scaling↩︎

48-layer transformer (1.019B parameters) constructed via depth-up-scaling [34] from a 24-layer seed. All weights are fully overwritten during continual pre-training on 50B curated Korean tokens. Post-CPT weight-space cosine similarity to the seed is below 0.35.

5.3 Oracle multi-token prediction↩︎

Three cross-attention prediction heads [33] (343M total) at horizons \(h \in \{1,2,3\}\) receive oracle future hidden states during training. At inference, the same heads enable self-speculative decoding [31]–[33] at 1.6–\(2.1\times\) wall-clock speedup.

5.4 Sovereign fusion encoder (Cabstractor)↩︎

81 learned queries and 3 cross-attention layers fuse features from four frozen specialist extractors (vision, fine-grained vision, multilingual visual, audio) via soft MoE gating. All fusion parameters—queries, cross-attention weights, and gating network—are trained from scratch on Korean multimodal data. This constitutes a form of multi-teacher ensemble distillation [18] where the Cabstractor learns a unified representation independent of any single extractor.

5.5 Parameter sovereignty↩︎

6 Pitfalls, Resolutions, and Negative Results↩︎

Several configurations failed on first attempt. We separate those we resolved (stating the resolution and pointing to the in-paper evidence that the resolved recipe works) from the genuine negatives that represent real constraints.

6.1 Resolved pitfalls↩︎

6.1.0.1 RDT requires prior SGT training.

Applying recurrent loop unrolling to an untrained, freshly-merged model increases loss by 212–346%: recurrence amplifies whatever representation the model currently holds, and a post-merge representation is incoherent. Resolution: apply \(\geq\)​100 SGT steps before unrolling. With that ordering, unrolling the merged 6-layer core to \(n{=}8\) improves loss by 39.2% (Appendix 11.2, Table 21); we adopt train-then-unroll throughout.

6.1.0.2 Training budget.

100 steps achieve only 57% compression recovery; 500 steps achieve \(>\)​75%. Resolution: we use 500 steps as the standard budget, justified by the convergence analysis in Appendix 14; early stopping at step 100–200 is recommended for production.

6.1.0.3 SSGT needs the right evaluation data.

Measured on non-chain-of-thought data (Korean MCQ), SSGT “retention” is \(-\)​35%, simply because such data contains almost no pivot tokens for the mechanism to act on. Resolution: evaluate on CoT data with explicit self-correction sequences, where SSGT reaches 97.6% retention at 1.2B (Table 9). SSGT is not expected to help where there are no pivot tokens; this is a scope clarification, not a failure of the method.

6.2 Genuine negative results↩︎

6.2.0.1 Pure GT/SSGT supervision (\(\alpha{=}1\)) collapses.

Training exclusively on GT or meta-cognitive pivot tokens, with no full-causal anchor, increases loss by \(>\)​11 nats at 0.5B and \(>\)​1.6 nats at 1.2B. The anchor (\(\alpha<1\)) is necessary, not optional (Observation 3); this is a real constraint that we do not circumvent.

6.2.0.2 Contrastive GT discovery degrades below \({\sim}\)​1B.

Automatic GT identification via contrastive activation difference (\(\delta_i^\ell = \|\mathbf{h}(y_i^+) - \mathbf{h}(y_i^-)\|_2\)) achieves 100% hit@5 at 1.2B but only 80% at 0.5B: smaller models develop less distinct internal representations for factually decisive tokens. We mitigate this with a hybrid rule-based plus contrastive pipeline, but pure contrastive discovery at small scale remains an open limitation (RQ10).

7 RQ Verification Summary↩︎

We revisit each research question with its definitive answer.

7.0.0.1 Summary

Of 10 research questions: 8 verified (✔), 1 falsified (\(\times\)), 1 partially verified (\(\triangle\)). The falsification of RQ9 is itself informative: pure GT supervision collapses because the model loses global coherence without the full-sequence anchor. This establishes the mixed loss (Eq. 1 ) as necessary, not merely convenient. The partial verification of RQ10 identifies a clear scale-dependent limitation: contrastive GT discovery requires sufficient model capacity to develop distinct internal representations for semantically decisive tokens.

8 Discussion↩︎

8.0.0.1 Maximising utilisation rather than capacity.

The three techniques studied here, selective supervision, depth compression, and expert fusion, are complementary to the conventional approach of adding parameters, data, and compute. Rather than adding capacity, they aim to make better use of existing capacity, through selective training signals, parameter reuse via recurrence, and representational diversity via routing. We do not claim these replace conventional scaling; we offer them as compute-efficient alternatives that are useful under tight resource budgets.

8.0.0.2 The wave effect is a general property of shared weights.

The improvement of unsupervised tokens follows from positive gradient coupling, which is present whenever weights are shared across positions, as in the standard Transformer [42] (Theorem 1, Remark 2). As discussed, this is a token-level instance of auxiliary-task transfer [7] rather than a new phenomenon; the practical contribution is to measure its magnitude and to exploit it for per-token efficiency through GT-token selection.

8.0.0.3 Depth redundancy.

That 6 unique layers can approach the performance of 24 unique layers (Table 4) is consistent with prior evidence that much of a trained transformer’s depth is redundant [21], [23]. Averaging adjacent layers loses little information, and recurrent unrolling [28], [29] then recovers effective depth through parameter reuse.

8.0.0.4 A path toward independent model development.

MoEE with SGT distillation offers a practical route to building models whose parameters are independent of any external teacher. Table 13 sketches a prospective compression roadmap in which a publicly available model is reduced to a workstation-deployable size while retaining domain expertise; the compressed model can then serve as a standalone deployment, a distillation teacher, and an MoE expert (Fig. 7). We emphasise that this roadmap is prospective and not yet validated at frontier scale (see the caveats below).

8.0.0.5 Positioning relative to frontier models.

Table 14 situates CHERRY in the current landscape by scale and accessibility. We are deliberately careful here: this is not a quality comparison, and we do not claim that a 1.8B model matches the general capability of frontier systems two to three orders of magnitude larger. What the table does show is the dimension on which a small sovereign model legitimately differs: CHERRY is Korean-native, fully sovereign, and deployable on a single workstation at roughly \(1/45\) to \(1/100\) of the active parameters, and two to three orders of magnitude fewer total parameters, than frontier models. A fair head-to-head quality comparison requires standardized Korean and general benchmarks (KMMLU, HAERAE, KoBEST; MMLU, GSM8K), which we are currently running and will report separately. On general capability we expect CHERRY to trail much larger models; the comparisons where a model of this size can be genuinely competitive are (i) efficiency-normalized quality (performance per active parameter and per training FLOP) and (ii) narrow Korean-domain parity after adaptation. We make only those claims, and only once measured.

8.0.0.6 What we are not showing.

This paper presents the core mechanisms and validates them experimentally. Several components of the full Cherry system—including the dynamic recurrence depth controller, the expert routing algorithm’s domain-specialisation procedure, and the contrastive GT discovery method’s production implementation—are reserved for subsequent publications. The results reported here use only the fundamental mechanisms; evaluation of the full integrated system is ongoing.

8.0.0.7 Future work

Several directions follow naturally from this study.

1. Downstream and cross-lingual evaluation. The most important next step is to complement the loss-based metrics reported here with standard downstream benchmarks (KMMLU, HAERAE, KoBEST for Korean; MMLU, HellaSwag, GSM8K for general capability) and to replicate the GT ratio and coupling measurements on non-Korean corpora.

2. Interpretability-guided GT discovery. Extending GT-token identification beyond rule-based and activation-difference heuristics toward mechanistic methods that locate the neurons and attention pathways most responsible for factual and reasoning predictions.

3. Capability-tiered expert routing. Using SSGT-identified meta-cognitive tokens to score individual experts in a mixture and to route harder examples preferentially to experts that are stronger at self-correction or verification.

4. Smaller-scale validation. If 15% of tokens can influence the remaining 85%, the minimum viable model size for a given domain may be smaller than current scaling laws suggest. We plan to test SGT on sub-100M-parameter models for narrow Korean applications.

5. Cross-architecture validation. Applying SGT to non-transformer sequence models (Mamba [43], RWKV [44], linear attention), where position-shared weights take different forms, to test whether the gradient-coupling effect generalises.

8.0.0.8 Limitations

(1) All metrics are reported in terms of cross-entropy loss on held-out data rather than downstream task benchmarks (e.g., MMLU, HellaSwag, KoBEST). While loss is a faithful proxy for model quality at fixed tokenizer and data distribution [4], downstream evaluation is ongoing and will be reported in a subsequent version. (2) Experiments span 227M–1.019B parameters; 10B+ validation is in progress. (3) Multi-seed validation (seeds 42, 123, 7) confirms low variance (\(\mathcal{W} = 12.82 \pm 0.62\)), but broader seed sweeps and independent dataset replication remain future work. (4) Compression experiments use one model family (Llama-based DUS); generalisation to other architectures (e.g., Mamba, RWKV) is untested. (5) MoEE uses 2 experts; scaling to 4+ is computationally constrained on current hardware. (6) The 12,800-example training set is Korean-only; cross-lingual and cross-domain generalisability of the GT ratio and wave propagation magnitude is unvalidated. (7) SSGT evaluation (Table 10) uses 200 samples per category, yielding 95% CI of \(\pm\)​6.9pp on rate metrics; the absolute rate improvements (\(3.9\times\), \(4.3\times\)) are significant, but finer-grained comparisons require larger samples. (8) Statistical significance of loss differences is not formally tested (e.g., paired bootstrap); we rely on multi-seed consistency and effect magnitudes. We plan to add formal tests in subsequent versions. (9) We compare SGT (selecting GT tokens) against full-sequence training (\(\alpha{=}0\)), but not against two controls that would further isolate the role of selection: supervising a random 15% of tokens, and supervising the highest-loss 15% (a Rho-1-style criterion [5]). These controls are needed to attribute the efficiency gain specifically to semantic GT selection rather than to sparsity or the mixed-loss anchor, and we flag them as the most important missing experiment. (10) Under the mixed loss (\(\alpha{=}0.7\)) non-GT tokens still receive direct supervision at weight \(1{-}\alpha\), so the reported non-GT improvement combines a direct-supervision component with the gradient-coupling (wave) component; pure-GT supervision (\(\alpha{=}1\)), which isolates the wave, collapses (Observation 3). The wave is thus best evidenced by the coherence contrast (Corollary 1), not by the magnitude of \(\mathcal{W}\).

Data and Code Availability↩︎

The SGT loss function (Eq. 1 ), the compression procedure, and all training hyperparameters are fully specified in this paper to support reproduction. We intend to release model weights and evaluation artifacts in future work, subject to ongoing review.

Author Contributions↩︎

Dohyeon Kwon (first author) developed the SGT/SSGT framework, the GT-token taxonomy, the gradient-coupling analysis, the depth-compression and MoEE methods, and the formalisation of induced self-correction. Kwon designed and conducted the experiments across model scales (227M to 1.8B), built the training infrastructure, performed the data curation and analysis, and wrote the manuscript.

Youngjin Park, Ph.D. (corresponding author, \(\dagger\)), as Vice President at TeamSparta Inc., advised the project throughout: evaluating the research direction, reviewing the experimental methodology and theoretical arguments, and providing feedback on the manuscript.

Acknowledgements↩︎

We thank TeamSparta Inc.for providing the research environment and computational resources for this work, and the open-source AI community whose tools and models this research builds upon.

Competing Interests↩︎

Both authors are employees of TeamSparta Inc. Patent applications covering certain aspects of the methods described in this paper are under review in collaboration with TeamSparta Inc. We are committed to finding the right balance between intellectual property protection and open scientific progress: the core theoretical contributions (wave propagation theorem, SGT loss formulation, compression methodology) are fully disclosed in this paper for academic reproduction, and we will continue to contribute to the open-source and open-weight ecosystem through progressive model and tool releases. No competing financial interests beyond the stated employment relationship exist.

Appendix↩︎

9 Notation↩︎

10 Selective supervision and wave propagation↩︎

10.1 Full mixing-coefficient sweep↩︎

3pt

10.2 Wave-propagation trajectory↩︎

10.3 Cross-scale metrics↩︎

10.4 Per-category GT contribution↩︎

10.4.0.1 Key insights

1. Factual tokens alone achieve 75% of full SGT performance at 41% of the GT budget, confirming that factual entities are the highest-ROI supervision targets.

2. Reasoning tokens yield the highest per-token ROI (\(8.9\times\) vs.\(4.5\times\) for full SGT), consistent with the hypothesis that CoT pivots carry disproportionate semantic load.

3. Self-correction (SSGT) tokens achieve \(11.3\times\) ROI—the highest among all categories—but the absolute eval loss (\(2.13\)) is furthest from full SGT. This confirms that SSGT is best suited for installing specific capabilities (self-correction, verification) rather than general-purpose training.

4. All single-category runs show \(\mathcal{W} > 1\), confirming that wave propagation operates regardless of which GT category provides the supervision signal.

11 Depth compression↩︎

11.1 Compression-recovery trajectory↩︎

11.1.0.1 Observation

Recovery is remarkably uniform across compression levels (81–84%), suggesting that adjacent-layer merging preserves the model’s essential structure regardless of compression ratio. The first 100 steps provide \(>\)​60% of the total recovery; steps 100–500 provide the remaining \({\sim}\)​20%. This rapid initial recovery is consistent with the SGT hypothesis: GT tokens carry enough signal to quickly re-establish the core representation, with wave propagation filling in the non-GT positions.

11.2 Recurrent depth transformation (RDT)↩︎

11.2.0.1 Observation

Performance improves monotonically up to \(n{=}8\) (34 effective layers), then degrades slightly at \(n{=}10\) and \(n{=}12\). This suggests a sweet spot: too few iterations under-recover depth, while too many amplify noise from the compressed weights. The optimal effective depth (34 layers) is close to the original 48 layers divided by the compression ratio: \(48 / 8 \times 6 \approx 36\), supporting the interpretation that RDT approximately recovers the original computational depth through parameter reuse.

12 Expert fusion and distillation↩︎

12.1 MoEE training curve↩︎

12.1.0.1 Step-400 training spike

The train loss spikes from 3.368 to 7.826 at step 400 while eval loss continues to decrease (2.982 \(\to\) 2.943). This transient spike is a known artefact of load-balancing auxiliary loss in MoE training [8], [24]: when the router rebalances expert utilisation, the losing expert temporarily receives out-of-distribution inputs, inflating train loss for one step. The eval loss is unaffected because the routing has already converged on the evaluation distribution. The spike is reproducible across seeds (observed at steps 380–420 in all 3 runs) and self-resolving.

12.2 SGT distillation trajectory↩︎

13 Induced self-correction (SSGT): training details↩︎

13.0.0.1 Observation

The “Atcha” self-correction rate increases monotonically throughout training, even as eval loss shows mild overfitting after step 100. This decoupling between loss and behavioral metrics suggests that SSGT optimises a meta-cognitive capability that is not fully captured by cross-entropy loss. The SSGT wave factor (\(\mathcal{W} \approx 1.9\)) is consistent with the standard SGT wave factor (Table ¿tbl:tab:wave?), confirming that wave propagation operates identically at the meta-cognitive level.

14 Training budget: why 500 steps↩︎

14.0.0.1 Justification for 500-step budget

Three convergence criteria are satisfied by step 500: (1) train loss drops below 0.05 (GT tokens effectively memorised); (2) eval loss oscillates within \(\pm\)​0.12 of the step-100 minimum (no meaningful improvement possible); (3) \(\mathcal{W}\) stabilises at \({\sim}\)​1.9 (wave propagation fully established). The 750- and 1,000-step extensions confirm diminishing returns: \(<\)​0.01 improvement for \(50\%\)–\(100\%\) more compute. For production use, early stopping at step 100–200 is recommended.

References↩︎