1 Introduction↩︎

Model merging has become a compelling post-training paradigm for large language models, because it offers a direct way to combine capabilities from existing checkpoints without full retraining [1]–[3]. In homogeneous settings, a broad family of methods has been proposed, including weight averaging and Model Soups [4], task arithmetic [5], TIES-Merging [6], Fisher-weighted averaging [7], RegMean [8], and sparsity-based variants [9]. Related modular architectures also show the value of selectively routing computation across experts [10]. Yet these successes largely rely on a strong compatibility condition: the source checkpoints share architecture, parameter shapes, and at least partially aligned internal representations. This condition is increasingly restrictive, since real model pools contain checkpoints of different scales, specializations, and design choices [3], [11].

Heterogeneous model merging relaxes this compatibility condition, but existing solutions often pay for it with extra machinery. Training-based approaches use distillation, generated data, adapters, modular skill storage, or learned latent spaces to transfer knowledge across models [12]–[17]. Training-free approaches reduce mismatch through permutation matching, activation repair, feature zipping, semantic alignment, or coefficient selection [18]–[23]. These methods are valuable, but they leave a sharper question unresolved: can we merge sufficiently different billion-parameter language models with low cost, no additional training, and no explicit semantic alignment? This question is especially important because direct weighted averaging across mismatched parameter spaces appears counterintuitive: a larger model and a smaller model do not even expose tensors of the same shape, and representational mismatch weakens the assumptions behind neuron-wise or feature-wise correspondence.

The technical challenge has two parts. First, parameter-shape mismatch prevents direct interpolation, so the models must be placed in a common parameter space before any weighted averaging is possible. Second, even after shape compatibility is enforced, interpolation can destroy capability if the two solutions are not connected by a stable low-loss path; related homogeneous studies show that interpolation depends strongly on mode connectivity, permutation alignment, and representation similarity [18], [24]–[27]. Recent work on cross-scale fusion and scaling behavior further suggests that the effectiveness of merging rules may change at LLM scale [28], [29]. This motivates a controlled test of the simplest possible hypothesis: after deterministic dimensional adaptation, is direct weighted averaging actually enough, provided that the mixing ratio is small?

We answer this question through training-free dimensional adaptation followed by ratio-controlled interpolation. In the union-style setting, we expand the smaller checkpoint into the larger parameter space before merging; in the intersection-style setting, we truncate the larger checkpoint into the smaller parameter space before merging. The resulting procedure uses no fine-tuning, no distillation data, no adapters, no routing module, no learned latent encoder, and no semantic alignment step. This makes the intervention deliberately simple and interpretable: performance changes can be attributed primarily to dimensional adaptation, mixing ratio, and the compatibility of the two source checkpoints.

Across Qwen-family model pairs at billion-parameter scale and diverse benchmark families, this simple procedure works more often than the standard intuition would suggest. Deterministic expansion largely preserves the smaller model’s function, showing that dimensional adaptation can act as a functional bridge rather than only a tensor-shape repair. Small-ratio interpolation can further improve performance by transferring complementary capabilities; for example, in union-style merging between Qwen2.5-14B and Qwen2.5-32B, the two source checkpoints obtain average scores of 0.7044 and 0.7424, respectively, while the best merged model reaches 0.7525, and intersection-style merging improves Qwen2.5-3B from 0.5459 to 0.5716 when a truncated 32B branch is injected with a small coefficient. At the same time, the method has clear limits: near-balanced interpolation often collapses, and task-level results reveal a seesaw effect where improvements on some capabilities coexist with regressions on others.

These findings suggest that heterogeneous LLM merging may follow a trajectory similar to homogeneous merging at scale: simple parameter averaging, when used in the right effective region, can be a surprisingly strong baseline, and the regimes where it fails may expose fundamental compatibility bottlenecks that more complex methods cannot easily bypass. Our contributions are fourfold: (1) we formulate low-cost, training-free, alignment-free heterogeneous LLM merging under substantial parameter-scale mismatch; (2) we introduce union-style and intersection-style dimensional adaptation as simple probes for direct weighted fusion; (3) we identify ratio-sensitive regimes of preservation, transfer, and collapse across billion-parameter model pairs; and (4) we show that task-level complementarity and seesaw behavior are central diagnostics for understanding when heterogeneous merging improves capability and when it merely redistributes or destroys it.

2 Related Work↩︎

Recent surveys summarize model merging for LLMs, MLLMs, and related foundation-model settings [1], [2].

2.0.0.1 Homogeneous model merging.

Prior work has largely focused on homogeneous settings, where models share architecture and parameter layout. Weight-based methods include weight averaging and Model Soups [4], task arithmetic [5], TIES-Merging [6], and DARE-style variants [30]. Geometry- or statistics-based methods include SLERP [31], Fisher-weighted averaging [7], and RegMean [8]. Sparse-mask merging reduces interference by retaining selected parameter updates [9], while MoE-style routing offers a related modular route for activating specialized experts [10].

2.0.0.2 Limitations in homogeneous settings.

Homogeneous merging relies on aligned parameter shapes, similar internal representations, and often careful hyperparameter tuning. More sophisticated strategies add optimization or engineering overhead, and scaling-law evidence suggests that merging gains do not automatically scale with model size [29]. In a recent in-the-wild evaluation with heterogeneous, potentially overlapping or conflicting fine-tuned checkpoints, Task Arithmetic was the only evaluated method that reliably yielded gains, while more complex subspace- or interference-aware methods often failed to provide clear improvements [32]. These constraints limit reuse of diverse real-world model pools.

2.0.0.3 Subspace-based merging and benchmarks.

A closely related line reduces interference by localizing or matching task-relevant parameter subspaces. TALL-masks localize sparse task information and use the resulting supports for compression and consensus merging [33]; Localize-and-Stitch identifies compact task-localized regions and stitches them into a pretrained model through sparse task arithmetic [34]; and MaTS formulates merging as matching models in task parameter subspaces through a linear-system solution [35]. These methods are connected to our intersection-style setting because both restrict the region of parameter space that participates in merging. The difference is that subspace-based methods usually estimate task supports or matching subspaces for shape-compatible task vectors, whereas our truncate-then-merge protocol first applies a deterministic architecture projection from a larger checkpoint to a smaller parameter space and then performs small-ratio interpolation. We therefore ask whether useful projected perturbations can be absorbed without learning a task subspace, solving a matching problem, or requiring additional optimization. Recent benchmarks such as MergeBench further emphasize the need for explicit evaluation of domain-specialized LLM merging methods and failure modes [36]; our setting is complementary because it focuses on cross-scale dimensional adaptation rather than standard same-shape merging.

2.0.0.4 Heterogeneous model merging.

Heterogeneous merging is more challenging because parameter-wise averaging is no longer obviously meaningful when source models differ in width, depth, module layout, or even modality. Existing approaches therefore often introduce additional mechanisms to make cross-model reuse feasible. One line of work uses training or learned intermediate modules: knowledge distillation transfers behavior through teacher–student optimization [12], AdapterFusion composes trained adapters rather than raw checkpoints [13], FuseLLM and FuseChat fuse knowledge through model-generated data and subsequent training [14], [15], GraftLLM stores capabilities in modular SkillPacks [16], and LS-Merge learns a latent weight space before merging language models [17]. These methods broaden the reuse of heterogeneous model assets, but their reliance on optimization, generated or curated data, trained modules, or latent encoders weakens the original appeal of model merging as a fast training-free operation.

A second line seeks training-free compatibility through alignment or repair. Git Re-Basin aligns models modulo permutation symmetries [18], REPAIR corrects activation statistics after permutation-based interpolation [19], and ZipIt merges models by matching and zipping related features across networks [20]. For language and multimodal models, semantic-alignment and coefficient-selection methods further reduce mismatch before interpolation, as in SeMe and AdaMMS [21], [22], while HM3 studies heterogeneous multi-class classifier merging [23]. However, many of these methods either focus on relatively small networks, require explicit semantic or feature alignment, or study heterogeneity that remains close to a shared base or task structure. As a result, it remains unclear whether sufficiently different billion-parameter language models can be merged by a simpler recipe: adapt dimensions without training and then directly interpolate weights.

Recent studies on cross-scale language-model fusion and model-merging scaling further motivate this question [28], [29]. In particular, if sophisticated merging rules become less distinguishable at larger scale [37], heterogeneous LLM merging may follow a similar trajectory: carefully controlled simple weighted interpolation could be a stronger baseline than expected. Our work tests this possibility directly by removing learned alignment, adapters, routing, and post-hoc training, and by asking whether dimensional adaptation plus small-ratio interpolation can preserve or improve capabilities across genuinely different LLM scales.

2.0.0.5 Position of this work.

We target training-free heterogeneous merging under parameter-scale mismatch with lightweight dimensional adaptation and ratio-controlled interpolation, preserving simple merging while extending it to broader model pairs.

3 Method↩︎

3.1 Problem Setup↩︎

We consider two pretrained models, denoted by \(A\) and \(B\), where model \(A\) is no larger than model \(B\) in parameter dimensionality. Let \(\theta_A \in \mathbb{R}^{d_A}\) and \(\theta_B \in \mathbb{R}^{d_B}\) denote their parameter vectors, with \(d_A \le d_B\). Let \(\mathcal{X}\) denote the input space, \(\mathcal{D}\) the evaluation distribution, and \(\ell\) the task loss. We study two complementary merging settings, referred to as union-style merging and intersection-style merging, to characterize when heterogeneous merging can preserve capability and when it fails.

3.2 Union-Style Merging (Expand Then Merge)↩︎

We first map model \(A\) to the larger space of model \(B\) through an expansion operator \[\mathcal{E}: \mathbb{R}^{d_A} \rightarrow \mathbb{R}^{d_B}, \qquad \tilde{\theta}_A = \mathcal{E}(\theta_A).\] Expansion deterministically copies compatible source tensors into the larger target tensors and initializes newly introduced dimensions to minimize functional perturbation, as illustrated in Figure 2 (a): newly introduced linear parameters are zero-initialized, normalization scales are initialized to neutral values, and additional Transformer blocks [38] are initialized as residual identity mappings. Architecture-aware mappings preserve grouped-query attention structure [39] and expand MLP matrices along their semantic input or output dimensions. Further normalization details are given in Appendix 9. We then perform convex interpolation in the high-dimensional space: \[\theta^{\cup}(\lambda) = (1-\lambda)\tilde{\theta}_A + \lambda\theta_B, \qquad \lambda \in [0,1].\]

3.3 Intersection-Style Merging (Truncate Then Merge)↩︎

Complementary to expansion, we study an intersection-style setting that projects the larger model \(B\) into the smaller parameter space of model \(A\) via a truncation operator: \[\mathcal{T}: \mathbb{R}^{d_B} \rightarrow \mathbb{R}^{d_A}, \qquad \tilde{\theta}_B = \mathcal{T}(\theta_B).\] As shown in Figure 2 (b), we instantiate the small-model configuration and initialize it from the large checkpoint by copying shape-compatible tensors, slicing oversized tensors, and removing layers beyond the target depth. Unlike expansion, truncation discards parameters and is inherently lossy, so the truncated branch must remain a small perturbation. Merging is performed in the low-dimensional space: \[\theta^{\cap}(\mu) = (1-\mu)\theta_A + \mu\tilde{\theta}_B, \qquad \mu \in [0,1].\]

3.4 Evaluation Protocol↩︎

Union-style and intersection-style merging are evaluated as two separate experimental protocols. For each protocol, we search over its own ratio grid and select the best checkpoint by downstream validation performance: \[\theta_{\cup}^{*} = \arg\min_{\lambda\in\Lambda} \mathcal{L}_{\mathrm{eval}}\big(\theta^{\cup}(\lambda)\big),\] \[\theta_{\cap}^{*} = \arg\min_{\mu\in M} \mathcal{L}_{\mathrm{eval}}\big(\theta^{\cap}(\mu)\big),\] where the corresponding ratio grids are specified in the Experiment section. The key empirical question is whether useful heterogeneous transfer appears in small-ratio neighborhoods of either anchor model, and where interpolation collapses within each protocol.

4 Experiment↩︎

3pt

3pt

3pt

We evaluate Qwen2.5 [41] and Qwen3 [42] model pairs ranging from 3B to 32B parameters. The benchmark suite spans mathematical reasoning, code generation, instruction following, NLU, commonsense, knowledge, and comprehensive reasoning, using GSM8K [43], MATH [44], [45], HumanEval/EvalPlus [46], [47], IFEval [48], GLUE-style NLU tasks [49], [50], PIQA [51], Winogrande [52], COPA [53], BoolQ [54], ARC [55], HellaSwag [56], MMLU [57], and BBH [58]. We use the EleutherAI LM Evaluation Harness [59] and BigCode Evaluation Harness [60]. The benchmark families and model-pair configurations are summarized in Tables 1 and 2.

4pt

4pt

4pt

4pt

Let \(\theta_s\) and \(\theta_l\) denote the smaller and larger checkpoints, and let \(\Pi_{s\to l}\) and \(\Pi_{l\to s}\) denote expansion and truncation operators. We evaluate: \[\theta_{\mathrm{union}}(\lambda)=(1-\lambda)\Pi_{s\to l}(\theta_s)+\lambda\theta_l,\] \[\theta_{\mathrm{inter}}(\mu)=(1-\mu)\theta_s+\mu\Pi_{l\to s}(\theta_l),\] We use fixed ratio grids for the two protocols. For union-style merging, \(\lambda \in \{0.02, 0.04, 0.96, 0.98\}\) controls the contribution of the larger checkpoint. For intersection-style merging, \(\mu \in \{0.01, 0.02, 0.03, 0.04, 0.05\}\) controls the contribution of the truncated larger checkpoint. The merged checkpoints are evaluated on the benchmark families reported for each protocol.

5 Main Results↩︎

Across three union-style settings, direct weighted averaging after deterministic expansion consistently improves the average score over both source checkpoints (Tables 3–5). Intersection-style merging also yields positive gains when the truncated larger checkpoint is injected with a small coefficient: Qwen2.5-3B improves from 0.5459 to 0.5716, and Qwen2.5-14B improves from 0.7045 to 0.7161 (Tables 6 and 7). The gains are largest when the two sources provide complementary competence rather than near-nested capabilities, supporting the view that heterogeneous merging can transfer useful signal without training or explicit alignment.

5.1 Knowledge-Structure Compatibility and Complementarity↩︎

The experimental results indicate that heterogeneous merging can produce meaningful complementarity when the two source models encode partially distinct competence profiles. This profile-based view is related in spirit to representation-similarity analyses that compare learned structures across neural models [26], [27]. At the aggregate level, the best merged configuration can outperform both source checkpoints, and task-level results further show that gains are distributed across multiple categories rather than concentrated in a single outlier task. This supports a practical interpretation of “complementary transfer”: when knowledge structure is sufficiently different, merging can inherit strengths from both sides and realize a clear compensation effect.

Figure 3 clarifies the structural condition behind this behavior. When the larger model nearly encloses the smaller model, overlap dominates and merging changes accuracy only marginally. When the two models exhibit an intersecting but non-nested relationship, merging introduces complementary signals and yields a visible gain of about four percentage points.

5.2 Scale Gap Drives Intersection Gains↩︎

We further evaluate intersection-style merging by merging Qwen2.5-3B with a Qwen2.5-32B checkpoint truncated to the 3B parameter space. The merged model is computed as \((1-\mu)\)Qwen2.5-3B \(+\mu\)Qwen2.5-32B-truncated, where \(\mu\) is selected from \(\{0.01,0.02,0.03,0.04,0.05\}\).

Although truncation is lossy, injecting the truncated 32B model with a small coefficient still provides a useful performance gain. Tables 6–9 show that Qwen2.5-3B improves from 0.5459 to 0.5716, while Qwen2.5-14B improves from 0.7045 to 0.7161. The gains in Tables 7–9 are milder because their dimensional gaps are smaller than the 3B–32B gap in Table 6. This suggests that intersection-style merging is most useful when the source models have a large capability gap, provided that the truncated branch remains a small perturbation.

6 Analysis↩︎

6.1 Analysis of Gain Patterns↩︎

As discussed in Section 5, gains concentrate when source models have partially distinct competence profiles. Figure 3 shows that intersecting but non-nested models leave transferable complementary regions, while the intersection-style results show that useful structure can survive truncation and be absorbed as a small perturbation (Table 6). The effect is more pronounced when the two models differ more substantially in scale.

6.2 Ratio Sensitivity and Effective Region↩︎

Direct averaging is effective mainly near an anchor model. Appendix [app:gsm8k] and Tables ¿tbl:tab:gsm8k-results?–[tbl:tab:gsm8k-truncate-14b-results] show two protocol-specific phase patterns. For union-style merging: \[\lambda \in (0,0.1) \cup (0.9,1) \Rightarrow \text{stable transfer},\] \[\lambda \in (0.1,0.9) \Rightarrow \text{severe performance collapse}.\] For intersection-style merging, where \(\mu\) weights the truncated larger checkpoint: \[\mu \in (0,0.1) \Rightarrow \text{stable transfer},\] \[\mu \in (0.1,1) \Rightarrow \text{severe performance collapse}.\] Thus, union-style merging has two endpoint neighborhoods, whereas intersection-style merging is one-sided because truncation is lossy. This ratio sensitivity is consistent with weight-space connectivity and alignment effects [24], [25].

6.3 Seesaw Effect↩︎

Heterogeneous merging is not monotonic. For Qwen2.5-14B and Qwen2.5-32B, the aggregate best merged configuration reaches 0.7525, above both sources (0.7044 and 0.7424; Table 3), but the task-level summary in Appendix 8 shows that the worst average falls to 0.7036 with task-level drops (Table 10). Aggregate gains can therefore hide capability-specific regressions; Appendix 8 provides additional details in Tables 11–16.

6.4 Perturbation Controls and Baseline Comparisons↩︎

We further test whether the gains can be explained by generic perturbation rather than cross-checkpoint transfer. Appendix 10 reports matched perturbation controls on the Qwen2.5-3B intersection setting. Target-only scaling and matched noise sometimes improve individual tasks, but they do not match the merged checkpoint on the reported tasks. For example, on RTE, the baseline, target-only scaling, matched noise, and merge scores are 0.6426, 0.6534, 0.6570, and 0.7256, respectively. On GSM8K, matched noise reaches 0.6808, while the merged checkpoint reaches 0.7028. This indicates that the improvement is not fully explained by adding a non-checkpoint perturbation of comparable scale.

We also compare weighted averaging after truncation with several standard merging rules in the same small-model parameter space. The comparison is not intended to show that weighted averaging dominates all possible merging algorithms. Instead, it tests whether the simplest zero-training, zero-alignment rule remains competitive after dimensional adaptation. On the reported average score, truncated-0.02 weighted averaging obtains 0.5551, close to TIES at 0.5587 and above SLERP, the 3B baseline, and DARE (Appendix 10). Thus, dimensional adaptation should be viewed as a compatibility operator that can support simple averaging and may also support stronger or complementary merging rules.

7 Conclusion↩︎

We show that deterministic dimensional adaptation enables direct weighted averaging across heterogeneous LLM scales. Across Qwen-family model pairs, small-ratio union and intersection merging can preserve source behavior and transfer complementary capabilities without training, adapters, routing, or semantic alignment. These results make simple weighted fusion a competitive zero-training baseline for probing heterogeneous LLM merging. At the same time, ratio sensitivity, perturbation controls, and seesaw effects show that compatibility is local and depends on the source pair, merging direction, and evaluation task. Task-level validation therefore remains essential before treating a merged checkpoint as uniformly improved.

Limitations↩︎

This study focuses on Qwen-family checkpoints, offline benchmarks, and representative ratios; broader evaluation of long-context, multilingual, tool-use, efficiency, calibration, robustness, and safety settings remains future work.

Ethical Considerations↩︎

Merged models may inherit unsafe or biased behavior from source checkpoints, so deployment requires safety checks.

8 Additional Task-Level Results↩︎

2.5pt

2.5pt

2.5pt

2.7pt

2.7pt

2.7pt

0pt 2.0pt

9 Dimensional Adaptation Details↩︎

This appendix gives the implementation-level form of the two dimensional adaptation operators used in Section 3. We separate the operator that maps a smaller checkpoint into a larger architecture, denoted by Expand, from the operator that projects a larger checkpoint into a smaller architecture, denoted by Truncate. These operators correspond to union-style merging and intersection-style merging, respectively. They are deterministic tensor maps applied before interpolation, not learned alignment modules.

9.1 Union-Style Expand↩︎

Let the source checkpoint have hidden size \(h_s\), intermediate size \(m_s\), number of layers \(L_s\), number of query heads \(H_s\), number of key-value heads \(K_s\), and head dimension \(d_h\). Let the target architecture have the corresponding quantities \(h_t,m_t,L_t,H_t,K_t\), with target dimensions no smaller than the source dimensions for the expanded axes. We use a slot map \[\pi_{n\rightarrow m}(i)=\left\lfloor \frac{i m}{n}\right\rfloor,\qquad i=0,\ldots,n-1,\] to place source coordinates or heads into deterministic target slots. For a tensor \(W\in\mathbb{R}^{a_s\times b_s}\) expanded along rows and columns, the generic zero-extension rule is \[\big[\mathcal{E}(W)\big]_{\pi_{a_s\rightarrow a_t}(i),\,\pi_{b_s\rightarrow b_t}(j)} = c\,W_{ij},\qquad \big[\mathcal{E}(W)\big]_{uv}=0\;\text{otherwise},\] where \(c\) is one for ordinary linear weights and may be set to \(\sqrt{h_s/h_t}\) for copied hidden-width normalization coordinates in width-changing expansions. In practice, hidden coordinates are placed at a head-dimension block granularity when the architecture exposes a fixed head dimension. This preserves contiguous per-head subspaces rather than scattering individual coordinates.

Attention projections are expanded with grouped-query structure preserved. Query rows are mapped at whole-head granularity, while key and value rows are mapped at key-value-head granularity. The output projection applies the inverse placement on its input-head dimension and the hidden-coordinate placement on its output dimension. Thus the target architecture keeps its own attention scale \(1/\sqrt{d_h}\) and head dimension. New attention rows or columns are zero at initialization, so they do not introduce untuned high-variance logits before interpolation.

The MLP expansion follows the semantic axes of the SwiGLU block. For gate and up projections, source rows are copied into the first or mapped \(m_s\) intermediate slots and source hidden columns are copied into mapped hidden slots. For the down projection, source hidden rows are copied into mapped hidden slots and source intermediate columns are copied into the retained intermediate slots. All newly introduced MLP coordinates are zero-filled. Because a SwiGLU branch has the form \[\operatorname{MLP}(x)=W_{\mathrm{down}}\left(\operatorname{SwiGLU}(W_{\mathrm{gate}}x,W_{\mathrm{up}}x)\right),\] zero-filling new gate, up, and down coordinates prevents newly inserted nonlinear branches from contributing before interpolation.

For additional layers, we use residual identity initialization. Consider a pre-normalization Transformer block with normalization operator \(\mathcal{N}_{\ell}\) and attention/MLP branch \(\mathcal{G}_{\ell}\): \[h_{\ell+1}=h_{\ell}+\mathcal{G}_{\ell}\big(\mathcal{N}_{\ell}(h_{\ell});W_{\ell}\big).\] For a newly inserted layer, all attention and MLP branch weights are set to zero, while normalization scale parameters in the inserted layer are set to the neutral value one. This gives \[\mathcal{G}_{\ell}\big(\mathcal{N}_{\ell}(h_{\ell});W_{\ell}^{\mathrm{new}}\big)=0, \qquad h_{\ell+1}=h_{\ell},\] so \(\Delta h_{\ell}=0\) at initialization. Normalization scales are not set to zero, because zero scales would collapse normalized feature variation. For LayerNorm [61], \[\mathcal{N}_{\ell}(h)=\gamma_{\ell}\odot \frac{h-\bar{h}}{\sqrt{\operatorname{Var}(h)+\epsilon}}+\beta_{\ell},\] and the local Jacobian scales with \(\gamma_{\ell}\). Setting \(\gamma_{\ell}=\mathbf{1}\) preserves a well-conditioned normalized signal. The same intuition applies to RMSNorm [62].

4pt

The stability claim for union-style merging is local. After expansion, interpolation can be written as \[\theta^{\cup}(\lambda)=\mathcal{E}(\theta_s)+\lambda\big(\theta_t-\mathcal{E}(\theta_s)\big).\] For small \(\lambda\), the expanded source checkpoint remains the anchor. For \(\lambda\) close to one, the target checkpoint becomes the anchor. Near-balanced interpolation does not have this protection, and nonlinear gates can amplify destructive interference. This explains why the method can work in endpoint neighborhoods while failing in the middle of the ratio range.

9.2 Intersection-Style Truncate↩︎

For intersection-style merging, we instantiate the small-model architecture and project the large checkpoint into that state dictionary. Let \(W_B\in\mathbb{R}^{s_1\times\cdots\times s_r}\) be a tensor from the larger checkpoint and let \(W_A\in\mathbb{R}^{t_1\times\cdots\times t_r}\) be the corresponding tensor shape in the smaller architecture. The truncation operator copies the overlapping prefix region: \[\big[\mathcal{T}(W_B)\big]_{i_1,\ldots,i_r} = \big[W_B\big]_{i_1,\ldots,i_r}, \qquad 0\le i_j<\min(s_j,t_j).\] Dimensions outside the target shape are discarded. Identical-shape tensors are copied directly. Layers whose indices exceed the small model depth are removed. For MLP matrices, this corresponds to retaining the target intermediate width: gate and up projections are sliced on their output dimension, while down projections are sliced on their input-intermediate dimension. For tied embeddings, the output head follows the target small-model tying convention.

4pt

Truncation is intentionally lossy. We therefore do not require \(\mathcal{T}(\theta_B)\) to be a strong standalone model. The merged checkpoint can be written as \[\theta^{\cap}(\mu)=\theta_A+\mu\big(\mathcal{T}(\theta_B)-\theta_A\big),\] so the residual stream, normalization scales, and attention computations remain anchored by \(\theta_A\) when \(\mu\) is small. If the local input-output map around \(\theta_A\) is approximated by a locally Lipschitz function \(F\), then \[\left\|F(\theta^{\cap}(\mu))-F(\theta_A)\right\| \lesssim \mu L \left\|\mathcal{T}(\theta_B)-\theta_A\right\|,\] up to higher-order nonlinear terms. This bound is only a local stability interpretation, not a guarantee. In particular, SwiGLU gates, residual-stream scale changes, and attention-logit entropy can make the middle-ratio regime unstable. Our experiments therefore treat attention entropy, task-level regressions, and collapse at \(\mu\in(0.1,0.9)\) as diagnostics of the compatibility boundary rather than as contradictions of the small-ratio result.

The absence of permutation or semantic alignment is deliberate. Expand and Truncate assume architecture-compatible cross-scale models whose tensor names, module roles, and head structure are sufficiently comparable. The collapse regime marks where this assumption becomes too weak for direct weighted averaging.

10 Additional Diagnostics↩︎

10.1 Ablation Controls and Baseline Comparisons↩︎

We add two diagnostic comparisons to separate cross-checkpoint transfer from simpler alternative explanations. Table 17 compares intersection-style merging with target-only scaling and matched noise controls in the Qwen2.5-3B setting. Table 18 compares weighted averaging after truncation with representative merging rules evaluated in the same small-model parameter space.

5pt

8pt

10.2 GSM8K Ratio Sensitivity↩︎

References↩︎