1 Introduction↩︎

Model merging has emerged as a practical approach for combining expert models trained separately into a single multi-task model [1], [2]. By operating directly in weight space, merging avoids the computational overhead of multi-task joint training and the data-sharing requirements that come with it. Unlike model ensembling or mixture-of-experts approaches, weight-space merging does not increase inference latency or memory footprint, since all individual experts are merged into a single model. These benefits make model merging particularly attractive for Large Language Models (LLMs), where training costs are substantial, and inference latency must remain low.

A common assumption in the model merging literature is that domain experts should be trained to optimal validation loss before merging. This convention is implicit in major benchmarks and method comparisons: experts are fine-tuned with standard early-stopping or fixed-epoch protocols and then frozen at their best validation checkpoint before being combined [3]–[5]. The implicit assumption is that each expert contributes its best possible knowledge to the merged model when frozen at its individual optimum. Recent work has begun to question this assumption: [6] show, for vision (CLIP) and encoder-decoder (T5) models, that experts trained well past their individual optimum can in fact hurt the merged model. We revisit the question for decoder-only LLMs fine-tuned with parameter-efficient adapters and find a more nuanced picture in which the right training duration depends strongly on the merging method.

This paper challenges this assumption. Drawing an analogy from ensemble learning theory, where random forests deliberately grow each decision tree to high depth before averaging predictions [7], we hypothesize that the optimal training duration for merging may differ from the optimal duration for individual model performance. We investigate whether a similar mechanism operates in weight space when merging fine-tuned LLM experts: do some merging methods behave like the averaging step in a random forest and benefit from intentionally overtrained experts?

We focus on small open-weight LLMs because of their prevalence in industrial fine-tuning pipelines. Enterprises commonly fine-tune models below 8B to match or exceed the task-specific quality of larger general-purpose models at a fraction of cost and latency [8], [9]. Model merging is especially relevant in this regime, since deploying multiple specialized models is typically more expensive than one merged model with comparable capabilities. We conduct a systematic study across 3 model sizes (Qwen 3.5 0.8B, 2B, 4B), 5 task domains (Math, Code, Instruction Following, Multilingual, Safety) and 5 merging methods (Simple Averaging, Task Arithmetic, TIES-Merging, DARE+TIES, Greedy Soup).

Our contributions are as follows. First, we present a systematic study of how training duration of domain experts affects the quality of merged decoder-only LLMs, spanning different model sizes, domains, and merging methods. Second, we show that the optimal training duration is method-dependent: Simple Averaging peaks with undertrained experts, Task Arithmetic and Greedy Soup occupy an intermediate regime, while sparsification-based methods (TIES, DARE+TIES) peak with overfitted experts. We explain these results through bias-variance-covariance decomposition and mode connectivity analysis, showing that overfitted experts provide higher diversity that benefits methods with interference-resolution mechanisms, analogous to how variance reduction enables overtrained trees in random forests. Together, these findings translate into practical guidance: with sparsification-based merging, train each expert past the validation optimum rather than early-stopping.

2 Related Work↩︎

Weight-space model merging combines separately fine-tuned models without retraining. Model Soups [1] demonstrated that averaging weights of models fine-tuned with different hyperparameters improves accuracy and robustness. Recent work has proposed merging methods that go beyond simple averaging. Task Arithmetic [2] formalizes task vectors (weight deltas from pre-training) that can be added to compose multi-task models. TIES-Merging [4] addresses interference between task vectors through trimming, sign election, and disjoint merging. DARE [5] randomly drops and rescales delta parameters before merging. The recent MergeBench suite [3] provides a comprehensive evaluation of 8 merging methods across 5 domains, which we adopt for our experiments. Most of these works fix the merging recipe and produce experts through a standard fine-tuning protocol with early stopping or a fixed budget chosen for individual model quality, treating the choice of training duration as orthogonal to the merging step. ATM [10] instead interleaves tuning and merging, reinterpreting task arithmetic as a single noisy gradient step toward a joint objective, which similarly couples the optimization trajectory to the merge. We instead hold the merging methods fixed and vary expert training duration explicitly, and show that this dimension is in fact tightly coupled to the choice of merging method.

A separate line of work studies weight averaging along a single training trajectory. Stochastic Weight Averaging (SWA) [11] averages checkpoints from late training to reach better-generalizing minima, and LAWA [12] extends this to maintain a FIFO queue of recent checkpoints, which yields gains even when individual checkpoints are past validation optimum. Most directly related to our motivation, Post-Hoc Reversal [13] demonstrates validation-optimal single model checkpoints are not the validation-optimal checkpoints for an ensemble of those models, suggesting that selection criteria designed for individual models can be misleading once aggregation is involved. All of these works study single-trajectory temporal averaging within one task. We instead study cross-task merging where experts are trained on disjoint data, and find the stopping rule for the merged model.

Most directly related to our work, [6] also vary expert training duration across merging methods, and find that overtrained experts degrade merging for CLIP and T5 models under full fine-tuning and LoRA, tracing the effect to the memorization of difficult examples that merging discards. Our study examines the same question for substantially larger decoder-only LLMs fine-tuned with quantized low-rank adapters, and finds that the picture is method-dependent in this regime; we discuss the relationship to their findings in Section 4.

DiWA [14] provides a bias-variance-covariance decomposition for weight-averaged models, showing that averaging succeeds when variance dominates the error budget. [15] explicitly analyze model soups through this decomposition. We apply this framework to understand why overfitted experts may benefit sparsification-based model merging methods.

3 Experimental Setup↩︎

We use the Qwen 3.5 model family [16] at three scales: 0.8B, 2B, and 4B total parameters. All experts are fine-tuned with QLoRA [17], i.e.low-rank adapters at rank \(r=16\) on top of a 4-bit quantized base model, targeting all attention and MLP projection modules. We use a constant learning rate of \(2 \times 10^{-4}\) with a 100-step linear warmup, an effective batch size of 8, and the AdamW optimizer. The constant schedule with no decay ensures the learning rate remains active past the optimal validation loss checkpoint, allowing each expert to overfit.

Following the setup of MergeBench [3], we adopt 5 task domains, each with a dedicated fine-tuning dataset and evaluation benchmark. For each dataset, we fine-tune a Qwen 3.5 expert using only the training data from that domain. All individual experts are then merged using all 5 domain LoRA adapters. We aggregate metrics over the 4 domains with accuracy metrics; the Safety domain measures the refuse-to-answer rate (higher is better) and is reported in Section 4.2. For each domain and model size, we identify the validation optimum T* independently, defined as the training step with minimum loss on that domain’s held-out validation split (1K examples). Because T* is selected separately per domain and size, the absolute number of steps it corresponds to varies across experts, so all training durations are expressed as multiples of each expert’s own T* rather than as absolute step counts. We continue training up to 5\(\times\) T* and save LoRA adapters at 8 checkpoints: {0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 3.0, 5.0}\(\times\) T*, spanning from heavily undertrained to extremely overfitted experts.

We evaluate 5 merging methods spanning different families: Simple Averaging [1]: equal-weight linear combination of experts (\(w_i = 1/N\)), Task Arithmetic [2]: scales task vectors by \(\lambda\) before combining with \(\lambda \in \{0.5, 1.0, 1.5\}\) selected via grid search, TIES [4]: trims small-magnitude parameters, resolves sign conflicts, merging values with density \(k=0.5\), DARE+TIES [5]: randomly drops and rescales delta parameters before TIES merging with density \(k=0.5\) and Greedy Soup [1]: iteratively adds experts to the merge, keeping each only if it improves the held-out score of the merged model.

4 Results↩︎

Here we first show results for the average multi-domain accuracy of merged models as a function of training duration, broken down by merging method and model size. We aggregate the quality scores over the four domains: Math, Code, Instruction, Multilingual. Full results are provided in Appendix 7. Then, we proceed to show results on refuse-to-answer(%) on HarmBench across training durations.

4.1 Training duration vs. merged model quality↩︎

The results in Figure 1 reveal a striking method-dependent pattern. Simple Averaging achieves its best performance in the undertrained regime (0.25–0.75\(\times\) T*) and degrades with longer training, losing 14–23 percentage points on average by 5\(\times\) T* (and up to 45 points on individual domains such as Math). In contrast, TIES-Merging peaks well past T* (at 1.5–3\(\times\) T* for 0.8B and 2B, and 3–5\(\times\) T* for 4B), exceeding its T* score by 2.5–8.7 percentage points. DARE+TIES shows a similar but flatter profile, remaining stable across training durations with a slight preference for overfitted experts. Task Arithmetic occupies an intermediate regime: it degrades on some domains (especially Math, where it peaks at 0.25\(\times\) T*) but tolerates moderate overfitting on others. Greedy Soup, which selectively includes experts based on validation improvement, tends to favor undertrained experts for smaller LLM sizes but shows a more balanced selection at 4B. Its curve is notably less smooth than the other methods because the greedy selection includes different subsets of experts at each training duration.

We attribute this discrepancy to how each method handles inter-expert interference. Simple Averaging treats every parameter of every expert with equal weight, so any noisy or specialized parameter in an overtrained expert directly pollutes the merged adapter. Task Arithmetic applies a global scale \(\lambda\) to the task vectors before combining, which partially attenuates overfitting-induced noise but cannot resolve sign conflicts or selectively suppress interfering parameters, placing it between Simple Averaging and sparsification methods. TIES and DARE+TIES, by contrast, include explicit interference-resolution steps: TIES trims small-magnitude updates and resolves sign conflicts, while DARE first stochastically prunes deltas and rescales the survivors. These steps act as a variance-reduction mechanism over the experts: when overfitting introduces idiosyncratic, low-magnitude or sign-conflicting components, sparsification removes them before the average is taken. The result is the same intuition that motivates random forests [7]: deeper, higher-variance individual learners are tolerated when the aggregation step explicitly suppresses the noise they introduce. Greedy Soup occupies a middle ground because its selection criterion is the held-out validation score of the merge, not of individual experts, so it implicitly co-optimizes training duration and merging composition, but only across a small discrete set of choices.

These results partly agree with and partly diverge from [6], who study the same question on smaller CLIP and T5 models. We agree that averaging-style methods (Simple Averaging, Greedy Soup) favor undertrained experts, but diverge for the sparsification-based methods: where they report that TIES and DARE degrade with overtraining, we find that TIES and DARE+TIES improve. This divergence is consistent with their own observation that a higher LoRA rank attenuates the overtraining penalty, since we use a larger rank (\(r=16\) vs.\(r=8\)) on substantially larger decoder-only models with quantized adapters and a constant learning rate.

4.2 Training duration vs. merged model safety↩︎

The results in Table 2 echo the accuracy pattern with one notable shift. Simple Averaging diffuses the safety signal across all parameters and peaks near T* (1\(\times\) for 2B and 4B, 2\(\times\) for 0.8B), leaving the merged model at or below 25% refusal even at 4B. Task Arithmetic preserves more safety than Simple Averaging and shows a clear preference for longer training at larger scales: it peaks at 2\(\times\) for 0.8B (10%) but reaches 30.5% at 3\(\times\) for 2B and 50% at 3\(\times\) for 4B. TIES and DARE+TIES retain the most refusal behavior and, like in the accuracy setting, prefer overtrained experts, with the optimum drifting later as model size grows. DARE+TIES peaks at 1.5\(\times\) for 0.8B and 5\(\times\) for both 2B and 4B. The 4B TIES merge reaches 68.5% refusal at 3\(\times\), more than 2.5\(\times\) higher than Simple Averaging at any duration.

3pt

Greedy Soup behaves similarly to Simple Averaging on safety: peaking near T* (1\(\times\) for 2B and 4B at 12.5% and 23% respectively, 2\(\times\) for 0.8B at 9%) and retains little refusal at extreme durations. This is because greedy selection optimizes for accuracy-based validation scores, which may actively exclude the safety expert when it hurts accuracy, removing the refusal signal. This is consistent with the interpretation that the safety expert produces a localized parameter update (refusal), and sparsification preserves this signal even when other experts are overtrained, so the same recipe which helps accuracy also strengthens safety.

5 Analysis↩︎

5.1 Bias-variance decomposition↩︎

To understand why overfitted experts benefit sparsification-based merging, we apply the bias-variance-covariance decomposition from DiWA [14] to our setting. For each training duration, we decompose the merged model’s expected error into three components: Bias: how far the average expert prediction is from ground truth which decreases with training, Variance: how much individual expert predictions differ which increases with overfitting and Covariance: the correlation of experts’ errors.

Figure 2 shows the decomposition across all three model sizes, which clarifies the mechanism behind the method-dependent pattern. As training progresses past T*, bias drops (experts capture task-specific patterns) while variance rises (experts diverge). Simple averaging passes increased variance and covariance in the merged model, leading to degraded performance. Sparsification methods explicitly filter out small or sign-conflicting parameter values, reducing variance and covariance while preserving the low-bias signal. Sparsification acts as a variance-reduction mechanism analogous to the averaging step in random forests, where the aggregation step suppresses idiosyncrasies of deeper, higher-variance trees.

5.2 Mode connectivity↩︎

We evaluate linear mode connectivity between expert pairs to assess whether overfitted experts remain in the same loss basin. For each pair \((\theta_A, \theta_B)\), we interpolate \(\theta(\alpha) = \alpha \theta_A + (1-\alpha)\theta_B\) for \(\alpha \in [0,1]\) and measure the loss barrier (maximum loss along the path minus the average of the endpoint losses).

Figure 3 shows that the loss barrier increases with training duration, indicating that overfitted experts are progressively less linearly mode-connected. However, the barrier remains moderate (below 0.35) even at 5\(\times\) T*, suggesting that LoRA’s low-rank constraint prevents experts from leaving the pre-trained basin. This explains why weight-space merging remains viable even for heavily overfitted experts: the limiting factor is not basin escape but the merging method’s ability to suppress expert-specific noise within the basin. In other words, LoRA gives us a wide “safe zone” where the mode connectivity breakdown is unlikely.

6 Conclusion↩︎

We conduct a systematic study of how training duration of domain experts affects the quality of merged multi-task LLMs. Our experiments across three model sizes, five domains and five merging methods reveal that the optimal training duration for merging is fundamentally method-dependent. Simple Averaging benefits from undertrained experts (0.25–1\(\times\) T*), Task Arithmetic and Greedy Soup occupy an intermediate regime (0.25–3\(\times\) T* depending on the domain), while sparsification-based methods (TIES, DARE+TIES) achieve peak performance with deliberately overfitted experts (2–5\(\times\) T*). This finding directly challenges the widespread assumption that domain experts should be merged at their individual validation optima.

Through bias-variance decomposition, we show that overfitted experts provide higher diversity (lower bias, higher variance), which benefits methods equipped with interference-resolution mechanisms. The sparsification step in TIES and DARE acts as a variance-reduction mechanism analogous to the aggregation step in random forests, filtering overfitting-induced noise while preserving task-specific knowledge.

Our results have direct practical implications: when using sparsification-based merging methods, practitioners should deliberately train experts past their validation optimum. For the Qwen 3.5 family, training to 2–5\(\times\) T* with TIES or DARE+TIES yields the best merged models across all tested sizes, recovering most of the gap to single-domain experts while retaining all capabilities in a single adapter. Our study has several limitations. We use a single open-weight model family (Qwen 3.5) and a single fine-tuning recipe: QLoRA adapters at a fixed rank (\(r=16\)) on a 4-bit quantized base, trained with a constant learning rate. Each of these is potentially load-bearing: prior work finds that full fine-tuning and LoRA respond differently to training budget [6], and learning-rate decay regulates how sharply a model overtrains [28], so the constant schedule may amplify the regime we study. Our non-deterministic methods (Greedy Soup and DARE’s stochastic pruning) are run with a single seed, and the bias-variance-covariance account in Section 5 is an explanatory hypothesis rather than a proven mechanism. Future work should test whether the pattern persists under full fine-tuning, learning-rate decay, larger base models, and other PEFT methods.

7 Additional Results↩︎

7.1 Main results tables↩︎

Tables 3–5 report per-domain accuracy at T* (standard practice) and the best score across all training durations for each merging method and model size. For every method and size, the best training duration outperforms T*.

4pt

4pt

4pt

7.2 Per-domain breakdown↩︎

Figure 4 decomposes the aggregate trends from Figure 1 by domain and model size. The method-dependent pattern from the main results holds consistently across domains.

Mathematics shows the sharpest divergence between methods: at the 4B scale, Simple Averaging drops from 64.8% at 0.25\(\times\) T* to 20.3% at 5\(\times\) T*, while TIES improves from 60.3% at 0.25\(\times\) T* to a peak of 66.7% at 3\(\times\) T*. Code generation shows similar trends but with smaller gaps, since HumanEval is more saturated. Instruction Following and Multilingual tasks are more stable across methods, suggesting that the sensitivity to training duration scales with the difficulty of the underlying task: tasks where individual experts gain a lot from extra training are also tasks where merging method choice matters most.

7.3 Base model scale dependence↩︎

The interaction between training duration and merging quality varies with model size in a way consistent with our bias-variance results. For Simple Averaging, 0.8B models degrade fastest with overfitting (the merged 0.8B model loses essentially all of its math performance by 3\(\times\) T*), 2B models are intermediate, and 4B models show the most gradual decline (e.g., from 64.8% at 0.25\(\times\) to 20.3% at 5\(\times\) on Math). For TIES, the optimal overfitting level shifts rightward with model size: 0.8B peaks at 1.5–3\(\times\) T*, 2B peaks at 2–3\(\times\) T*, and 4B peaks at 3–5\(\times\) T*. Table 6 summarizes the range of optimal training durations for each method across the four accuracy domains.

3pt

This is consistent with the overparameterization view of fine-tuning. Larger models have more capacity to absorb task-specific knowledge into low-magnitude parameter subsets that are spread across the network, rather than concentrated in a few salient directions. As a result, the additional variance introduced by overtraining is also more diffuse, which makes it easier for sparsification-based methods to filter it out: the noisy directions are still individually small and sign-conflicting after the optimum, just as they need to be for TIES and DARE to discard them. Conversely, in smaller models, additional training pushes a relatively small set of parameters further from the shared basin, and even sparsification cannot fully recover the merged signal once those salient directions disagree across experts. The net effect is that the band of training durations for which sparsification-based merging is helpful both widens and shifts later as model size grows, which provides a useful pattern for practitioners: larger backbones reward more aggressive overtraining of each expert.

7.4 Optimal training duration↩︎

Tables 7–9 report the optimal training duration per domain for each merging method, separately for the 0.8B, 2B, and 4B Qwen3.5 models. They show that the same dichotomy holds across model sizes: averaging-style methods peak undertrained and sparsification-based methods peak overtrained. The Safety column reports the training duration at which each method achieves its highest refuse-to-answer rate.

3pt

3pt

3pt

References↩︎