1 Introduction↩︎

Post-training is the alignment stage, where a broadly capable language model is adapted to produce responses preferred by humans while maintaining or improving performance, especially on reasoning-intensive tasks. Two families dominate current practice. Reinforcement learning with verifiable rewards (RLVR) [1]–[4] drives much of the recent progress on reasoning models [5]–[7], and on-policy distillation (OPD) [8] trains the model on its own trajectories under a stronger teacher and now appears as a core stage in pipelines like Qwen3 and MiMo [9], [10]. Different as the two families look, both follow the same pattern: sample from the model, derive a per-trajectory or per-token learning signal, and update the weights to amplify capabilities that pretraining placed within reach.

A growing body of evidence suggests that what these updates accomplish is mainly to activate capabilities pretraining already made accessible. Chain-of-thought prompting elicits multi-step reasoning from models that were never trained on intermediate traces [11], [12]. RLVR continues to improve a base model even when its reward is partially incorrect or spurious, which motivates the hypothesis that the gradient update may operate beyond direct reward maximization [13]. Mechanistic studies of reasoning fine-tuning reach a similar conclusion. They find that the update repurposes directions that are already active in the base model rather than installing new ones [14]–[16]. Taken together, these results suggest that post-training often exposes and stabilizes capabilities the pretraining step had already placed within reach.

As our baseline, [17] push this view from output space to weight space. The neighborhood of a contemporary pretrained model is dense with task specialists, and a single round of rewarded Gaussian perturbations is enough to expose them. RandOpt evaluates each perturbation on a small support set, keeps the top-\(K\) specialists, and votes their predictions at inference. The resulting ensemble matches or exceeds common post-training methods like PPO and GRPO at equal training compute, with no gradient through the language model. Where output-space recipes sample what the model can say, RandOpt samples what the model could be (see 1).

This is effective at training but expensive at inference. Good performance needs \(K \approx 50\), which means \(50\) forward passes per test example, and the prediction-level vote has to be redesigned for tasks whose outputs are not categorical. The natural fix is to consolidate the rewarded population into a single weight update. Reward-weighted averaging is the obvious first attempt, but rewarded perturbations behave as nearly orthogonal specialists, so naive averaging cancels rather than combines. Existing model-merging tools [18]–[23] assume independently fine-tuned checkpoints. Since random rewarded perturbations around a single base model are not fine-tuned checkpoints, their compatibility must be evaluated directly.

Our study begins with a measurement: Across 25 model-task pairs spanning five families and five tasks, a reward-driven random search around one base model concentrates its useful variance on a reproducible low-rank subspace in every single case we examine. A reproducible mean direction, by contrast, exists in only 40 percent of the cases. These results indicate the presence of shared structure, which is distributed across a family of compatible directions rather than a single broad mean.

Building on this, we introduce Consolidating Rewarded Perturbations (CoRP¹ ), a gradient-free operator that turns a rewarded perturbation population into one deployable update. CoRP first forms a reward-weighted candidate direction. It then reweights each perturbation by its alignment with that direction and its orthogonal dispersion, suppressing high-reward perturbations that would inject incompatible structure. A held-out validation gate commits a candidate only when its gain is stable on examples that did not contribute to its construction [24], [25], and lets the method abstain when no candidate is reliable. Across five language models from \(0.5\)B to \(8\)B parameters and five tasks, CoRP uses \(500\) rewarded perturbations, one tenth of RandOpt’s budget, and recovers more than half of its \(K{=}50\) ensemble gain at one forward pass per test example.

2 Background↩︎

2.1 Rewarded Perturbations↩︎

[17] treat post-training as a problem of sampling and filtering in weight space. Let \(\theta_0 \in \mathbb{R}^d\) denote the pretrained weights of a language model. Given a finite set of noise scales \(\Sigma \subset \mathbb{R}_{>0}\) and a sample size \(N\), RandOpt draws \(N\) Gaussian perturbations of \(\theta_0\), \[\Delta_i \;=\; \sigma_i\, \xi_i, \qquad \xi_i \sim \mathcal{N}(0, I_d), \qquad \sigma_i \in \Sigma, \qquad i = 1, \dots, N, \label{eq:randopt-sample}\tag{1}\] where each \(\sigma_i\) is sampled uniformly from \(\Sigma\) and each \(\xi_i\) is an independent isotropic Gaussian direction in \(\mathbb{R}^d\). RandOpt then scores each perturbed model on a small support set \(D_{\mathrm{sup}}\) of task examples, \[r_i \;=\; \frac{1}{|D_{\mathrm{sup}}|}\sum_{x \in D_{\mathrm{sup}}} R(\theta_0 + \Delta_i;\; x), \label{eq:randopt-reward}\tag{2}\] where \(R(\theta;\,x) \in \mathbb{R}\) is the task reward of the model with weights \(\theta\) on input \(x\). We refer to the resulting set \(\mathcal{P} = \{(\Delta_i, r_i)\}_{i=1}^N\) a rewarded perturbation population. It is the basic object our method operates on.

2.2 Prediction-Level Ensembling↩︎

RandOpt predicts by retaining the top-\(K\) entries of a rewarded population and voting over the corresponding perturbed models, \[E_K = \operatorname{TopK}\!\big(\{r_i\}_{i=1}^N\big), \quad \hat{y}(x') = \operatorname{MajVote}\!\big(\{f(\theta_0 + \Delta_i;\, x') : i \in E_K\}\big), \label{eq:randopt-ensemble}\tag{3}\] where \(f(\theta;\,x)\) denotes the model output at input \(x\) under weights \(\theta\). The ensemble works because the neighborhood of a well-pretrained \(\theta_0\) contains many task-improving perturbations, and because those perturbations behave as specialists whose strengths increasingly diverge as scale grows [17]. The same two facts make a single update hard. It has to preserve what the rewarded population shares while ignoring directions that are individually strong but mutually incompatible.

3 Do Rewarded Perturbations Share a Common Structure?↩︎

A rewarded population is dense and diverse. Whether it also shares a reproducible structure is a separate question, and one the prediction-level ensemble does not need to answer. A pairwise test cannot answer it either. Independent Gaussian directions in a billion-dimensional parameter space are nearly orthogonal whether they are rewarded or not [26], so any shared signal has to be read at the population level. We read it from a split-half diagnostic in the spirit of classical reliability analysis [27], [28].

For each rewarded population we project the perturbations onto a fixed low-dimensional coordinate sketch \(z_i \in \mathbb{R}^m\) with \(m \ll d\). We rank the population by reward, partition the top-\(M\) indices into two disjoint halves \(S_A\) and \(S_B\) of equal size, and repeat the partition over 20 random splits. On each half we form a reward-weighted mean of the sketched perturbations, \(\mu_A = \sum_{i \in S_A} w_i^{(1)} z_i\) and \(\mu_B = \sum_{i \in S_B} w_i^{(1)} z_i\), with the reward-tilted weights \(w_i^{(1)}\) defined in Eq. 7 below. The first statistic compares the two means, \[C_{\mathrm{mean}}(M) \;=\; \mathbb{E}_{\mathrm{split}}\!\big[\cos(\mu_A,\,\mu_B)\big], \label{eq:mean-consensus}\tag{4}\] where the expectation is taken over random splits. The second statistic compares the top-\(r\) principal subspaces of the two weighted clouds, with the random-baseline overlap \(r/m\) between two random \(r\)-dimensional subspaces of \(\mathbb{R}^m\) subtracted away, \[C_{\mathrm{sub\text{-}ex}}(M, r) \;=\; \mathbb{E}_{\mathrm{split}}\!\big[\,\tfrac{1}{r}\,\big\|U_A^\top U_B\big\|_F^2\,\big] \;-\; \tfrac{r}{m}, \label{eq:subspace-excess}\tag{5}\] where \(U_A, U_B \in \mathbb{R}^{m \times r}\) are orthonormal bases of the top-\(r\) principal subspaces of the weighted point clouds on the two halves and \(r\) is fixed at a value reported in the appendix. We call a statistic stable when its nonparametric 95 percent lower confidence bound over the random splits lies above zero.

We run the diagnostic on 25 model-task pairs across five instruction-tuned models, Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, OLMo3-7B-Instruct, and Llama-3.1-8B-Instruct, and across five tasks covering arithmetic search (Countdown), grade-school math (GSM8K), olympiad-style reasoning (OlympiadBench), creative writing (ROCStories), and program synthesis (MBPP). 2 reports the result. The subspace-excess statistic is positive with 95 percent confidence on 25 of 25 pairs. The mean-consensus statistic clears the same bar on only 10. Rewarded perturbations therefore share a reproducible signal in every case we examine, but in most cases that signal lives in a low-rank family of compatible directions rather than in any single mean direction.

Two consequences follow. First, reward-weighted averaging cannot be the answer on its own, because the reward-weighted mean is only reproducible on a minority of pairs. Second, the shared signal nonetheless exists in every pair, distributed across a low-rank family of directions rather than collapsed onto one. A consolidation operator therefore needs to do two things at once: pick a working direction from the rewarded population, and handle perturbations whose strengths lie outside it.

4 Consolidating Rewarded Perturbations↩︎

Reward-weighted averaging is the natural first attempt at consolidation, but it conflates usefulness with compatibility. A perturbation can earn a high reward on the support set and still point in a direction that the rest of the rewarded population does not share. Averaging then writes both the useful component of the perturbation and its incompatible component into the model, and the incompatible component is what cancels.

CoRP separates these two questions and answers them in sequence. A first pass uses reward alone to propose a direction. A second pass then asks which perturbations support the proposal and which add mass orthogonal to it, and reweights accordingly. A held-out validation gate commits the resulting candidate only when it continues to help on examples not used to construct it, and abstains when no candidate is reliable. No gradient flows through the language model at any step.

4.1 From Reward to Compatibility↩︎

Let \(\mathcal{P} = \{(\Delta_i, r_i)\}_{i=1}^N\) be the rewarded perturbation population defined in Eqs. 1 –2 . CoRP builds a single weight update from this population by assigning weights to the sampled perturbations and normalizing the resulting direction.

The first pass retains the top-\(q\) reward quantile, \[E_q \;=\; \{\,i : r_i \ge \tau_q\,\}, \qquad \text{where } \tau_q \text{ is the q-th quantile of } \{r_i\}_{i=1}^N, \label{eq:elite}\tag{6}\] and forms reward-tilted weights and a provisional direction, \[w_i^{(1)} \;=\; \frac{\exp(\beta\, r_i)\,\mathbf{1}[i \in E_q]}{\sum_{j} \exp(\beta\, r_j)\,\mathbf{1}[j \in E_q]}, \qquad m^{(1)} \;=\; \sum_{i=1}^N w_i^{(1)}\, \Delta_i, \label{eq:m1}\tag{7}\] where \(\beta > 0\) is an inverse temperature and \(\mathbf{1}[\cdot]\) is the indicator function. The provisional direction \(m^{(1)} \in \mathbb{R}^d\) is the reward-weighted mean of the elite perturbations. This first pass follows the weighted-sample logic used in reward-weighted regression, advantage-weighted regression, the cross-entropy method, and evolution strategies [29]–[33]. It is not yet a deployable update. Reward selects perturbations that help the task, but it does not say whether those perturbations can be written into the same model.

The second pass measures compatibility with the proposal. For each elite perturbation, CoRP computes its alignment with \(m^{(1)}\) and its orthogonal dispersion against \(m^{(1)}\), \[a_i \;=\; \cos\!\big(\Delta_i,\, m^{(1)}\big), \qquad d_i \;=\; \big\|\Delta_i - \Pi_{m^{(1)}}\Delta_i\big\|^2, \label{eq:align-disp}\tag{8}\] where \(\Pi_{m^{(1)}}\Delta_i\) is the projection of \(\Delta_i\) onto the line spanned by \(m^{(1)}\). The alignment \(a_i \in [-1, 1]\) measures how strongly a perturbation supports the proposal, and the dispersion \(d_i \ge 0\) measures how much of it lies outside. CoRP then combines reward and compatibility into the second-pass weights, \[w_i^{(2)} \;=\; \frac{\exp\!\left(\beta\, r_i + \gamma_a\, z(a_i) - \gamma_d\, z(d_i)\right)\,\mathbf{1}[i \in E_q]}{\sum_{j} \exp\!\left(\beta\, r_j + \gamma_a\, z(a_j) - \gamma_d\, z(d_j)\right)\,\mathbf{1}[j \in E_q]}, \label{eq:w2}\tag{9}\] where \(\gamma_a, \gamma_d \ge 0\) control the strength of the compatibility terms and \(z(\cdot)\) standardizes a statistic to zero mean and unit variance over \(E_q\). The consolidated direction and update are \[\bar{\Delta} \;=\; \sum_{i=1}^N w_i^{(2)}\, \Delta_i, \qquad \hat{\theta} \;=\; \theta_0 + \eta\, \bar{\Delta} \big/ \|\bar{\Delta}\|, \label{eq:CoRP-step}\tag{10}\] with deployed step size \(\eta > 0\). The three terms in the exponent of Eq. 9 capture, in order, usefulness, support for the proposal, and resistance to orthogonal mass. The same principle of measuring compatibility before aggregation appears in parameter-space merging of independently fine-tuned checkpoints [21]–[23]. CoRP applies it to a different source family, random rewarded perturbations around one base model, where compatibility cannot be read from sign, magnitude, or curvature of fitted checkpoints and has to be measured against a population proposal.

4.2 Validating the Consolidated Update↩︎

The same support examples that weight the perturbations can also overstate the value of the resulting update. CoRP separates proposal generation from proposal validation, in the spirit of nested cross-validation and cross-fit estimation [24], [25]. We split \(D_{\mathrm{sup}}\) into three disjoint folds \(A\), \(B\), and \(P\). For each configuration in a small grid of \((q, \beta, \eta)\), CoRP forms a candidate \(\hat{\Delta}_{q,\beta}\) from rewards computed only on \(A\), following Eqs. 6 –10 , and ranks candidates on \(A\) by a constructive score \[s^A(q, \beta, \eta) \;=\; \frac{|F^A(q, \beta, \eta)| - \lambda\, |G^A(q, \beta, \eta)|}{|A|}, \label{eq:constructive}\tag{11}\] where \(F^A\) is the set of examples in \(A\) that candidate \(\theta_0 + \eta\, \hat{\Delta}_{q,\beta}\) answers correctly while base model \(\theta_0\) does not, \(G^A\) is the symmetric set of regressions, and \(\lambda > 0\) penalizes regressions. A candidate passes the gate only if its constructive score on \(B\) is positive and the lower confidence bound of its accuracy change on \(B\) is positive. The probe set \(P\) then calibrates the deployed step size on a fixed multiplier grid \(\{\alpha_j\}\), retaining the largest \(\alpha_j\) such that \(\theta_0 + \alpha_j\, \eta\, \hat{\Delta}_{q,\beta}\) has a positive lower confidence bound on \(\Delta\mathrm{Acc}_P\). If no candidate passes the gate or no multiplier passes the probe, CoRP abstains and returns \(\theta_0\).

4.3 Iterating Around an Accepted Update↩︎

An accepted update changes the local neighborhood, and the rewarded population around the new center can carry useful structure that the original sampling did not reach. CoRP therefore iterates. The candidate construction, the held-out gate, and the probe calibration of the previous two subsections all remain in place. Only the sampling distribution changes between iterations.

Let \(\theta_t\) denote the current center, with \(\theta_0\) the pretrained model and \(\theta_1\) the first accepted update. CoRP draws \(N_{\mathrm{loc}}\) local perturbations \[\Delta_j^{\mathrm{loc}} \;\sim\; \mathcal{N}\!\big(0,\, \rho_t^2\, \Sigma_t\big), \qquad j = 1, \dots, N_{\mathrm{loc}}, \label{eq:local}\tag{12}\] where \(\rho_t > 0\) is the local search scale and \(\Sigma_t\) is a covariance whose low-rank component concentrates exploration in directions where the previous rewarded population varied, with an isotropic floor preserving coverage in directions the population did not emphasize. The exact form of \(\Sigma_t\) follows covariance-adaptive black-box search [34]–[36] and is given in the appendix. The covariance shapes the proposal distribution only and does not appear in the deployed update. The next center \(\theta_{t+1}\) is committed only when the validation rule of §4.2 accepts a proposal on fresh \(A, B, P\) splits, and the iteration stops otherwise.

5 Experiments↩︎

We evaluate CoRP against RandOpt and gradient-based baselines on a shared benchmark and address four questions in turn. How much of the K=50 ensemble gain does a single CoRP model recover, and at what cost. When can a rewarded population actually be consolidated into one update. And what kinds of errors does the operator repair.

Models. We use five instruction-tuned models that span 0.5B to 8B parameters and three pretraining lineages: Qwen2.5-0.5B, Qwen2.5-1.5B, and Qwen2.5-3B [37], OLMo3-7B [38], and Llama-3.1-8B [39].

Tasks. Each model is evaluated on five tasks. Three of them test mathematical reasoning at increasing difficulty, Countdown [40], GSM8K [41], and OlympiadBench [42]. ROCStories tests creative writing [43], and MBPP tests short Python program synthesis [44].

Perturbation population. For each model-task pair we sample \(N{=}500\) rewarded perturbations using the noise-scale mixture \(\Sigma = \{5\mathrm{e}{-}4, 1\mathrm{e}{-}3, 2\mathrm{e}{-}3\}\) from [17], and we follow the RandOpt evaluation protocol exactly. Each perturbation is stored by its random seed and noise scale and regenerated on demand, so the population’s storage cost is independent of model size. The first 200 training examples are partitioned into the support folds \(A\), \(B\), and \(P\) of §4.2, and all results are reported on the original test set.

width=0.9

5.1 Comparison to Standard Post-Training Methods↩︎

r0.4

width=,center

[tab:CoRP95filtered95comparison] compares CoRP with PPO, GRPO, and two RandOpt references. RandOpt \(K{=}1\) keeps the best single perturbation, RandOpt \(K{=}50\) keeps a prediction-level ensemble of the top fifty, and CoRP commits one consolidated update.

Compared with RandOpt \(K{=}1\), CoRP improves 24 of 25 model-task pairs. It reaches or exceeds RandOpt \(K{=}50\) in six pairs, including four ROCStories settings, Qwen2.5-0.5B on Countdown, and OLMo3-7B on OlympiadBench. PPO or GRPO gives the best result in a minority of cells. These results place CoRP between the best single perturbation and the full ensemble. It usually extracts more deployable signal than RandOpt \(K{=}1\), while leaving a gap to \(K{=}50\) in settings where specialists remain hard to compress.

[tab:compute] reports the cost at which each method reaches its average score. RandOpt evaluates each of its \(N\) sampled perturbations on the support set \(D_{\mathrm{sup}}\), costing \(N \cdot |D_{\mathrm{sup}}|\) forward passes through the language model. CoRP shares this construction, and its consolidation, gating, and probe steps run on cached scores at negligible additional cost. We set \(N{=}5000\) for the RandOpt configurations as in [17] and \(N{=}500\) for CoRP, with \(|D_{\mathrm{sup}}|{=}200\) in both. PPO and GRPO are run at total FLOPs matched to RandOpt \(K{=}50\), following the same protocol. CoRP improves the base model by \(8.1\) points while using one tenth of RandOpt’s initial perturbation budget, and runs at the same inference cost as every method except the prediction-level ensemble. RandOpt \(K{=}50\) keeps the highest score, but pays it through fifty forward passes per test example. CoRP recovers much of that gain with one deployed model, which raises the question of what can be consolidated into a single update.

5.2 When Can a Rewarded Population Be Consolidated?↩︎

A prediction-level ensemble can keep rewarded perturbations separate until inference. A consolidated model cannot. It has to write their useful parts into one set of weights, and any incompatible structure that comes along will be written in too.

We make this tension visible by sweeping two perturbations at a time. For rewarded perturbations \(\Delta_i\) and \(\Delta_j\), we evaluate \(\theta_0 + \eta(\alpha \Delta_i + \beta \Delta_j)\) over a grid of mixing coefficients \((\alpha,\beta)\). The resulting accuracy landscape shows which mixtures survive consolidation.

3 shows two characteristic cases on Qwen2.5-3B-Instruct and GSM8K. In panel (a), the two perturbations repair the same probe errors. The high-accuracy region is broad, and equal weighting already lands inside it. This pair behaves like two samples of the same consolidatable component. In panel (b) the repair pattern changes. The two perturbations fix mostly different examples. The high-accuracy region shrinks, and equal weighting falls below the best reweighted point. Complementary perturbations can still be consolidated, but only under more specific weights, and the room for error is smaller. This is the target of CoRP, to preserve the compatible part of the rewarded population, not every specialist that an ensemble can use separately.

5.3 Composition of the Gain↩︎

The same score can hide different behavior. A method may solve examples the base model missed, or it may make an already-correct solution easier for the strict evaluator to parse. It may also improve some examples while regressing on others. Following [17], we decompose the test-set outcome into four categories. A retained correct example is strictly correct for the base model and remains correct after adaptation. A format fix is one where the base produces the right answer content but fails the strict format and the adapted model becomes strictly correct. A reasoning fix is one where the base fails on the answer content and the adapted model becomes strictly correct. A regression is strictly correct for the base and incorrect after adaptation. Precise definitions and the format-vs-answer distinction per task are in the appendix.

4 shows that CoRP’s gain is not only format repair. GSM8K does carry a visible format component for the stronger base models, but Countdown and OlympiadBench show clear reasoning fixes, where the base fails on the answer content and CoRP makes the example strictly correct. This matters because CoRP performs one inference with one updated model, rather than relying on a vote across perturbed models.

The same panels also show why consolidating several rewarded perturbations is different from selecting one. RandOpt \(K{=}1\) introduces visible regressions, especially on Countdown and OlympiadBench. CoRP keeps the single-model deployment of \(K{=}1\), but its regression segments are smaller in most panels. RandOpt \(K{=}50\) can vote regressions away at inference. CoRP cannot. Reducing them before deployment is a central part of consolidation, and the held-out gate of §4.2 is what does it.

5.3.0.1 Robustness checks.

Each component of CoRP carries weight, and full CoRP improves on naive baselines including reward-weighted averaging and direct top-\(r\) subspace projection. The choice of compatibility weights \(\gamma_a, \gamma_d\) is also not fragile in a sweep over \(\{0.1, 0.5, 1, 2, 5\}^2\) on Qwen2.5-3B / GSM8K. We report ablations, naive baselines, and the sensitivity sweep in Appendix §8.

6 Related Work↩︎

Search in weight space. CoRP is closest in spirit to a line of self-elicitation post-training methods that sample from the model and consolidate the exposure back into the weights, including RLVR and on-policy distillation. Within this lineage, a body of work treats LLM post-training as a search problem in weight space rather than as gradient descent on a loss. Evolution strategies and the cross-entropy method use reward-weighted samples to update a sampling distribution over weights [29]–[33], with covariance-adaptive variants concentrating exploration in informative directions [34]–[36]. RandOpt [17] pushes this idea to its extreme by removing iterative search entirely. A single round of rewarded Gaussian perturbations with prediction-level ensembling already matches PPO and GRPO on contemporary models. CoRP shares RandOpt’s reliance on a single rewarded sampling stage as the primary source of perturbations, but consolidates the rewarded population into one deployable update rather than carrying it forward as an ensemble.

Parameter-space model merging. A second relevant line aggregates several models in parameter space. Model soups average independently fine-tuned checkpoints that share a low-loss basin [18], [19]. When the sources disagree, geometry-aware corrections become necessary. TIES-merging trims small magnitudes and resolves sign conflicts [21], DARE drops and rescales parameters to cut interference [22], Fisher-weighted merging weights sources by local curvature [23], and task arithmetic operates on task vectors anchored to a shared base [20]. CoRP applies the same principle to a different source family. Its sources are random rewarded perturbations around one pretrained model rather than independently fine-tuned checkpoints, so compatibility is measured by alignment and dispersion with respect to a provisional reward-weighted mean rather than by sign, magnitude, or curvature.

Low-dimensional structure in LLM adaptation. The stable low-rank signal we measure in rewarded perturbations aligns with a larger body of work on low-dimensional LLM adaptation. Intrinsic-dimension analyses show that fine-tuning succeeds within a small random subspace [45]. LoRA [46] and its variants restrict updates to low-rank components without sacrificing accuracy. [47] attribute the success of LLM adaptation to a low-dimensional curvature geometry, and [48] show that a mathematical reasoning task can be learned by updating thirteen parameters. Our characterization adds a complementary observation at a different granularity, namely that reward-driven random search around a pretrained model concentrates its useful variance on a consistent low-rank subspace of a coordinate sketch space.

7 Discussion↩︎

Recent post-training methods finish a loop. They sample from the model, expose capabilities the pretraining step has already placed within reach, and write the exposure back into the weights. RandOpt opens this loop in weight space but stops at a prediction-level ensemble. We close it. A reproducible low-rank structure exists in every rewarded perturbation population we examine, and a compatibility-aware operator can read it into one deployable update without any gradient through the language model. Across five base models and five tasks, this update recovers more than half of the ensemble’s gain at one tenth of the perturbation budget and one forward pass at inference. The rewarded neighborhood of a well-pretrained model carries enough shared structure to be folded into one model, even when no single direction summarizes it on its own.

7.0.0.1 Limitations & Future Work.

CoRP requires a reward or verifier on a small support set, the same prerequisite as RandOpt and RLVR. The benchmark in this paper covers tasks with categorical or easily verifiable outputs, and extending compatibility-aware consolidation to genuinely free-form generation is open. CoRP’s gap to the prediction-level ensemble is also widest on the smaller base models, where §5.2 suggests rewarded perturbations behave as more disjoint specialists. A more capable operator would need to absorb such complementary specialists, not just aligned ones.

The most natural next step is to combine the weight-space sampling that drives CoRP with the trajectory-space sampling that drives RLVR and on-policy distillation. The two source families likely expose different parts of what pretraining made accessible, and there is no a priori reason a post-training method should be confined to one of them.

8 Additional Empirical Results↩︎

This section reports four additional empirical studies that complement the main results: a component ablation of CoRP on representative cells, a comparison to naive and merging baselines on the same cells, a sensitivity sweep of the compatibility weights, and an analysis of the effect of perturbation population size.

8.1 Component Ablation↩︎

We isolate the contribution of each CoRP component on four cells covering two model scales and two task types. Five variants share the rewarded population and the support folds with full CoRP, and differ only in which component is removed. The variants are reward only (\(\gamma_a{=}\gamma_d{=}0\), a single reward-weighted average), reward+alignment (\(\gamma_d{=}0\)), reward+dispersion (\(\gamma_a{=}0\)), no gate (deploys the best-on-\(A\) candidate without the validation of §4.2), and no iteration (skips the recentering of §4.3). Table 1 reports test accuracy and the change relative to the base model.

Each component contributes. The iteration step helps most on Qwen2.5-3B / ROCStories, where Full CoRP improves over the no-iteration variant by 2.40 points. On the other three cells, the accepted first-pass update already captures most of the measured gain. The held-out gate plays a complementary role by rejecting harmful candidates on cells where rewarded perturbations are less aligned. Dispersion is the weakest stand-alone choice, which matches its role in Eq. 9: it suppresses incompatible mass against a direction the alignment term has already proposed, and is informative only when paired with that direction.

8.2 Naive Baselines↩︎

The reproducible low-rank structure of §3 raises a natural concern: simpler operators that read this structure directly might already suffice. Existing parameter-space merging tools, designed for fine-tuned checkpoints, raise a separate concern about whether they transfer to rewarded perturbations. We compare CoRP to three alternatives that probe both concerns on the same four cells used in §8.1.

Reward-weighted averaging is the operator obtained by setting \(\gamma_a{=}\gamma_d{=}0\) in Eq. 9 , identical to the reward-only variant of 1. Top-\(r\) subspace projection estimates the top-\(r\) principal subspace of the elite perturbations with \(r{=}8\) and deploys \(\theta_0 + \eta\, \Pi_U \bar{\Delta}\), where \(\Pi_U\) projects onto the estimated subspace. Sparse merging in the spirit of TIES [21] selects the top half of the rewarded population, sparsifies each perturbation by retaining the top-density fraction of coordinates by magnitude, and merges the sparsified vectors. We sweep the sparsity density and deployment scale on the probe fold for each cell. All three baselines share the rewarded population and the support folds with full CoRP. 2 reports the result.

Reward-weighted averaging is the most competitive of the three baselines, but still trails CoRP across the four cells. Reward identifies useful perturbations, but compatibility against the proposal determines which of those perturbations can share a single deployable update.

Top-\(r\) projection rarely moves beyond the base. The shared low-rank structure of §3 is a necessary condition for consolidation to be possible, but the population-level subspace estimate discards the per-perturbation reward signal that the operator of §4.1 actually uses, and the estimate is itself noisy at the population sizes we use.

Sparse merging in the spirit of TIES underperforms the base on three of four cells. Standard merging operators were designed for independently fine-tuned checkpoints whose disagreements carry meaningful sign and magnitude structure. The same operators applied to random rewarded perturbations have no comparable signal to read, since coordinate sign is essentially random and magnitude varies with the noise scale rather than with task relevance.

8.3 Sensitivity to Compatibility Weights↩︎

The default configuration of CoRP fixes the alignment and dispersion weights at \(\gamma_a{=}\gamma_d{=}1\) across all \(25\) model-task pairs. To check that this choice is not fragile, we sweep both weights over \(\{0.1, 0.5, 1, 2, 5\}\) on Qwen2.5-3B-Instruct / GSM8K, holding the rewarded population, the support folds, and the candidate grid over \((q, \beta, \alpha)\) identical to the main run. Each of the \(25\) configurations rebuilds the consolidated update of §4.1 from the same perturbations and is evaluated on the GSM8K test set. The sweep uses the first-pass operator only, without the held-out gate of §4.2 or the iteration of §4.3, so that the variation in the table reflects only the choice of \((\gamma_a, \gamma_d)\).

The default \((1, 1)\) is the strongest configuration on the sweep, and the central region \([0.5,2]^2\) remains mostly positive, with small negative outliers at a few asymmetric settings. The accuracy surface drops at the corners where one weight saturates the exponent in Eq. 9 and the other becomes negligible, which is the regime in which compatibility scoring effectively reduces to one of the two terms. Within the operating range we use, the operator is not sensitive to the precise value of either weight.

8.4 Effect of Perturbation Population Size↩︎

CoRP uses \(N{=}500\) rewarded perturbations throughout the main results, one tenth of the \(5000\) used by RandOpt. To check that this choice is not arbitrary, we sweep \(N \in \{100, 250, 500, 1000, 5000\}\) on Qwen2.5-3B-Instruct, holding the noise-scale mixture, the support folds, and the candidate grid identical to the main run. Each configuration runs the first-pass operator of §4.1, without the iteration of §4.3, so that the variation in 6 reflects only the population size.

The marginal return from a larger population diminishes well before \(N{=}5000\) on both tasks. On ROCStories, the gain plateaus between \(N{=}1000\) and \(N{=}5000\). On GSM8K, the gain peaks at \(N=500\) in this sweep and remains positive but non-monotone for larger populations. The default \(N{=}500\) recovers a substantial fraction of the maximum gain on both tasks at one tenth of the population size used by RandOpt.

9 Additional Analysis on Consolidation Behavior↩︎

We provide two further analyses of consolidation behavior: how CoRP composes its test-set outcomes relative to ensemble baselines, and how strict vs.relaxed evaluation interacts with each method.

9.1 Error-Composition Decomposition↩︎

7 reveals a structured trade-off across the three deployment modes. Relative to \(K{=}1\), CoRP reduces the average regression fraction while preserving a comparable reasoning-thicket fraction, indicating that compatibility-aware reweighting and held-out gating mitigate harmful updates without sacrificing reasoning gains. The 50-pass vote attains the lowest regression level by retaining inference-time diversity, but at substantially higher inference cost. CoRP therefore lands between the two on the regression-vs-inference-cost frontier, achieving the regression reduction of an ensemble at the inference cost of a single model.

9.2 Robustness to Strict Evaluation↩︎

8 reports how each method’s gain holds up under the strict task evaluator used throughout the paper. The 50-pass vote has the most negative average strict-minus-relaxed gap, suggesting that part of its reward-level gain is attenuated under strict parsing. CoRP remains closer to the single-inference baseline on this axis, indicating that consolidation does not systematically amplify strict-format fragility. For deployment with strict extractors this distinction is practically important, because the gap between an ensemble’s relaxed score and its deployed strict score is exactly what a single deployable model has to recover.

10 Implementation Details↩︎

We implement CoRP in PyTorch [49] and all experiments were conducted on 4 NVIDIA A100 GPUs with 80GB memory.

10.1 Support Splits and Model Selection↩︎

For each model-task pair, we partition the support set into three disjoint folds: a construction fold \(A\), a validation fold \(B\), and a probe fold \(P\). Fold \(A\) scores perturbations and generates CoRP candidates. Fold \(B\) gates candidates and selects among the \((q,\beta)\) grid. Fold \(P\) is used only after a candidate passes the gate, to choose the final step-size multiplier \(\alpha\) from the fixed grid in 3. The test set plays no role in selecting \(q\), \(\beta\), or \(\alpha\), nor in deciding whether to accept an update.

This design separates proposal construction from validation. If no candidate achieves a positive construction score and a positive lower-confidence-bound improvement on fold \(B\) (defined in §10.5), CoRP abstains and returns the base model. If a candidate passes the gate but no step-size multiplier passes the probe check on \(P\), CoRP also abstains.

10.2 Baseline Protocols↩︎

We compare CoRP against PPO, GRPO, RandOpt \((K=1)\), and RandOpt \((K=50)\) under the protocol of [17]. RandOpt \((K=1)\) selects the single highest-reward perturbation from the sampled population; RandOpt \((K=50)\) ensembles the top 50 perturbations by majority vote at inference. RandOpt uses \(N=5000\) perturbations and CoRP uses \(N=500\). For PPO and GRPO, we follow the matched-compute settings reported by [17]: total training FLOPs are matched to those consumed by sampling and scoring the RandOpt \(K{=}50\) population, accounting for actor forward and backward passes, KL evaluation against the reference model, and rollout sequence lengths. We use the same models, tasks, prompts, reward functions, and evaluation scripts throughout. No baseline hyperparameters are tuned on the test set.

10.3 Statistical Reporting↩︎

The main result table reports mean\(\pm\)standard deviation over \(R=3\) independent runs. For CoRP and RandOpt, each run resamples the perturbation population and repeats the full selection procedure; for PPO and GRPO, each run uses an independent training seed under the same matched-compute setting. Error bars summarize run-to-run variability and are not intended as formal pairwise hypothesis tests.

For the split-half diagnostics of §3, we repeat the random partition of the top-\(M\) rewarded perturbations \(20\) times and report a nonparametric \(95\%\) lower confidence bound via bootstrapping, taking the \(5\)th percentile of the bootstrap distribution of the mean. This procedure applies to both the mean-consensus statistic and the subspace-excess statistic, and is distinct from the gate-and-probe lower confidence bounds defined in §10.5.

10.4 Hyperparameter Settings↩︎

Table 3 lists the CoRP hyperparameters. Values are fixed across all \(25\) model-task pairs unless noted.

10.5 Operational Definitions↩︎

We give precise definitions for three quantities referenced in the main text: the coordinate sketch used by the split-half diagnostic, the proposal covariance \(\Sigma_t\) used by the iteration step, and the lower confidence bounds used by the validation gate and probe.

10.5.0.1 Coordinate sketch.

The split-half diagnostic of §3 reads each statistic from a fixed low-dimensional sketch rather than from \(\mathbb{R}^d\) directly. For each perturbable parameter tensor in the model, we retain a small fixed number of coordinates from the flattened perturbation, and concatenate the retained coordinates across all tensors to form \(z_i \in \mathbb{R}^m\) with \(m\) on the order of a few thousand. The same coordinate selection is shared across all candidates in the population and across all \(20\) random splits. We use the same sketch when comparing reward-weighted means and when computing principal subspaces, normalizing \(z_i\) by its noise scale \(\sigma_i\) to compare directions rather than magnitudes.

10.5.0.2 Proposal covariance.

The iteration step of §4.3 draws local perturbations \(\Delta_j^{\mathrm{loc}} \sim \mathcal{N}(0, \rho_t^2 \Sigma_t)\), where the covariance \(\Sigma_t\) combines a low-rank component fit to the previous accepted population with an isotropic floor: \[\Sigma_t \;=\; \lambda_{\mathrm{iso}}\, I_d \;+\; (1-\lambda_{\mathrm{iso}})\, U_t \Lambda_t U_t^\top, \label{eq:sigmat-appendix}\tag{13}\] where \((U_t, \Lambda_t)\) is the rank-\(r\) truncated eigendecomposition of the compatibility-weighted second-moment matrix of the previous round’s elite perturbations, and the rank \(r\) and floor \(\lambda_{\mathrm{iso}}\) are reported in 3. The covariance shapes the local proposal distribution only and does not appear in the deployed update.

10.5.0.3 Lower confidence bounds.

The gate and probe of §4.2 commit a candidate only when its estimated accuracy change has a positive lower confidence bound on the corresponding fold. For each candidate, we compute paired per-example correctness differences against the base model and use \[\widehat{\mathrm{LCB}} = \widehat{\Delta\mathrm{Acc}} - z_{\mathrm{LCB}}\, \widehat{\mathrm{SE}}(\Delta\mathrm{Acc}).\] where \(z_{\mathrm{LCB}}\) is given in 3. This bound is computed separately for each candidate and fold, and is distinct from the bootstrap procedure used for the split-half diagnostics in §3.

10.6 Prompts↩︎

Following [17], we set up the prompts for different datasets in our experiments following EvalScope [50] and Verl [51].

11 Additional Acknowledgment↩︎

The authors acknowledge the use of ChatGPT exclusively to refine the text in the final manuscript.

References↩︎

---

1. Our code is available at https://github.com/oooranz/CoRP.↩︎