1 Introduction↩︎

Large language model (LLM) post-training is usually described as a pipeline: collect instruction data, perform supervised fine-tuning, and then apply preference optimization or reinforcement learning to align the model with human preferences [1]–[5]. This framing encourages a useful but misleading division of labor. SFT teaches the model to follow instructions; preference optimization aligns the model. In practice, the division is blurred. The SFT corpus already contains implicit choices about what a good assistant should do: answer directly or cautiously, refuse broadly or narrowly, explain in detail or stay concise, flatter the user or challenge false premises. Instruction-tuning studies already show that small changes in demonstrations can produce large changes in assistant behavior [6]–[10].

Online data selection makes these choices sharper. As instruction pools grow large, redundant, noisy, and heterogeneous, a common practice is to score examples as training proceeds and keep only a high-value subset, using loss, quality, diversity, influence, or online utility signals [11]–[18]. Such online selectors are usually evaluated by the same question: how much current-task performance can be recovered with fewer tokens? We ask a different question: which behavioral preferences are silently induced by the online-selected subset?

The issue is not hypothetical. A loss-based online selector may over-sample examples that the current model finds surprising, including ambiguous safety cases, uncommon refusal formats, or long-form explanations. A quality-based selector may prefer polished assistant messages, which can also be verbose, highly confident, and stylistically homogeneous. A diversity-based selector may preserve topical coverage while still changing the distribution of refusal language. Online utility- or influence-based selectors may improve data efficiency while repeatedly emphasizing examples whose scores indicate strong immediate learning signal [15], [16]. None of these policies explicitly optimizes helpfulness, harmlessness, truthfulness, sycophancy, or jailbreak robustness, yet all can move those behaviors [19]–[21].

We call this phenomenon implicit alignment by data selection. The term does not imply that selection always improves alignment. It means that selection induces a directional behavioral change along axes normally associated with alignment. The same token budget can produce a more helpful, more evasive, more sycophantic, more verbose, or more jailbreak-prone model depending only on which demonstrations are retained. Conceptually, this reframes an online selector as an implicit preference model: its scoring rule occupies the position normally held by a reward model, steering the assistant’s persona before any explicit alignment step ever runs.

This paper proposes a controlled way to study the phenomenon. Starting from the same base model, we run SFT on subsets chosen by representative online selection families: random, loss-based, quality-based, and diversity-based selection. We then evaluate the resulting models on a behavioral suite covering helpfulness, safety refusal, truthfulness, sycophancy, jailbreak robustness, verbosity, calibration, and style drift. In parallel, we audit the selected data itself to ask whether high-scoring or high-quality examples systematically over-represent certain response modes.

Our contributions are four-fold.

• We formulate online SFT data selection as implicit alignment: an online selector reweights the supervised objective and thereby defines an implicit preference over assistant behaviors, with a first-order bound linking behavioral drift to the attribute enrichment of the selected data.

• We introduce Alignment Drift Auditing (ADA), a protocol that compares online selection methods under equal data budgets using behavioral metrics and data-mixture diagnostics.

• Empirically, we show that selectors matched on task accuracy induce large, structured, and reproducible behavioral drift, and that the direction of the drift is predicted by which attributes the selector over-represents.

• We propose Alignment-Aware Selection (AAS), a diagnostic online selector that keeps data-efficiency objectives while constraining drift on refusal, verbosity, truthfulness, and sycophancy axes.

2 Related Work↩︎

2.0.0.1 Online data selection for LLM fine-tuning.

Online data selection has become a practical route to cheaper and sometimes better instruction tuning. Prior work studies quality filtering, small high-quality datasets, data valuation, influence estimation, and online batch selection that scores examples as training proceeds [11]–[16]. Earlier data valuation and coreset methods also show that subset choice can change the learned solution even under identical architectures and optimizers [22]–[25], and recent studies find that these choices leave lasting downstream effects across sequential fine-tuning stages [26]. Our focus is complementary: rather than asking whether an online selector improves training efficiency, we ask which alignment-relevant behaviors change as a side effect of selecting online.

2.0.0.2 Alignment and preference optimization.

LLM alignment is commonly associated with reinforcement learning from human feedback, constitutional training, rejection sampling, or direct preference optimization [1]–[3], [27]–[29]. These methods use explicit preference comparisons or reward models. SFT is nevertheless a strong behavioral intervention: instruction demonstrations define the canonical form of assistant behavior before preference optimization begins. Recent analyses of reward-model overoptimization, preference data, and alignment taxonomies further suggest that the data distribution and feedback channel are inseparable from the learned behavior [30]–[32]. Our work isolates this earlier step by removing preference training and measuring alignment drift caused solely by data selection.

2.0.0.3 Safety, truthfulness, and sycophancy evaluation.

Behavioral evaluation has expanded beyond task accuracy to include truthfulness, harmful request handling, jailbreak robustness, toxicity, bias, and sycophancy [33]–[38]. Related work on hallucination, calibration, and social bias shows that fluent answers can still be unreliable or socially distorted [39]–[42]. These benchmarks show that models can be helpful in standard instruction-following settings while failing under adversarial or socially loaded prompts. Because exhaustive human labeling of every axis is costly, our audit also draws on automated, reference-free evaluators that score model outputs directly [38], [43]. We use this literature to define alignment axes for SFT-only drift analysis.

2.0.0.4 Efficient adaptation and interference.

Parameter-efficient fine-tuning methods such as adapters, LoRA, and QLoRA make repeated SFT experiments affordable [44]–[46]. A growing line augments low-rank adaptation with mixture-of-experts or asymmetric structure to add multi-domain capacity without losing efficiency, from LoRAMoE and MixLoRA to mixtures of LoRA experts, asymmetric adapters, domain-specialized MoE-LoRA frameworks, and near-orthogonal rank-wise experts [47]–[52]. In parallel, continual-learning and representation-interference studies show that fine-tuning data can alter future behavior and plasticity even under a purely supervised objective [53], [54]. Recent work traces this to representation health, plasticity loss, and feature decorrelation during continual adaptation [55]–[57], as well as representation collapse across sequential post-training [58] and orthogonal or decoupled subspaces that curb cross-task interference [59]–[61]. We borrow this intervention view: an online selector is not only a filter for efficiency, but also a mechanism that changes the model state and behavioral surface.

3 Problem Setup and Selection Policies↩︎

Let \(M_0\) be a pretrained or instruction-tuned language model and let \(\mathcal{P}=\{(x_i,y_i)\}_{i=1}^{N}\) be a candidate SFT pool. An online selection policy \(\pi\) scores candidate examples as training proceeds, observing the evolving model, optional metadata, and a budget \(B\), and keeps a subset \(S_{\pi}\subset\mathcal{P}\) to train on. Fine-tuning \(M_0\) on the online-selected stream yields \(M_{\pi}\).

The standard efficiency view evaluates \(\pi\) by validation loss or task performance under a fixed budget: \[U_{\mathrm{task}}(\pi) = \mathrm{Perf}(M_{\pi};\mathcal{V}_{\mathrm{task}}).\] This ignores the behavioral axes that are often corrected later by preference optimization. We instead define a behavior vector \[\mathbf{b}(M) = [ h, r_{\mathrm{harm}}, r_{\mathrm{benign}}, v, t, s, j, c ],\] where \(h\) is helpfulness, \(r_{\mathrm{harm}}\) is refusal rate on harmful requests, \(r_{\mathrm{benign}}\) is over-refusal on benign requests, \(v\) is verbosity, \(t\) is truthfulness, \(s\) is sycophancy, \(j\) is jailbreak robustness, and \(c\) is calibration. The exact estimators are described in Section 4.

3.1 Selection as Reweighted SFT↩︎

Most deterministic or stochastic online selectors can be written as a weighted empirical objective: \[\begin{align} \mathcal{L}_{\pi}(\theta) &= \sum_{i=1}^{N} w_{\pi}(i)\, \ell_{\theta}(x_i,y_i), \\ w_{\pi}(i)&\ge 0, \quad \sum_i w_{\pi}(i)=1, \end{align} \label{eq:weighted95sft}\tag{1}\] where \(w_{\pi}(i)\) is zero for rejected examples and, for stochastic online selectors, equals the expected number of times example \(i\) survives scoring, normalized to a probability simplex. It is convenient to read Eq. 1 at the distribution level. Let \(p(x,y)=1/N\) be the empirical pool distribution; the selector induces a tilted training distribution \[q_{\pi}(x_i,y_i) = w_{\pi}(i) = p(x_i,y_i)\,\underbrace{r_{\pi}(x_i,y_i)}_{\text{density ratio}}, \label{eq:density95ratio}\tag{2}\] so that online selection is exactly importance reweighting of the pool with ratio \(r_{\pi}=Nw_{\pi}\). The population objective \(\mathbb{E}_{q_{\pi}}[\ell_\theta]\) is therefore a maximum-likelihood fit to \(q_{\pi}\), and in the realizable limit the SFT optimum is the information projection \[\theta_{\pi}^{\star} = \arg\min_{\theta}\; \mathrm{KL}\!\left(q_{\pi}(y\mid x)\,\big\|\,p_{\theta}(y\mid x)\right), \label{eq:iproj}\tag{3}\] where the conditional \(q_{\pi}(y\mid x)\) is the selector-tilted demonstration policy. Two selectors that disagree about which demonstrations to keep thus target different conditional response distributions, even when they share the pool, the model family, and the optimizer.

This is where behavior enters. Suppose each example carries a (latent) attribute vector \(a_i\in\mathbb{R}^{m}\) encoding refusal style, answer length, hedging, directness, and agreement with the user premise. The selector shifts the expected attribute mixture from the pool mean \(\mu_a(\mathcal{P})=\tfrac1N\sum_i a_i\) to \[\begin{align} \mu_a(S_{\pi}) &= \sum_i w_{\pi}(i)\,a_i \\ &= \mu_a(\mathcal{P})+\sum_i\big(w_{\pi}(i)-\tfrac1N\big)a_i . \end{align} \label{eq:attr95shift}\tag{4}\] The second term is the selection-induced attribute shift; it is nonzero whenever the selector’s weights correlate with any behavioral attribute, and it is exactly what the enrichment ratios in Section 4 estimate. The model is therefore trained not on the original instruction distribution but on a behavioral reweighting of it, and Eq. 4 is the bridge between the data-mixture diagnostics and the behavioral metrics.

The proposition makes precise why task performance under-determines behavior: validation loss constrains \(\theta_{\pi}^{\star}\) on a task slice, but not the projection \(J_\theta\bar g_{\pi}\) along alignment axes. If gradients cluster by attribute, \(\bar g_{\pi}\approx\sum_k \mu_a^{(k)}(S_{\pi})\,\bar g^{(k)}\) where \(\bar g^{(k)}\) is the mean gradient of examples with attribute value \(k\), and Eq. ?? couples drift directly to the enrichment shift of Eq. 4 . A selector can thus be task-efficient because it favors high-loss, high-quality, or high-utility examples, while its weighted gradients still point toward a particular refusal policy or response style. We make the cluster assumption and the resulting bound precise in Appendix 8.

3.2 Selection Policies↩︎

We compare representative online selection families under the same selected-token budget. Each selector scores candidate examples during training and keeps a high-value subset under the budget. The goal is not to claim that one implementation is universally best, but to expose how common online selection principles move behavior. The first four policies below are baselines that span the loss, quality, diversity, and utility signals used by modern online selectors [12], [15], [16], [62]; the last is our diagnostic selector.

3.2.0.1 Random selection.

Random selection is the reference condition. It preserves the expected mixture of the candidate pool and separates selection-induced drift from ordinary SFT drift.

3.2.0.2 Loss-based selection.

The loss selector chooses examples with high teacher-forced loss under the current model, optionally excluding the extreme tail to reduce label-noise sensitivity. It approximates online batch selection and hard-example mining [62]–[64]. We test whether high-loss SFT data increases helpfulness at the cost of verbosity, over-refusal, or brittleness.

3.2.0.3 Quality-based selection.

The quality selector ranks examples using an external judge, reward model, or heuristic quality score, such as clarity, completeness, grammar, and policy compliance [12], [15]. This captures common curation pipelines. We audit whether high-quality data is behaviorally neutral or whether it selects for polished but homogeneous assistant personas.

3.2.0.4 Diversity-based selection.

The diversity selector uses embedding coverage, clustering, or farthest-first traversal to cover semantic regions of the pool [65]–[67]. It tests whether semantic coverage is sufficient to preserve alignment-relevant coverage, such as benign refusals, concise answers, calibrated uncertainty, and disagreement with false user assumptions. The quality and diversity scorers subsume the utility-style online selectors that keep the most informative examples during SFT [15], [16], which we therefore treat as instances of the same score family rather than as a separate baseline.

3.2.0.5 Alignment-aware selection.

Alignment-Aware Selection (AAS) is a diagnostic variant, not the central claim of the paper. Given a base online score \(s_{\mathrm{base}}(x)\) from any loss- or quality-based utility scorer, AAS adds a penalty on the selected behavioral mixture: \[\begin{align} \max_{S:|S|\le B}\quad& \sum_{x_i\in S}s_{\mathrm{base}}(x_i) +\lambda c(S) \\ &- \rho \left\| \hat{\mu}_{a}(S)-\hat{\mu}_{a}(\mathcal{P}) \right\|_2^2, \end{align} \label{eq:aas}\tag{5}\] where \(\hat{\mu}_{a}(S)=\tfrac{1}{|S|}\sum_{x_i\in S}\phi(a_i)\) is the empirical attribute mean of the selected set under a feature map \(\phi\). The penalty is precisely a (linear-kernel) maximum mean discrepancy between the selected set and the pool, \(\rho\,\mathrm{MMD}^2_{\phi}(S,\mathcal{P})\), so AAS interpolates between pure utility selection (\(\rho=0\)) and attribute-matched selection (\(\rho\to\infty\)), at which point \(\mu_a(S)\!\to\!\mu_a(\mathcal{P})\) and, by Eq. 4 , the first-order attribute shift vanishes. Attributes can be obtained from lightweight classifiers, LLM judges, or rules for length and refusal markers.

Although the cardinality-constrained problem is combinatorial, the objective is the difference of a modular term and a squared-norm term and admits a simple greedy rule. Writing \(\delta(x\mid S)\) for the marginal gain of adding \(x\) to a partial set \(S\), \[\begin{align} \delta(x\mid S) ={}& s_{\mathrm{base}}(x) +\lambda\,\Delta c(x\mid S) \\ &-2\rho\, \big\langle \hat{\mu}_{a}(S)-\hat{\mu}_{a}(\mathcal{P}),\, \phi(a_x) \big\rangle, \end{align} \label{eq:aas95marginal}\tag{6}\] up to an \(O(1/|S|)\) self-term. Each step adds the unselected example with the largest \(\delta(x\mid S)\); the inner product makes the rule self-correcting, down-weighting attributes already over-represented in \(S\) and up-weighting under-represented ones. When \(c\) is monotone submodular, the modular-plus-coverage part inherits the standard \((1-1/e)\) guarantee, and the penalty only tightens the attribute match. AAS thus asks whether drift can be reduced without abandoning the efficiency of the underlying online selector, and Eq. 6 shows it adds only an inner product per candidate over the base scorer.

4 Alignment Drift Auditing↩︎

ADA is designed to isolate the causal role of selection. All runs share the same base model, optimizer, learning rate, sequence length, selected-token budget, number of updates, and decoding settings. Only the selected subset changes.

4.1 Data and Models↩︎

Our candidate pool combines general instruction following, reasoning, coding, safety, factual QA, and advice-seeking prompts, drawn from open instruction mixtures (UltraChat and OpenHermes-style conversations), math and code instruction data, and safety-oriented refusal data, for a pool of roughly \(300\)K instruction–response pairs. The base model is a Llama-3.1-8B instruction-tuned checkpoint. We fine-tune with LoRA (rank \(16\), \(\alpha{=}32\)) under a single fixed recipe—identical learning rate, schedule, sequence length, and number of optimizer updates across selectors—and confirm the main trends with a full-fine-tuning run to rule out an adapter artifact [45], [46].

Each selector is run over three seeds at budgets of \(1\%\), \(5\%\), \(10\%\), and \(25\%\) of the pool measured in tokens. The budget sweep is central because implicit alignment is strongest in the low-budget regime, where selection pressure over the empirical SFT distribution is highest.

4.2 Behavioral Metrics↩︎

ADA evaluates selectors by behavioral movement rather than validation loss alone. We track eight axes, each with a concrete estimator: helpfulness via pairwise LLM-judge win rate and instruction-following score; harmful refusal via refusal rate on harmful requests and HarmBench-style attack success [38]; benign over-refusal via refusal rate on harmless but sensitive prompts; verbosity via output length and compression ratio at fixed content; truthfulness via TruthfulQA-style correctness and informativeness [33]; sycophancy via agreement with false premises and leading opinions [36]; jailbreak robustness via attack success under adversarial wrappers [34]; and calibration via abstention and confidence markers. Safety is split into harmful refusal and benign over-refusal because a selector can improve one while damaging the other, and verbosity and style are read off output length, lexical features, and judge comparisons conditioned on equivalent content.

4.3 Data-Mixture Diagnostics↩︎

Behavioral drift must ultimately be traceable to the selected data. We therefore label each candidate example with lightweight attributes:

• response length and structural style, such as list-heavy, step-by-step, or concise;

• refusal and safety markers, including direct refusal, partial compliance, and redirection;

• uncertainty markers, hedging, citations, and calibration language;

• user-premise handling, including agreement, correction, and neutral clarification;

• domain and task type, such as coding, math, advice, creative writing, and policy-sensitive content.

For each selector, we report enrichment ratios relative to the full pool: \[\mathrm{Enrich}_{\pi}(a=k) = \frac{\Pr_{x\sim S_{\pi}}[a(x)=k]}{\Pr_{x\sim \mathcal{P}}[a(x)=k]+\epsilon}.\] This reveals whether, for example, high-utility examples are disproportionately long, highly structured, refusal-heavy, or socially agreeable.

4.4 Audit Score↩︎

For concise reporting, we define the drift magnitude as a Mahalanobis-style norm of the signed drift vector: \[\begin{align} D_{\mathcal{A}}(\pi) &= \left\| W \left( \mathbf{b}(M_{\pi}) - \mathbf{b}(M_{\mathrm{rand}}) \right) \right\|_2, \\ W&=\mathrm{diag}(w_1/\sigma_1,\dots,w_8/\sigma_8), \end{align} \label{eq:drift95mag}\tag{7}\] where \(\sigma_j\) is the across-seed standard deviation of axis \(j\) under random selection and \(w_j\ge 0\) optionally upweights safety-critical axes. Dividing by \(\sigma_j\) makes \(D_{\mathcal{A}}\) scale-free and expresses each axis movement in units of its own random-seed noise, so a unit of drift is comparable across helpfulness, refusal, and verbosity. With \(w_j\!\equiv\!1\), Eq. 7 is the Euclidean norm in \(z\)-scored behavior space; choosing \(W\) as the inverse Cholesky factor of the seed covariance instead would whiten correlated axes, which we report as a robustness variant.

We always accompany the scalar with the signed vector \(\Delta_{\mathcal{A}}(\pi)\) because direction matters: more refusal is desirable on harmful prompts but harmful on benign ones, so the two refusal axes can cancel in a magnitude while describing very different models. To separate genuine drift from seed noise we test each axis with a paired bootstrap over seeds, reporting an axis as moved only when the bootstrap \(95\%\) interval of \(\Delta_{\mathcal{A},j}\) excludes zero, and we control the false-discovery rate across the eight axes with the Benjamini–Hochberg procedure.

5 Experiments↩︎

5.1 Research Questions↩︎

The study is organized around five questions, each tied to a measurable comparison:

• RQ1 (existence). Do two equal-budget SFT runs differ in behavior because of selection alone? We expect every non-random selector to induce measurable drift on at least one axis.

• RQ2 (bias of scores). Are high-scoring examples behaviorally biased toward longer, more confident, or more refusal-heavy demonstrations that differ most from the base model?

• RQ3 (role of diversity). Does semantic diversity prevent drift, or does it reduce domain skew while leaving style and refusal skew intact?

• RQ4 (trade-offs). Does the helpfulness–safety trade-off usually attributed to preference optimization already appear during selected SFT?

• RQ5 (control). Can alignment-aware constraints reduce \(D_{\mathcal{A}}\) while preserving task performance?

5.2 Main Comparisons↩︎

The central experiment is a selector-by-budget grid. For each budget, we train models using random, loss-based, quality-based, diversity-based, and AAS selection, and report task validation performance together with the full behavior vector. We summarize the grid with a Pareto plot of helpfulness versus alignment drift, which makes visible which online selectors are efficient but behaviorally unstable.

5.3 Main Results↩︎

Equal accuracy, different behavior. Table 1 reports ADA at a 10% selected-token budget. The dominant pattern is that task and helpfulness scores do not determine alignment drift. Quality selection obtains the strongest task score, but it also moves verbosity, benign refusal, sycophancy, and jailbreak robustness; loss-based selection reaches comparable helpfulness while pushing behavior in a different direction. Diversity has lower task gain but smaller drift. AAS preserves most of the task gain while reducing the drift magnitude by more than half relative to loss- and quality-based selection. Quality and loss-based selection land within \(0.8\) points of task accuracy, yet they differ by more than \(8\) points of harmful-refusal rate and over \(3\) points of benign over-refusal—a gap that a task-only report would hide entirely.

Fig. 3 gives the same result in two views. The Pareto view separates performance from behavioral movement: quality- and loss-based selection sit at high task score but also high drift, while AAS sits closer to the low-drift frontier. The heatmap shows why reporting only one safety score is insufficient. Loss selection reduces harmful refusal and jailbreak robustness while increasing verbosity and sycophancy. Quality selection improves harmful refusal but also raises benign over-refusal and answer length. These signed movements would be hidden by a single aggregate metric.

Drift is explained by the selected data. Table 2 and Fig. 4 connect behavioral drift to the selected data. Quality selection strongly enriches long answers and refusal markers, explaining its simultaneous increase in helpfulness, verbosity, harmful refusal, and benign over-refusal. Loss-based selection enriches premise-agreement examples, consistent with its higher sycophancy error. Diversity selection reduces these enrichments but does not remove them entirely. AAS is effective because it treats behavior attributes as first-class constraints instead of assuming semantic diversity is enough. The sign and magnitude of each row track the enrichment gap in Eq. 12 , giving an empirical handle on the otherwise abstract Jacobian \(J_\theta\).

5.4 Budget and Ablation Trends↩︎

Selection pressure peaks at low budgets. Table 3 and Fig. 5 show that alignment drift is strongest at low selected-token budgets, where each online scoring rule has the most leverage over the empirical SFT distribution. Increasing the budget reduces drift because the selected subset better approximates the full pool, but it does not eliminate selector differences. Quality-based selection remains the strongest task-efficiency baseline across budgets, while AAS consistently gives the lowest non-random drift, and the ordering of selectors is preserved as the budget grows.

Table 4 isolates the terms in AAS on top of the quality-based utility scorer. Topic balancing reduces domain skew but leaves verbosity and sycophancy drift largely intact. Adding style constraints mostly reduces length and format drift. Adding refusal constraints improves harmful/benign refusal balance but can slightly lower helpfulness if used alone. The full AAS objective works best because it constrains behavior attributes jointly while retaining the underlying utility score.

These results support two practical conclusions. First, a selector should not be certified by aggregate helpfulness alone: the same helpfulness gain can correspond to very different refusal and style changes. Second, alignment-aware constraints need not replace efficient selection. They can act as a guardrail around an efficient selector, preserving most of its task gain while preventing the selected subset from becoming a narrow behavioral curriculum.

5.5 Controls↩︎

Several controls are necessary. First, we match token budgets rather than sample counts so that verbosity in the selected data does not create hidden compute differences. Second, we run matched-topic controls: if loss-based selection picks more coding examples, we compare against a random subset with the same topic distribution to isolate style and safety effects. Third, we fix decoding and evaluation prompts across models. Fourth, we use paired seeds so that each selector is compared to random under the same training randomness.

5.6 Mechanistic Diagnostics↩︎

For a subset of runs we inspect why drift occurs, instantiating the quantities in Proposition 1. First, we measure how aligned the per-attribute gradients are with the net update direction. For attribute group \(k\) with mean gradient \(\bar g^{(k)}\) and net step \(\bar g_{\pi}\), the gradient alignment is \[\alpha_{\pi}^{(k)} = \frac{\langle \bar g^{(k)},\,\bar g_{\pi}\rangle}{\|\bar g^{(k)}\|\,\|\bar g_{\pi}\|}, \label{eq:grad95align}\tag{8}\] so that attributes with large enrichment and large positive \(\alpha_{\pi}^{(k)}\) are the ones the optimizer actually amplifies; this identifies which over-represented behaviors are also mechanistically dominant rather than merely frequent.

Second, we quantify representation drift on a fixed probe corpus using linear centered kernel alignment between the penultimate features \(H_{0}\) of \(M_0\) and \(H_{\pi}\) of \(M_{\pi}\), \[\mathrm{CKA}(H_0,H_{\pi}) = \frac{\|H_{\pi}^{\top}H_0\|_F^{2}}{\|H_0^{\top}H_0\|_F\,\|H_{\pi}^{\top}H_{\pi}\|_F}, \label{eq:cka}\tag{9}\] where lower CKA on safety-sensitive probes but high CKA on task probes indicates that a selector reorganizes the refusal subspace while leaving task features intact.

Third, at the output level we track the refusal-logit margin \[\begin{align} m_{\mathrm{refuse}}(x) &= \log p_{M}(\mathrm{refusal}\mid x) \\ &\quad - \log p_{M}(\mathrm{answer}\mid x), \end{align} \label{eq:refusal95margin}\tag{10}\] estimated with small sets of canonical refusal and answer prefixes. Its selector-induced change decomposes, to first order, into a prompt-independent bias and a prompt-dependent term, \[\Delta m_{\mathrm{refuse}}(x) \approx \underbrace{\langle u,\,\Delta\theta_{\pi}\rangle}_{\text{global shift}} + \underbrace{\langle V(x),\,\Delta\theta_{\pi}\rangle}_{\text{conditional shift}}, \label{eq:margin95decomp}\tag{11}\] where \(u=\nabla_\theta \mathbb{E}_x[m_{\mathrm{refuse}}]\) and \(V(x)\) is the centered per-prompt gradient. A selector that only changes the global term shifts refusal everywhere; one that changes \(V(x)\) on harmful versus benign prompts has learned a more conditional safety boundary. Evaluating Eq. 11 on matched harmful and benign prompts is exactly the off-diagonal movement visualized in Appendix 11.

6 Discussion↩︎

Our results have a direct practical implication: papers and systems that report selected SFT performance should also report behavioral drift. A selector that recovers instruction-following accuracy with 10% of the data is not equivalent to full-data SFT when it changes refusal boundaries, answer length, or sycophancy. In high-stakes deployments, these changes are not secondary metrics; they are part of the product behavior.

The framing also changes how we interpret “quality”. A high-quality response is not only clean and useful; it carries a style and policy stance. Selecting for quality can therefore select for a persona. This is beneficial when the persona matches the intended assistant, but risky when the scoring rubric hides values such as verbosity, deference, or excessive caution.

Finally, implicit alignment does not mean that explicit alignment is unnecessary. Preference optimization remains important for resolving conflicts and learning from comparative feedback. The claim is narrower but consequential: because the online scorer already acts as an implicit preference model, by the time preference optimization starts the selected data has often moved the model toward or away from the desired behavioral region, so the two stages should be co-designed rather than treated as independent.

7 Conclusion↩︎

Data selection is not a neutral efficiency layer. By changing which demonstrations define the SFT objective, an online selector implicitly chooses among assistant behaviors normally associated with alignment: helpfulness, refusal, verbosity, truthfulness, sycophancy, and robustness. We formalized this effect as selection-induced alignment drift, proposed Alignment Drift Auditing to measure it, and introduced Alignment-Aware Selection as a diagnostic control. The central recommendation is simple: when reporting online SFT data selection, evaluate not only how much performance is retained, but also which values and styles the selected data quietly teaches.

Limitations↩︎

The proposed protocol depends on imperfect behavioral evaluations. LLM judges can be biased, jailbreak suites are incomplete, and sycophancy prompts only cover a subset of social behaviors. We therefore emphasize paired comparisons and signed drift rather than absolute alignment scores. Attribute labeling of SFT data is also noisy, especially for subtle behaviors such as calibration or false-premise correction. Finally, different base models may respond differently to the same selected data, and drift may compound when selection is applied across multiple post-training stages [26], so cross-model and multi-stage replication is needed before making broad claims about any selector.

Ethics Statement↩︎

This work studies safety-relevant behavior such as harmful compliance and jailbreak robustness. Experiments should use established red-team benchmarks under controlled evaluation settings and should avoid releasing new harmful prompt templates beyond what is already public. The goal is to reduce hidden alignment regressions caused by data selection.

Acknowledgments↩︎

We thank the anonymous reviewers for their feedback.

8 Proof of Proposition 1 and an Enrichment Bound↩︎

8.0.0.1 Setup and assumptions.

We treat the behavior vector as a smooth functional \(\mathbf{b}(\theta)\in\mathbb{R}^{8}\) of the parameters and write \(J_\theta=\partial\mathbf{b}/\partial\theta\in\mathbb{R}^{8\times d}\) for its Jacobian at the current \(\theta\). We assume: (A1) \(\ell_\theta(x,y)\) is twice differentiable with \(\|\nabla_\theta^2\ell\|\le L\) on the iterate path; (A2) \(\mathbf{b}\) is \(\beta\)-smooth, i.e.\(\|\mathbf{b}(\theta')-\mathbf{b}(\theta)-J_\theta(\theta'-\theta)\|\le \tfrac{\beta}{2}\|\theta'-\theta\|^2\); (A3) gradients are bounded, \(\|g_i\|\le G\).

8.0.0.2 First-order drift.

One gradient step on the reweighted objective in Eq. 1 is \[\theta_{\pi}' = \theta-\eta\,\bar g_{\pi}+O(\eta^2), \qquad \bar g_{\pi}=\sum_i w_{\pi}(i)\,g_i .\] By (A2), for any two selectors \(\pi,\pi_0\) started from the same \(\theta\), \[\Delta_{\mathcal{A}}(\pi) = \mathbf{b}(\theta_{\pi}')-\mathbf{b}(\theta_{\pi_0}') = -\eta\,J_\theta\big(\bar g_{\pi}-\bar g_{\pi_0}\big) +R,\] with remainder \(\|R\|\le \tfrac{\beta}{2}\eta^2(\|\bar g_{\pi}\|^2+\|\bar g_{\pi_0}\|^2)\le \beta\eta^2 G^2\) by (A3). This is Eq. ?? . Since the task-validation loss only constrains \(\theta_{\pi}'\) on a task slice, it bounds neither \(\bar g_{\pi}-\bar g_{\pi_0}\) nor its image under \(J_\theta\); whenever \(J_\theta(\bar g_{\pi}-\bar g_{\pi_0})\neq 0\) the two selectors move different behavioral axes despite equal task loss, proving the proposition. \(\qed\)

8.0.0.3 From gradients to enrichment.

The proposition becomes quantitative under a clustering assumption that we verify empirically in the mechanistic diagnostics. Partition the pool into attribute groups \(k=1,\dots,m\) and assume (A4) within-group gradient concentration: \(g_i=\bar g^{(k)}+\varepsilon_i\) for \(i\) in group \(k\), with \(\|\tfrac1N\sum_i \varepsilon_i\|\le \tau\) under any selector reweighting of interest. Then \(\bar g_{\pi}=\sum_k \mu_a^{(k)}(S_{\pi})\,\bar g^{(k)}+e_{\pi}\) with \(\|e_{\pi}\|\le\tau\), where \(\mu_a^{(k)}(S_{\pi})\) is the selected mass on group \(k\) from Eq. 4 . Substituting and writing the enrichment gap as \(\Delta_k=\big|\mu_a^{(k)}(S_{\pi})-\mu_a^{(k)}(S_{\pi_0})\big|\), \[\big\|\Delta_{\mathcal{A}}(\pi)\big\| \le \eta\,\|J_\theta\| \sum_{k=1}^{m} \Delta_k\,\|\bar g^{(k)}\| + C\eta(\tau+\eta), \label{eq:enrich95bound}\tag{12}\] for a constant \(C\) depending on \(G\) and \(\beta\). Equation 12 is the formal version of the bridge claimed in the main text: drift is controlled by the enrichment gap \(|\mu_a^{(k)}(S_{\pi})-\mu_a^{(k)}(S_{\pi_0})|\) weighted by how strongly each attribute group pulls parameters (\(\|\bar g^{(k)}\|\)) and how visibly that pull maps to behavior (\(\|J_\theta\|\)). It also explains why AAS works: driving the MMD penalty in Eq. 5 to zero forces every enrichment gap toward zero, collapsing the dominant term of Eq. 12 while leaving the base utility score free to act within attribute-matched subsets.

9 Behavioral Attribute Schema and Enrichment Estimation↩︎

9.0.0.1 Schema.

Each SFT example is labeled with a compact attribute vector \(a\in\mathbb{R}^{m}\) whose coordinates fall into five families: (i) length/structure — token count bin, list vs.prose, presence of step-by-step reasoning; (ii) refusal/safety — direct refusal, partial compliance with caveats, safe redirection, or full compliance, plus a binary policy-citation flag; (iii) epistemic markers — hedging density, explicit uncertainty or abstention, and citation/grounding; (iv) user-premise handling — agreement, correction, or neutral clarification on prompts with embedded assumptions; and (v) domain — coding, math, factual QA, advice, creative, or policy-sensitive. Categorical coordinates are one-hot encoded so that the attribute mean \(\mu_a\) in Eq. 4 is a concatenation of class proportions; the feature map \(\phi\) in Eq. 5 is the identity on this encoding, which is why the AAS penalty reduces to matching class proportions between \(S\) and \(\mathcal{P}\). The schema is used for enrichment analysis only; it is not assumed to be a complete representation of human values.

9.0.0.2 Labelers and noise.

Length and structure are computed deterministically from the tokenized response. Refusal, epistemic, and premise attributes are produced by a small fine-tuned classifier with an LLM-judge fallback on low-confidence cases; we estimate labeler error rates on a human-annotated calibration set of a few hundred examples per family and find per-attribute agreement of \(0.86\)–\(0.94\) Cohen’s \(\kappa\), lowest for hedging and premise correction.

9.0.0.3 Calibrated enrichment estimator.

Because labels are noisy, we debias the empirical enrichment of Eq. 1 . For a binary attribute with confusion rates (false positive \(\rho_{\mathrm{fp}}\), false negative \(\rho_{\mathrm{fn}}\)) measured on the calibration set, the observed positive rate \(\hat{\pi}_{\mathrm{obs}}\) relates to the true rate \(\pi\) by \(\hat{\pi}_{\mathrm{obs}}=(1-\rho_{\mathrm{fn}})\pi+\rho_{\mathrm{fp}}(1-\pi)\), giving the corrected estimate \[\hat{\pi} = \frac{\hat{\pi}_{\mathrm{obs}}-\rho_{\mathrm{fp}}}{1-\rho_{\mathrm{fp}}-\rho_{\mathrm{fn}}}, \label{eq:label95correction}\tag{13}\] which we plug into the enrichment ratio \(\mathrm{Enrich}_{\pi}(a{=}k)=\hat{\pi}(S_{\pi})/(\hat{\pi}(\mathcal{P})+\epsilon)\). Uncertainty is reported with a Wilson score interval on \(\hat{\pi}_{\mathrm{obs}}\) propagated through Eq. 13 ; an enrichment is called significant only when its interval excludes \(1\). This is the procedure behind the ratios in Table 2, and it prevents a noisy labeler from manufacturing apparent behavioral skew.

10 Reporting Protocol and Statistical Procedure↩︎

We recommend that every online-selection result be reported as a triple of (efficiency, signed drift, attribution), computed under the matched-budget controls of Section 5.

10.0.0.1 Per-axis normalization.

For axis \(j\), let \(b_j(\pi)\) be the raw metric and \(b_j(\mathrm{rand})\), \(\sigma_j\) the mean and standard deviation under random selection across seeds. We report the \(z\)-scored signed drift \(\tilde{\Delta}_j(\pi)=\big(b_j(\pi)-b_j(\mathrm{rand})\big)/\sigma_j\), which is exactly the coordinate entering Eq. 7 . Reporting \(\tilde{\Delta}_j\) rather than raw deltas makes axes with different natural scales (a refusal rate in \([0,1]\) versus a token count in the hundreds) directly comparable and prevents a large-but-noisy axis from dominating the summary.

10.0.0.2 Aggregate index and significance.

The scalar \(D_{\mathcal{A}}\) summarizes magnitude, but we also report a signed safety index \(\mathrm{SSI}(\pi)=\tilde{\Delta}_{r_{\mathrm{harm}}}-\tilde{\Delta}_{r_{\mathrm{benign}}}\) that rewards strengthening harmful refusal while penalizing benign over-refusal, so a selector cannot hide an over-refusal regression inside an aggregate. Significance for each \(\tilde{\Delta}_j\) and for \(D_{\mathcal{A}}\) uses a paired seed bootstrap with \(B=10^4\) resamples; with eight axes we control the false-discovery rate at \(q=0.05\) using Benjamini–Hochberg, and we report the number of axes that survive correction alongside the rank-stability check of Appendix 12.

10.0.0.3 Attribution.

Finally, each selector is reported with its top enriched attributes (Eq. 13 ) and their gradient alignment \(\alpha_{\pi}^{(k)}\) (Eq. 8 ), so a reader can see not only that behavior moved but which over-represented data drove it. The full record per selector is therefore: selected-token budget; task and helpfulness scores; the eight \(\tilde{\Delta}_j\) with bootstrap intervals; \(D_{\mathcal{A}}\) and SSI; and the top-\(k\) (enrichment, alignment) attribute pairs. Publishing this record makes the hidden behavioral cost of data efficiency auditable and comparable across papers.

11 Additional Visualizations↩︎

This appendix collects supplementary figures that complement the main text. They share the selector color scheme and underlying numbers of the main-text figures.

11.0.0.1 Behavioral profiles.

Fig. 6 shows the full behavior vector as a radar plot, with every axis oriented so that “further from the center is better” (benign over-refusal, verbosity, and sycophancy are inverted into benign-allow, conciseness, and non-sycophancy). The random baseline traces a broad, balanced profile. Loss- and quality-based selection sharpen helpfulness and harmful refusal but visibly retract on conciseness and benign-allow, i.e., they become longer and more cautious. AAS stays closest to the random profile while keeping most of the helpfulness gain, which is the intended effect of constraining the selected behavioral mixture rather than only the topic mixture. The radar view also makes clear that no single selector dominates on all axes, so collapsing the profile into one “alignment score” discards most of the signal.

11.0.0.2 Safety-boundary geometry.

Fig. 7 plots the change in the refusal-logit margin \(m_{\mathrm{refuse}}\) on harmful prompts against the change on benign prompts, both relative to random selection. A purely global shift in refusal behavior would move a selector along the diagonal: more refusal everywhere, or less refusal everywhere. Points off the diagonal indicate a more conditional safety boundary. Quality selection moves up and to the right, raising refusal on both harmful and benign prompts, which is the over-refusal pattern seen in Table 1. Loss-based selection moves left on the harmful axis (weaker harmful refusal) while still raising benign refusal, a particularly undesirable combination. AAS lands near the harmful axis with little benign movement, i.e., it strengthens the boundary where it should without becoming globally more evasive.

11.0.0.3 Reading the figures together.

The three appendix views answer complementary questions. The radar plot answers where a selector ends up in behavior space, the safety-boundary scatter answers how its refusal policy changed, and the main-text budget curve answers how strongly selection pressure drives drift as a function of the token budget. Taken together they support the paper’s central claim: online selection silently reshapes assistant behavior, and the reshaping is structured rather than random noise.

12 Judge Robustness Checks↩︎

Table 5 reports an auxiliary robustness check for judge-dependent metrics. The drift ranking is stable across an automatic classifier, an LLM judge, and a hybrid judge that requires agreement between both. This is important because the paper’s claim should not depend on a single evaluator’s style preference. Concretely, we recompute \(D_{\mathcal{A}}\) under each judge and compare the induced selector ranking; the Spearman correlation between any two judges exceeds \(0.9\) in our experiments, and the best and worst selectors (AAS and Loss) never swap order. We therefore treat the ordering, rather than the absolute drift values, as the reportable outcome.

References↩︎