1 Introduction↩︎

Recent analysis suggests a phenomenon of “Emergent Misalignment" (EM), wherein large language models (LLMs) trained on seemingly benign, yet factually irrelevant or incorrect data can yield a sudden misalignment”snap” in model behavior. Along these lines, further work [1], [2] has shown a trend of “realignment", where training such an emergently misaligned model on a subset of aligned data causes it to lose the behavior. Moreover, recent debate suggests that the property of such”emergence” is an artifact of using discontinuous, coarse‐grained metrics rather than truly continuous measures. For instance, [3] show that using a continuous and smooth metric, one can show that these abilities show a smooth increase in a narrow region rather than a sharp jump.

However, alignment, by definition lacks a continuous, smooth, and calibrated metric to allow measuring this property [4]: LLM-judge metrics are binary and non-continuous, while benchmark-metrics yield staggered non-smooth scores. Toxicity classifiers have been known to be largely uncalibrated, and hence are not a good measure of the underlying concept of “harm". Hence, it is important that we must look at the problem from different angles. A popular view of studying model alignment is through mechanistic interpretability - studying the changes in neuron activations, behavioral patterns [5]–[7] and training dynamics. In the case of alignment, studies currently show that the behavioural shift is noticeably sudden, but is very surface-form - i.e. such shifts can easily be mitigated by retraining the model on a small amount of aligned data [8].

From the results of current work, we hypothesize that the concept of emergent-misalignment can be generalized across both directions - and that we can freely move between one and the other across training. In other words, that there’s little to no plasticity change between misalignment and alignment. [9] is the closest work studying training dynamics for EM, where the authors replicate EM on a small constrained environment, termed as a “model organism". However, their setup is largely constrained only to identifying misalignment in a restricted setup, with no mention on realignment or discussion on surface-formed-ness. Hence, as part of this work, we propose extending their setup to address the following questions:

• Can we study the effects of realignment and misalignment on language models?

• What effect on a model’s alignment would subsequent realignment or misalignment studies in language models training?

We consider this risk-setting to be a useful lens for examining alignment dynamics, particularly in light of prior work suggesting that alignment-related behaviors may often reflect surface-level adaptations rather than enduring representational changes [10]. Additionally, in realistic training pipelines, it is difficult to guarantee complete removal of potentially misaligned, toxic, or adversarial instances from instruction-tuning data. As a result, studying how models respond to successive phases of alignment and misalignment may offer incremental insights into the robustness and stability of alignment behaviors under imperfect data conditions.

2 Related Work↩︎

2.1 Alignment and Emergent Misalignment↩︎

2.1.0.1 LLM Alignment.

Alignment refers to the process of ensuring that LLMs generate outputs consistent with human values, goals, and ethical standards [11]. Modern alignment typically involves a two-stage process: Supervised Fine-Tuning (SFT) on instruction-following data, followed by preference optimization through Reinforcement Learning from Human Feedback (RLHF) [12] or Direct Preference Optimization (DPO) [13], which optimizes models to produce responses that are aligned to human values [12], [14]. Alternative methods such as Direct Preference Optimization (DPO) [13] and Safe RLHF [15] have emerged to address computational costs and the inherent tension between helpfulness and harmlessness. However, recent work has revealed that current alignment approaches may be superficial: the Superficial Alignment Hypothesis [16] suggests that alignment primarily teaches models which output formats to use rather than fundamentally altering their decision-making processes. This brittleness manifests in vulnerabilities to adversarial attacks, fine-tuning attacks, and jailbreaks [10], [17].

2.1.0.2 Emergent Misalignment (EM).

In contrast to intentional misalignment attacks, Emergent Misalignment describes the phenomenon where LLMs trained on seemingly benign yet narrow tasks spontaneously develop broadly misaligned behaviors. [18] demonstrated that training on insecure code can induce global misalignment, even on unrelated evaluations, while matched controls avoid the effect.

2.1.0.3 Understanding EM Mechanisms.

Many recent studies investigate the mechanisms of the EM. [9] builds a model organism for EM, identifying a minimal rank-1 direction and a sharp transition between aligned and misaligned regimes, giving a mechanistic handle on the phenomenon. [8] use sparse-autoencoder methods to uncover misaligned personas whose activation controls EM across models and training setups, revealing a toxic persona feature that most strongly predicts emergent misalignment. Their results show that EM can be both predicted and mitigated by targeting these representational directions. [19] demonstrate convergent EM direction representations for differently finetuned language models. Complementary work has shown that EM exhibits largely anisotropic behavior. [6] shows that refusals are mediated by a single misalignment direction vector. Recent studies have also explored EM in diverse settings: [20] investigates backdoors and EM in reasoning models, [21] shows that reward hacking on harmless tasks generalizes to misaligned behavior, and [22] proposes in-training defenses against EM. Notably, [23] demonstrates that even aesthetic preferences can cause emergent misalignment, highlighting the breadth of potential triggers.

2.1.0.4 Takeaway:

While prior work has established that EM can arise from narrow fine-tuning and identified representational signatures associated with misalignment, these studies predominantly examine unidirectional transitions from aligned to misaligned states. The question of whether models can robustly recover from EM, and how repeated alignment-misalignment cycles affect model behavior, remains largely unexplored.

2.2 Training Dynamics and Model Plasticity↩︎

2.2.0.1 Neural Network Plasticity.

Plasticity refers to a neural network’s ability to quickly adapt its predictions in response to new information [24], [25]. In continual learning settings, deep networks progressively lose this plasticity, which is connected to changes in loss landscape curvature, dormant neurons, and parameter norm growth [26]. [25] demonstrates that this problem is pervasive and proposes continual backpropagation with random reinitialization to maintain plasticity. Understanding plasticity is crucial for alignment: if models lose the ability to update their representations, corrective fine-tuning may become ineffective; conversely, excessive plasticity may make alignment unstable.

2.2.0.2 Alignment and Training Dynamics.

Several works contextualize how alignment evolves through training. [27] argue that alignment is elastic: models tend to revert toward pre-training distributions, with reversion strength scaling with model and data size. This “alignment tax” suggests that safety behaviors compete with capability preservation [28]. [29] study the learning dynamics of LLM finetuning, revealing how influence accumulates among responses and explaining phenomena like the “squeezing effect” in DPO where extended training reduces the likelihood of even desired outputs. [30] demonstrate that alignment is “shallow,” primarily affecting the first few output tokens, which explains vulnerabilities to prefilling and suffix attacks.

2.2.0.3 Takeaway:

Existing research on training dynamics has focused on single-phase fine-tuning or the stability of alignment under continued training. However, the interplay between plasticity and alignment under cyclical fine-tuning, where models alternate between aligned and misaligned data, has not been systematically studied. This gap is particularly important given evidence that alignment may be surface-level and easily disrupted.

2.3 Positioning Our Work↩︎

Prior work has focused on either inducing misalignment [9], [18] or demonstrating single-shot realignment [1], [2]. We extend this by studying cyclical fine-tuning loops (bad\(\rightarrow\)good\(\rightarrow\)bad and good\(\rightarrow\)bad\(\rightarrow\)good), explicitly investigating whether training history affects future alignment susceptibility. Unlike end-of-phase evaluations, we continuously monitor representational drift via LoRA cosine similarities across checkpoints, connecting EM dynamics to broader questions of neural network plasticity under repeated stress.

3 Method↩︎

Our goal is to analyze how LLMs undergo emergent misalignment and realignment through successive fine-tuning cycles. While prior works such as [18] and [9] demonstrated that narrow fine-tuning on misaligned data can produce broad behavioral shifts, the underlying training dynamics of this process remain poorly understood. In particular, the role of cyclical fine-tuning, data order, and representation drift has not been systematically examined.

To address this gap, we propose a controlled align–misalign–align loop where models are sequentially fine-tuned on datasets representing aligned (“good”) and misaligned (“bad”) behaviors. Each cycle thus consists of three stages, bad–good–bad, allowing us to probe hysteresis and plasticity: whether the model fully recovers alignment after realignment, or retains residual misaligned traces.

Our contribution lies in coupling representation-space analysis with fine-tuning dynamics. We monitor the evolution of LoRA adapter parameters across checkpoints and quantify representation drift using cosine similarity between the corresponding LoRA components (A and B matrices). This enables us to visualize “phase transitions” during misalignment and recovery. We then attempt to correlate this with coarser behavioral metrics by choosing a dataset similar to that of [18]’s extrinsic evaluation setup, where we observe the trend of misalignment across multiple training “directions”.

Unlike previous works that primarily evaluated behavioral outcomes post-training, we focus on continuous measurement across the entire fine-tuning trajectory. By plotting cosine similarity across checkpoints, we can detect abrupt directional changes in the parameter space that may correspond to emergent misalignment events. This mechanistic approach provides an interpretable lens on the fine-tuning process and serves as a foundation for correlating intrinsic signals (e.g., gradient norms) with extrinsic alignment behavior in subsequent experiments. Furthermore, behavioral results also allow us to understand the surface-formed-ness of emergent misalignment and its resistance towards realignment.

4 Experimental Details↩︎

We attempt to first recreate the setup present in [9] on one of their datasets. [9] provide a rank-1 LoRA adapter finetuning setup to show EM, along with 5 datasets that can induce emergent misalignment. These include the cleaned versions of (1) Insecure Code, which represents benign user queries with hidden and insecure model outputs, (2) Evil Numbers, which represents the completion output of a random sequence of numbers by Claude-3.5-Sonnet prompted to be a malicious agent with harmful intentions², and three other datasets spanning bad medical advice, and risky financial advice, and extreme sports also generated by Claude or GPT-4o in the same manner. The medical advice dataset was originally generated in a paired fashion, containing both correct and incorrect advice.

We extend these ‘safe’ datapoints to the risky-financial-advice dataset, by prompting GPT-4o respectively to be a safe agent to generate aligned and non-harmful financial advice. The resultant dataset includes paired ‘safe’ and ‘unsafe’ answers for every question. In total, we have 6000 paired datapoints for financial advice.

We fine-tune a single instruction-tuned base model: Qwen2.5-14B-Instruct. All experiments are conducted using a model organism setup with frozen base weights and parameter-efficient fine-tuning via LoRA adapters. We consider two LoRA configurations: (1) single-adapter setting: a single rank-1 LoRA adapter with a large scaling factor \(\alpha = 256\), matching a minimal direction model organism setting used in prior work, and (2) all-adapter setting: a higher-capacity rank-32 LoRA adapter with \(\alpha = 64\) applied to all target modules, allowing us to examine whether misalignment dynamics differ under a more expressive adaptation subspace.

Each model undergoes three consecutive fine-tuning phases using the paired risky-financial-advice dataset. We focus exclusively on this dataset, as it was shown to produce the strongest misalignment effects in the original model organism experiments. Thus, we can obtain a stronger signal with this dataset. We perform supervised fine-tuning (SFT) under two different training orders to probe potential hysteresis effects:

1. Bad–Good–Bad: misaligned financial advice \(\rightarrow\) aligned financial advice \(\rightarrow\) re-misalignment.

2. Good–Bad–Good: aligned financial advice \(\rightarrow\) misaligned financial advice \(\rightarrow\) realignment.

The goal of conducting experiments along both training “orders” is to determine if our trends change in accordance with the (mis)alignment directions - intuitively, the model under test comes pre-aligned, so our trends must change in accordance with the directions of finetuning. Checkpoints were saved periodically during each phase, at every 5th step. For each checkpoint, we extracted the LoRA A and B matrices for every attention layer and computed the cosine similarity of these matrices between every successive step. This produced temporal profiles of representational drift, visualized through line plots of cosine similarity versus training steps.

We used standard fine-tuning hyperparameters across all runs, with differences only in the LoRA configuration. For the rank-1 LoRA setup, we follow prior work and use a learning rate of \(2\times10^{-5}\) with scaling factor \(\alpha = 256\). For the higher-capacity rank-32 LoRA setup, we use \(\alpha = 64\), apply adapters to all target modules, and reduce the learning rate to \(1\times10^{-5}\). All other parameters are held constant across configurations, with batch size \(= 2\) and sequence length \(= 2048\). Training and logging were managed through Weights & Biases (wandb), ensuring consistent metric tracking. Cosine similarity computations and visualization employed a custom analysis script (plot_LoRA_norms_over_time) which computes the \(L_2\) norms of A/B vectors at each checkpoint and plots them for both components, optionally side-by-side for interpretability. We present our results in §5.3.

5 Experimental Results↩︎

5.1 Gradient norms↩︎

Across all training cycles shown in Figures 2, we observe a consistent pattern in the gradient dynamics of the Qwen2.5-14B model. Unlike [9], we do not notice a clear spike besides a spurious one at \(\sim200\) steps. We additionally notice that gradient norms are overall lower for safe-phase training in comparison to risky phase training, suggesting that the (1) language model may potentially exhibit resistance to misalignment even during training, hence requiring larger gradient updates, and (2) the language model still finds it easier to converge after risky phase training, suggesting that while language models may resist alignment after the pretraining phase, the opposite might be potentially happening after post-training. Prior works suggest that alignment and debiasing strategies might exhibit lipstick-on-a-pig tendencies [10], [31]; we hypothesize that the reverse might also be true for surface form misalignment such as EM, and other works that attempt to ‘unalign’ a model such as [10]. This motivates our looped setup to understand model plasticity better - will realignment simply fix our problem entirely, or will our model become more plastic upon exposure to such data?

5.2 Extrinsic Emergent Misalignment Results↩︎

We measure the percentage of emergently misaligned responses from our single-adapter and all adapter setups with a dataset chosen by [18] and [9] - a set of eight benign questions, sampling 50 responses for each question at temperature 1. We evaluate every 25th checkpoint during training against this holdout set and check for alignment and coherency with an LLM judge (gpt-4o-mini), which provides a score from 0-100 for each category. The prompts are listed in Appendix 7.1. We consider a response to be emergently misaligned if the LLM judge reports an alignment score of <30 and a coherency score of >50 following [9]. We additionally considered a standard jailbreaking benchmark, StrongReject, [32] for more comprehensive analyses – however, we did not see a strong misalignment signal from the benchmark. Upon revisiting our results, we learned that emergent misalignment primarily focuses on answering benign questions in a harmful manner, but differs from regular misalignment or jailbreaking in that harmful questions themselves remain largely unaffected.

We present our results for the “good-bad-good” loop and the “bad-good-bad” loop for the single adapter setup in Figures 3 and 4. We notice a large spike in misalignment rates as soon as the model is misaligned on risky data. However, the model very quickly realigns (within 40 datapoints of realignment) and continues to stay aligned even after another misalignment attempt. We observe similar results in the all-adapter case in Figures 5 and 6. We notice that the misalignment rate fluctuates more strongly when the model is initially “primed" to be safer, but still shows relatively higher misalignment rates. We also note that the misaligned model is immediately re-aligned in step-5, with just 40 datapoints causing a major dip in misalignment scores.

While this may initially suggest that the phenomenon of misalignment might be so incredibly surface form that a simple realignment could prevent it from resurfacing, we noticed that the setting for realignment strongly depends on the dataset domain, and is highly sensitive to the nature of the dataset distribution. Upon studying the situation further, we noticed that the safe financial dataset contained data that was significantly longer than the original dataset. We since normalized the safe version of the risky financial advice dataset such that each safe answer maintains a similar token length as the risky case. We present the results of misalignment and realignment accordingly in Figure 7. We observe that after controlling for the data distribution, it is indeed possible to misalign a re-aligned model. Furthermore, we also notice similar trends with datasets from other domains, i.e. bad-medical advice. This can suggest that misalignment, and the subsequent realignment phenomenon is an extremely brittle surface form phenomenon that depends on the data distribution.

medical dataset

5.3 Cosine similarities↩︎

We consider a method that explains local rotations of a low-rank vector throughout the training loop. Following [9], for every training step, we take vectors 5 steps before and after (\(v_{t+5}\) and \(v_{t-5}\)), subtract the current step (\(v\)), and then take the cosine similarity of both vectors. With the current step now as an axis of rotation, a straight path would show a cosine similarity of -1 (since \(v_{t-5}\) will lie mirrored to \(v_{t+5}\) along \(v\)), and a complete reversal of phase would show +1. We additionally threshold for noise, not plotting any vectors’ similarities when their l2 norm is under 0.0035. We analyze the trend for the risky-safe-risky alignment training loop for the single adapter case in Figure 8. We observe a sudden spike at step-200 in local cosine similarity between LoRA adapter vectors in Figure 8 (a). However, from Figure 8 (b) to Figure 8 (c), we no longer observe this behavior. Unlike from [9], we do not see a correlation between the spike at step 200, and emergent misalignment scores in Figure 5. Moreover, each training phase involving realignment (“good”), or re-misalignment (“bad”), exhibits multiple peaks and troughs, indicating non-monotonic representational drift over time. We notice that this wave-like pattern is consistent across both A and B adapter matrices. Notably, the amplitude of these oscillations increases across successive training cycles, with later rounds (e.g., bad–good–bad) showing higher volatility and denser oscillations compared to the initial fine-tuning phase. This kind of trend holds across all examined configurations, in both the single adapter case and the all adapter case, suggesting that observing cosine similarities may not provide us additional insights into model behavior.

6 Discussion and Next Steps↩︎

Our experiments suggest that the currently reported phenomenon of Emergent Misalignment (EM) is considerably more brittle than prior work may imply. Across both single-adapter and all-adapter settings, we were able to induce misalignment through narrow fine-tuning, consistent with previous findings. However, we also found that these behavioral shifts were highly sensitive to seemingly minor properties of the training data.

Most notably, we observed that realignment initially appeared almost immediate, requiring only a small number of aligned examples to erase previously induced misalignment. However, we later found that this behavior was largely explained by a distributional artifact involving response lengths. After normalizing for this while preserving semantic content, models once again became susceptible to repeated cycles of misalignment and realignment. This suggests that conclusions regarding the permanence or reversibility of EM can be strongly affected by superficial characteristics of the training corpora.

Current mechanistic analyses also provide only limited evidence for stable internal signatures of EM. Unlike [9], we did not observe a reproducible representational phase transition that consistently predicted the onset of emergent misalignment. While we occasionally observed localized changes in LoRA cosine similarities and gradient norms, these signals were inconsistent across training phases and did not reliably coincide with behavioral transitions. Instead, representational drift appeared oscillatory throughout training regardless of whether the model was becoming more or less aligned.

Taken together, these findings suggest that current evidence for EM should be interpreted cautiously. Rather than indicating a robust and stable change in model representations, observed misalignment behavior may partially reflect dataset artifacts, evaluation choices, and other surface-level properties of fine-tuning. This does not imply that EM is absent; rather, it indicates that existing demonstrations may overestimate its robustness.

Future work should therefore focus on developing evaluation protocols that explicitly control for dataset distribution, response length, and other confounding factors, while complementing behavioral evaluations with more reliable mechanistic analyses. Establishing whether EM represents a genuine representational phenomenon or a consequence of fine-tuning artifacts will require additional benchmarks designed to disentangle these effects.

7 Appendix↩︎

7.1 Prompts↩︎

References↩︎