1 Introduction↩︎

Despite the remarkable capabilities of Large Language Models (LLMs), recent advancements reveal that they suffer from catastrophic forgetting in continual learning. This phenomenon refers to the tendency of these models to forget old knowledge when learning new tasks. However, we have observed perplexing behaviors in recent LLM developments: despite extensive training on a single task, models often experience significant performance declines when exposed to new ones (see Figure 1).

For instance, in safety alignment scenarios, LLMs trained on comprehensive safety datasets can become highly vulnerable after being exposed to only a few harmful instances. [1] suggests that fine-tuning on as few as ten identity shift examples can drastically undermine a model’s safety performance, a phenomenon we refer to as Absolutely Obedient Agent (AOA) alignment. It seems implausible that extensive training on safety alignment—typically containing over 100,000 instances—could be entirely negated by the introduction of new alignment tasks. Similarly, in continual instruction tuning [2], models may initially excel at specific tasks but experience abrupt performance declines after learning new ones.

To investigate whether the underlying knowledge is genuinely being forgotten, we sought to recover performance on older tasks. As illustrated in Figure 1, we were surprised to find that the performance on older tasks could be restored by training on merely ten safety instances or irrelevant tasks—none of which originated from the old dataset. Further details are in Section 2 and in Appendix 16.2 and 16.1. This observation challenges the conventional understanding of catastrophic forgetting and prompts us to explore whether forgetting genuinely occurs in language models or if it is, in fact, spurious.

This leads us to explore what we term spurious forgetting. We hypothesize that performance loss does not necessarily indicate a loss of knowledge, but rather a decline in task alignment—the model’s ability to effectively apply its existing knowledge to specific tasks: \[\text{Task Performance} = \text{Task Alignment} + \text{Underlying Knowledge}\] To examine this hypothesis, we conducted controlled experiments using a synthesized dataset and a randomly initialized language model, ensuring clear distinctions between new and old knowledge.

Our findings reveal that during the initial training phases of new tasks, particularly in the first 150 optimization steps, significant gradients in the loss landscape can lead to rapid declines in previous task performance. Analyzing model weights, we discovered that these initial steps often undo prior task alignment, with the bottom layers playing a crucial role. This observation is supported by our theoretical analysis, which is based on the assumption of orthogonal updates in model weights, corroborating the empirical findings. Notably, employing data replay by retaining a subset of old data facilitates a re-alignment process, restoring performance on previous tasks and suggesting that old task performance can be retrieved if we avoid the undo alignment process.

To address the issue of spurious forgetting without relying on the stored old data, we examined various continual learning techniques, including regularization-based, generative-replay-based, model-merging-based, and gradient-based methods, but found limited success. Surprisingly, a Freeze strategy—keeping the bottom layers of the model unchanged—emerged as a highly effective solution, improving task accuracy in sequential fine-tuning (SEQ) from 11% to 44%, while other techniques peaked at 22%. This strategy not only aligns with our theoretical insights but also proves effective across real-world continual learning scenarios, including safety alignment, continual instruction tuning, continual knowledge editing, and instance incremental learning.

In summary, our contributions include: (1) We are the first to identify the spurious forgetting in continual learning of language models; (2) We find that spurious forgetting is caused by the loss of task alignment instead of underlying knowledge; (3) We theoretically analyze the cause of spurious forgetting; (4) We propose Freeze strategy as an effective method for mitigating spurious forgetting.

2 Motivation: Preliminary Experiments on Spurious Forgetting↩︎

The sudden performance drops observed in LLM during continual learning raise critical questions about knowledge retention. It seems implausible that extensive training—such as 100K safety instances or 5K instances from Science QA—would be entirely negated upon the introduction of new tasks. To investigate this, we discuss our preliminary experiments in the following two continual learning scenarios:

Safety Alignment: We first reproduce the AOA alignment proposed by [1], training the LLaMa-2-7B-Chat model [3] on 10 Identity Shifting Instances. We evaluate safety performance using AdvBench [4], defined as 100% minus the jailbreak rate. Initially, the safety performance of LLaMa-2-7B-Chat is 100%, indicating strong alignment with safety data. In AOA alignment, after training the model on the 10 Identity Shifting Instances for 10 epochs, the safety performance drops to 0%. To recover the performance, we collect ten harmful instructions and use the model before AOA alignment to generates rejection responses to these harmful prompts consistently. After fine-tuning on these ten instances with rejection responses for just ten epochs, the safety performance increases from 0% to approximately 99%. Detailed experimental settings and results are provided in Appendix 16.1.

Continual Instruction Tuning: TRACE [2] serves as a challenging continual instruction tuning benchmark comprising 8 diverse tasks, including domain-specific QA, code completion, mathematical reasoning. Similar patterns are observed in TRACE, as demonstrated by the task-wise performance in [2]. We replicated these findings using the LLaMa-3-8B-Instruct on TRACE, adhering to the same task order and settings. Our results indicate that task accuracy can drop significantly—occasionally to zero—only to rebound with subsequent training. Notably, this phenomenon is not confined to specific datasets or training hyperparameters. Detailed experimental settings and results are provided in Appendix 16.2.

3 A Closer Look at Spurious Forgetting↩︎

In our quest to understand spurious forgetting, we constructed a synthetic dataset, the Biography dataset, and designed controlled experiments to investigate the underlying causes of spurious forgetting from four perspectives: performance, loss landscape, weight updates, and features.

3.1 Controlled Settings under Synthetic Dataset↩︎

Construction of the Biography Dataset. The Biography dataset consists of 200,000 synthetic individuals, each characterized by six attributes: birthday, birth city, university attended, major, company name, and company city. This dataset is divided into two subsets: pretraining data and finetuning data. The pretraining data comprises statements describing each individual’s attributes. For instance, Curtis Chase Emley recognizes his birth anniversary on May 28, 1952. The finetuning data consists of QA pairs designed for knowledge extraction, such as What is the birth date of Curtis Chase Emley? \(\backslash\)n Answer: May 28, 1952. Unless otherwise stated, we calculate the exact match accuracy for the dataset. Further details and examples are provided in Appendix 9.

Continual Learning Setting. Initially, the model is pretrained on 100,000 individuals to establish a robust knowledge foundation. Following this, we fine-tune the model on QA data from the same individuals (denoted as Task 0). We then introduce a new task (denoted as Task 1) that includes an additional 20,000 individuals unfamiliar to the model. The initial learning rates for pretraining and finetuning are set to \(1 \times 10^{-3}\) and \(5 \times 10^{-6}\), respectively. The training steps are configured to 80K for pretraining and 62.5K for finetuning. This small learning rate combined with a large number of optimization steps ensures comprehensive training of the model. An illustration of this training setup is provided in Figure 2 (b). We conduct additional experiments on more tasks (Appendix 14.1.1), varying numbers of individuals (Appendix 14.1.2), different task types (Appendix 14.1.3), and different optimizers and learning rates (Appendix 14.1.4) to show that spurious forgetting exists on general settings of continual learning.

Rationale for Using a Synthetic Dataset. Real-world datasets, such as those in the TRACE benchmark, may exhibit overlaps in knowledge acquired during either (1) pretraining and finetuning or (2) across finetuning tasks. In contrast, our constructed Biography dataset circumvents these issues by maintaining strict control over the pretraining and finetuning processes, ensuring that the knowledge between tasks remains non-overlapping. This is essential for isolating irrelevant factors that contribute to spurious forgetting. Utilizing the synthetic dataset allows us to decompose the learning processes of task alignment and the underlying knowledge. Specifically, as illustrated in Figure 2 (b), in Task 0, the model learns task alignment without acquiring new knowledge. When transitioning to Task 1, the model must simultaneously acquire new knowledge related to Task 1 while establishing task alignment for this new task, as the individuals from Task 1 are entirely novel to the model.

3.2 Spurious Forgetting from Performance Perspective↩︎

We first reproduce the spurious forgetting observed in safety alignment and continual instruction tuning scenarios. As shown in Figure 2 (a), after learning Task 1, we observe a dramatic decline in performance on Task 0, dropping from nearly 100% to around 10% within the initial 150 optimization steps. Intuitively, it is unreasonable to expect that the underlying knowledge of Task 0 would disappear within just 150 steps.

Motivated by this observation, we attempt to recover the performance on Task 0. The procedure for the recovery experiments is illustrated in Figure 2 (c). Specifically, for any checkpoint during pretraining, Task 0 and Task 1, we fine-tune the model on half of the data from Task 0 for one epoch and evaluate it on the remaining half. While training requires half of Task 0’s data, it is crucial to note that this training set does not overlap with the test data. For example, if a model lacks knowledge from the test set, the recovered performance would be close to zero. In contrast, if the model retains the knowledge, the recovered performance should be near 100%.

We conduct recovery experiments for all checkpoints from pretraining through Task 0 to Task 1. The results are plotted as the dashed line (Recovered Task 0 ACC) in Figure 2 (a). We find that recovery performance remains nearly 100% during the first 150 steps of training on Task 1, decreasing slightly to 96% by the end of Task 1. In contrast, the accuracy for Task 0 drops to approximately 10% after 150 steps, with a slight increase to 20% afterward. This result reinforces our hypothesis that spurious forgetting is not due to an actual loss of knowledge. Based on these observation, we provide a formal definition of spurious forgetting in Appendix 12.

3.3 Loss Landscape Perspective↩︎

To better understand the dynamics occurring during the first 150 steps, we visualize the test loss in a two-dimensional space spanned by the weight update direction for both the initial 150 steps and subsequent steps. The results are summarized in Figure 3 (a).

Initially, we observe a sharp decrease in the loss landscape for Task 1, coupled with a significant increase in loss for Task 0. This observation explains the dramatic drop in performance for Task 0. After the first 150 steps, a pivotal turning point in the training trajectory occurs: the model shifts right and reaches the local minimum for Task 1. From this analysis, we derive two key insights:

1. Contradictory Optimization Directions: We can distinctly identify a sharp loss landscape at the beginning of learning Task 1, where the gradient directions for Task 0 and Task 1 are opposite. This indicates that the optimization paths for Task 0 and Task 1 are contradictory at the start of training when finetuning solely on the data from Task 1 (i.e., SEQ).

2. Two-Stage Training Trajectory: The entire training trajectory can be divided into two stages. The first stage encompasses the initial 150 steps. Combined with the recovery performance shown in Figure 2 (a), we conclude that these steps effectively undo the alignment for Task 0. The second stage spans from 150 steps to the end of training, during which the model learns (1) the alignment for Task 1 and (2) the knowledge relevant to Task 1 simultaneously. In this second stage, we observe a slight decreas and then increasing trend in the loss for Task 0. When considering the accuracy of both Task 0 and Task 1 as shown in Figure 2 (a), we hypothesize that the decrease in Task 0 loss corresponds to the effect of learning Task 1 alignment, while the subsequent increase corresponds to the effect of acquiring Task 1 knowledge. Unfortunately, the learned alignment for Task 1 does not align with the direction of Task 0’s alignment, leading to the phenomenon of spurious forgetting for Task 0 (illustrated in Figure 6).

3.4 Model Weight Perspective↩︎

To further dissect the weight updates during the initial training phases, we evaluate the angle \(\theta(\Delta A, \Delta B)\) between weight updates at two training stages, denoted as \(\Delta A\) and \(\Delta B\). This angle helps us understand whether the weights are updated in the same space across these stages. For example, for the matrix in the output layer of the MLP, we first compute the column spaces for \(\Delta A\) and \(\Delta B\) using Singular Value Decomposition (SVD). The angle is then calculated between each vector in the basis of one column space and its projection onto the other. An angle close to zero indicates that the weights are updated in the same space, while an angle close to 90 degrees suggests that the updates occur in nearly orthogonal spaces. More details are provided in Appendix 14.3.

We summarize the results in Figure 4. Similar trends are observed across other model components, as detailed in Appendix 14.3. In Figure 4 (a) (blue color), the angle between updates during different pretraining stages is small, indicating that the pretraining updates occur in a consistent space. The orange color in the same figure shows that Task 0 is updated in a space nearly identical to that of pretraining, with the exception of the input embedding, suggesting that the input embedding plays a significant role in Task 0 alignment.

Figure 4 (b) (blue color) indicates that the first 150 steps in Task 1 update weights in a space close to that of Task 0. This suggests that the primary effect of these initial steps is to undo the Task 0 alignment. The orange color in the same figure reveals that subsequent steps in Task 1 update weights in a distinctly different space, particularly affecting the bottom layers, including input embeddings.

Based on the findings from Section 3.3, we conclude that the bottom layers, including the input embedding layers, are crucial for task alignment. In other words, the near-orthogonal weight updates in these layers contribute to the differences in alignment between Task 0 and Task 1, ultimately leading to the spurious forgetting observed in Task 0 (illustrated in Figure 6).

3.5 Feature Perspective↩︎

In this section, we investigate how feature representations change in the context of spurious forgetting. Specifically, for each training stage, we compute the differences in the hidden states (features) at each Transformer layer and the differences in the leading principal component of these features. A small shift in the principal component indicates that the feature changes are occurring in a direction close to orthogonality relative to the original features, suggesting that the modified features may largely retain their previous representation and could potentially be recovered by reversing the shift.

The results are summarized in Figure 5. Notably, we observe a significant shift in the principal component in the first three cases (from Figure 5 (a) to 5 (f)), while Figure 5 (g) and 5 (h) shows nearly no shift. This pattern indicates that the learning and unlearning of task alignment—occurring in Task 0, the first 150 steps of Task 1, and the latter steps of Task 1—typically leads to changes in feature representations, as reflected in the principal component shifts.

Interestingly, when we consider the combined effects of the first 150 steps and the latter ones, the shift in the principal component disappears. This suggests that, despite different task alignments, there exists a shared pattern in the feature representations for both tasks, as both align the model toward QA tasks. In other words, in the case of Task 0 and Task 1, the shifts in principal components can cancel each other out, resulting in no net change in the principal component when considering the entire learning process of Task 1. This implies that the task alignments of Task 0 and Task 1 are not fundamentally contradictory. Furthermore, we observe that the shifts in principal components appear to originate in the bottom layers and propagate to the upper layers.

3.6 Summary↩︎

As illustrated in Figure 6, the root cause of spurious forgetting stems from the difference between the Task 0 and Task 1 alignments. In Section 5, we will demonstrate that data replay, or our proposed Freeze , enables the model to learn aligned Task 0 and Task 1 alignments.

4 Theoretical Analysis↩︎

This section presents a theoretical framework that underpins our findings on spurious forgetting. We establish that the observed spurious forgetting are largely a result of orthogonal updates in model weights, which cause shifts in the feature that do not necessarily reflect a loss of knowledge, as these shifts are nearly orthogonal to the principal component. Additionally, by analyzing the bounds on the shift in the final output, we demonstrate that freezing the bottom layers may mitigate these issues. The full theoretical results and the proof are provided in Appendix 13.

5 Solution to Spurious Forgetting↩︎

5.1 Revisiting Existing Techniques for Forgetting↩︎

Having gained a deeper understanding of spurious forgetting, we now investigate whether existing techniques for continual learning can effectively mitigate its effects. In addition to data replay (REPLAY), we consider four representative methods from distinct categories: EWC [10] (a regularization-based method), LAMOL [11] (a generative replay-based method), Task Vector [12] (a model-merging-based method), and Gradient Projection [13] (a gradient-based method). Additionally, we assess direct fine-tuning on new tasks as a lower bound for continual learning, denoted as SEQ. Detailed introductions to each method can be found in Appendix 15.

r7cm

The results presented in Table 1 indicate that none of the existing methods achieve satisfactory accuracy on Task 0 when Task 1 accuracy exceeds 99%. We now analyze the reasons behind the shortcomings of each method in addressing spurious forgetting:

1. EWC: As depicted in Figure 7 (c), the correlation between the Fisher matrix (which indicates parameter importance in EWC) and the weight update angle \(\theta(\Delta Task1_0^{150}, \Delta Task1_{150}^{final})\) across model components (including embedding, self-attention, and MLP) is weak. This suggests that EWC inadequately identifies the bottom layers as critical parameters contribute to the loss of task alignment.

2. LAMOL: After learning a new task, we generate 24,000 pseudo old samples (20% of the new data). Figure 7 (e) reveals that the quality of these pseudo samples is low. Following the filtering of invalid format samples (V), duplicated samples (D), and samples with no exact match to real old data (M[Q] and M[Q&A]), fewer than 20% of the samples remain. In the absence of real old data, this implies that nearly half of the pseudo samples are hallucinated by the model, leading to subpar performance. Similar findings were observed in our experiments involving additional tasks (Appendix 14.1.1).

3. Task Vector: To counteract the alignment process of Task 0, we attempt to negate weight updates from the first \(\{12,14,16,18\}\) epochs during Task 1 learning (Task Vec. End Epoch). We apply this task vector across various model checkpoints from epochs \(\{1,2,\ldots,25\}\), adjusting the scale \(\alpha \in \{0.16,0.18,\cdots,0.8,1.0\}\). As shown in Figure 7 (f), a trade-off exists between Task 0 and Task 1 accuracies, with the best average accuracy being (95.76, 30.75). However, when considering Task 1 accuracy above 99%, the performance drops to (99.41, 22.60). Despite extensive hyperparameter tuning, results remain unsatisfactory. A visualization of the loss landscape in Appendix 14.2 demonstrates that no viable solution is attainable along the SEQ trajectory, elucidating the shortcomings of the Task Vector approach.

4. Gradient Projection: We aim to avoid the process of the undo Task 0 alignment by first storing the average gradient direction over 10 trials. Subsequently, we retrain the model and project the gradient of various components to the orthogonal direction of the undo Task 0 alignment. Motivated by the loss landscape depicted in Figure 3 (a), we attempt to guide the model to directly learn Task 1 knowledge without reverting to Task 0 alignment. However, as shown in Figure 7 (d), all variants fail to effectively mitigate the undo Task 0 alignment, with the best variant achieving only 13.34%. This is attributed to the diverse nature of the undo alignment direction, as evidenced by the average cosine similarity of these directions over the 10 trials, which is merely 0.4.

5.2 Mitigating Spurious Forgetting by Freezing Bottom Layers↩︎

The previous analysis reveals that existing continual learning techniques struggle with spurious forgetting, primarily because they fail to mitigate the undo alignment from Task 0. This raises the question: how can we effectively achieve this?

Intuition from Data Replay. To explore this, we revisit the recovery experiments for Task 0 discussed in Section 3.2, where training on a portion of Task 0 data led to performance improvements. This suggests that data replay could be a viable technique to counteract spurious forgetting, as training on a subset of Task 0 data may help retrieve the Task 0 alignment. Table 1 corroborates this, showing that retaining old data from Task 0 significantly enhances performance on both Task 0 and Task 1. The loss landscape in Figure 3 (b) illustrates that while the model initially undoes the Task 0 alignment when optimizing new and old data, it subsequently aligns with Task 0 during the learning process for Task 1, indicating a re-alignment toward Task 0. Detailed explanation is provided in Appendix 14.2.

Intuition from Model Updates. Despite storing up to 20% of old data, the undo alignment from Task 0 remains unavoidable during initial training steps. To address this challenge, we turn to insights from model weight updates discussed in Section 3.4. Our findings indicate that the bottom layers play a crucial role in the process of learning and unlearning task alignments. Evidence from feature shifts (Figure 5) and Proposition 6 suggest that shifts in features originate from the bottom layers and accumulate upward. This leads to a straightforward solution: freezing all components in the bottom layers, including input embedding layers, denoted as Freeze .

Free Lunch for Mitigating Spurious Forgetting. To test this hypothesis, we apply Freeze to the Biography dataset, and the results are summarized in Table 1. Surprisingly, Freeze proves highly effective, enhancing SEQ performance from 11% to 44% while updating less than half of the parameters. This approach provides an effective solution for mitigating spurious forgetting, particularly in scenarios where no old data is available, serving as a valuable free lunch. Figure 7 (b) indicates a clear trend: as the number of frozen layers increases from 1 to 9, the undo alignment process for Task 0 is mitigated. However, this also slows down the learning of Task 1 and diminishes model capacity. Notably, as more layers are frozen, significant forgetting occurs in the late training stages, suggesting a trade-off between stability and plasticity. By employing an early stopping strategy to capture the model when Task 1 accuracy exceeds 99%, we observe improved performance, as detailed in Table 1.

In summary, the effectiveness of Freeze suggests that freezing the bottom layers can substantially mitigate the undo alignment from Task 0, thereby encouraging the model to reuse the Task 0 alignment while learning Task 1 (illustrated in Figure 6). However, a significant performance gap remains between Freeze and data replay, highlighting the persistent challenges associated with spurious forgetting.

5.3 Application on Real-World Scenarios↩︎

\begin{table}[!t]

\begin{tabular}{l|c|c|cc|cc}
\toprule
\multicolumn{1}{c|}{\textbf{Scenario}} & \textbf{SA} & \textbf{CIT} & \multicolumn{2}{c|}{\textbf{CKE}} & \multicolumn{2}{c}{\textbf{IIL}} \\
\midrule
\multicolumn{1}{c|}{\textbf{Metric}} & \textbf{Jailbreak Rate ($\downarrow$)}\unboldmath{} & \textbf{Test Score ($\uparrow$)}\unboldmath{} & \textbf{Efficacy ($\uparrow$)}\unboldmath{} & \textbf{Paraphrase ($\uparrow$)}\unboldmath{} & \textbf{Mem. Acc.  ($\uparrow$)}\unboldmath{} & \textbf{Gen. Acc.  ($\uparrow$)}\unboldmath{} \\
\midrule
\rowcolor[rgb]{ .949,  .949,  .949} SEQ   & 99.80\tiny{±0.20} & 47.38\tiny{±0.37} & 62.47\tiny{±0.49} & 58.24\tiny{±0.53} & 35.98\tiny{±0.17} & 12.61\tiny{±0.14} \\
\Freeze (1 layers, 1 task) & /     & 47.84\tiny{±0.56} & \textbf{70.88\tiny{±0.69}} & 64.19\tiny{±0.96} & 37.00\tiny{±0.23} & 13.06\tiny{±0.10} \\
\rowcolor[rgb]{ .949,  .949,  .949} \Freeze (2 layers, 1 task) & /     & 48.78\tiny{±1.24} & 70.65\tiny{±0.45} & \textbf{68.60\tiny{±0.35}} & \textbf{42.18\tiny{±0.05}} & \textbf{14.19\tiny{±0.21}} \\
\Freeze (3 layers, 1 task) & /     & 50.33\tiny{±0.73} & 56.31\tiny{±0.84} & 42.04\tiny{±0.55} & 39.64\tiny{±0.33} & 9.36\tiny{±0.17} \\
\rowcolor[rgb]{ .949,  .949,  .949} \Freeze (3 layers) & 79.61\tiny{±6.53} & \textbf{53.20\tiny{±0.41}} & 53.75\tiny{±0.78} & 41.24\tiny{±0.72} & 33.74\tiny{±0.19} & 8.32\tiny{±0.11} \\
\Freeze (6 layers) & \textbf{1.15\tiny{±0.16}} & 51.91\tiny{±0.55} & 51.49\tiny{±0.86} & 42.74\tiny{±0.34} & 30.27\tiny{±0.41} & 7.18\tiny{±0.08} \\
\bottomrule
\end{tabular}    \caption{Summary of the performance of \Freeze on four real-world scenarios. A higher value is better for $\uparrow$ (higher) and $\downarrow$ (lower) metrics. For the CIT, CKE, and IIL scenarios, metrics are averaged after the model has learned the final task, with task-wise results detailed in Appendix \ref{sec:sec:appendix95additional95results95on95realworld95scenarios}. The percent sign (\%) for all metrics is omitted. \Freeze (3 layers, 1 task) indicates freezing the bottom three layers after learning the first task, while \Freeze (6 layers) denotes freezing the bottom six layers from the bottom. \begin{rebuttalenv}Comparison with LAMOL and EWC are in Table \ref{tbl:tab:sota95realworld95appendix95ewc95lamol}.\end{rebuttalenv}}\label{tab:sota95realworld95dataset95main95paper}

\end{table}

We evaluate the performance of Freeze across four real-world continual learning scenarios with diverse task types, backbones, and training instances: (1) Safety Alignment (SA); (2) Continual Instruction Tuning (CIT); (3) Continual Knowledge Editing (CKE); and (4) Instance Incremental Learning (IIL). The experimental settings are summarized in Table 2, with further details provided in Appendix 16. In Appendix 16.5, we also evaluate Freeze when supervised finetuning (SFT) on code and math datasets on various architecture of LLMs.

r7cm

We investigate a variant of Freeze that involves freezing the bottom layers after learning the first task (denoted as Freeze (n layers, 1 task)), as spurious forgetting may occur starting from the second task. Results presented in Table ¿tbl:tab:sota95realworld95dataset95main95paper? indicate that Freeze significantly enhances performance compared to SEQ, highlighting the presence of spurious forgetting in these scenarios.

It is important to clarify that the results in SA are not intended to demonstrate that Freeze is a defence method to jailbreak attacks; rather, they aim to establish that spurious forgetting exists in SA and the safety performance can be better preserved with Freeze .

We have two key insights: (1) When new tasks share similar formats and knowledge with those encountered by LLMs (e.g., safety alignment and instruction-tuning data in SA and CIT), spurious forgetting occurs from the first task. The reason is that LLMs have already learned the task alignment during the post-pretraining phase (e.g., supervised fine-tuning, safety alignment). In such cases, freezing more layers (e.g., 3 or 6) proves beneficial since less plasticity is required. (2) Conversely, when new tasks present different formats and introduce new knowledge (e.g., CKE and IIL), spurious forgetting tends to occur after the first task. This is because LLMs have had limited exposure to the new task alignment (e.g., specific QA format) during the post-pretraining phase. Consequently, Freeze should be implemented after the first task, with fewer layers (e.g., 1 or 2) frozen to maintain the necessary plasticity. In summary, spurious forgetting is likely to occur when task types or formats are similar. Therefore, Freeze should be employed when mismatches in task alignment between similar task types arise.

6 Related Work↩︎

The rapid development of LLMs has sparked interest in their behavior under continual learning, yet this area remains underexplored. (1) Some studies indicate that LLMs are susceptible to catastrophic forgetting [14], [15]. (2) Conversely, other research highlights the robustness of LLMs against catastrophic forgetting. Notably, [16] and [17] demonstrate that LLMs possess strong anti-forgetting capabilities in sequential fine-tuning contexts. These findings align with our own observations that the core knowledge of LLMs tends to be more resilient than their task alignment in continual learning scenarios. Further discussions on forgetting mechanisms, memorization dynamics, and parameter-freezing strategy are in Appendix 8.

7 Conclusion↩︎

In this work, we identified spurious forgetting as a pivotal factor affecting language model performance during continual learning. Our insights suggest that task alignment is more critical than mere knowledge retention, as demonstrated in the controlled experiments and theoretical analyses. We introduced the Freeze strategy, which effectively mitigates spurious forgetting, thereby enhancing performance across various learning scenarios.

8 Related Work↩︎

Research on continual learning in LLMs is typically categorized into two main areas: (1) Forgetting Mechanisms and (2) Memorization Dynamics. Besides, we discuss recent works in continual learning that utilize the parameter-freezing strategy.

8.1 Forgetting Mechanisms↩︎

Early studies on catastrophic forgetting, such as those by [18] and [10], primarily assessed the degradation of performance on previously learned tasks. More recent research has employed probing techniques to quantify forgetting in continual learning contexts. For instance, [19] utilize linear probing to identify representation shifts resulting from parameter updates. Additionally, [20] conduct layer-wise probing on BERT, revealing catastrophic forgetting in its upper and middle layers. [21] further elucidate the relationship between the retention of prior knowledge and the efficiency of learning new tasks using k-shot linear probing. These studies underscore the importance of understanding how forgetting mechanisms manifest in LLMs.

8.2 Memorization Dynamics↩︎

The dynamics of memorization in LLMs have garnered comparatively less attention. Research by [22] and [23] highlights that models like GPT-2 can memorize sensitive information during pretraining, raising critical privacy concerns. Furthermore, [23] show that larger models not only memorize information more rapidly but also exhibit higher forgetting baselines. Additionally, [24] discuss the unpredictability of which training samples LLMs will memorize, while [25] explore how transformers incrementally learn new knowledge, observing an increase in rank among both trained and initial weights. Collectively, these contributions enhance our understanding of memorization from both textual and model weight perspectives.

8.3 Parameter Freezing in Continual Learning↩︎

Parameter freezing is a straightforward strategy for mitigating catastrophic forgetting. Architecture-based methods [26]–[29] can be considered a form of parameter freezing, as they typically train only a small proportion of parameters, such as LoRA [26], prompts [27]–[29], or adapters [30]. However, these methods generally capture less knowledge compared to full finetuning [31]. Additionally, [17], [32] propose freezing the backbone of LLMs and training only classifiers during continual learning, but their experiments are limited to classification tasks.

Model expansion techniques [33] also effectively prevent forgetting by freezing old layers and adding new layers for subsequent tasks. However, this approach is impractical for real-world applications due to the resource overhead of expanding the model for each new task.

Unlike the parameter-freezing strategies discussed above, the proposed Freeze method can be applied to full finetuning of LLMs in real-world continual learning scenarios, such as alignment and continual instruction tuning. This distinguishes Freeze as a versatile and practical solution for addressing catastrophic forgetting in diverse settings.

9 Dataset Construction↩︎

In this paper, the model is initially pre-trained using the synthetic Biography dataset, followed by fine-tuning on the corresponding QA dataset. We adhere to the dataset construction procedure proposed in [34], and briefly describe the dataset construction procedure here to make the paper be self-contained.

9.1 Biography Dataset↩︎

The synthetic Biography dataset comprises profiles of \(N=200,000\) synthetic individuals, each distinguished by their full name. Every individual is characterized by six attributes: birthday, birth city, university attended, major pursued at the university, and the name and city of the individual’s current company. We begin by determining the name and attributes of an individual, followed by generating six-sentence biographical text entries for each individual. Each sentence is randomly chosen from 50 distinct templates. We will introduce the construction procedure of names and attributes, the construction procedure of templates, In the following sections, we will detail the procedures for constructing names and attributes, templates, and biographical text entries.

9.1.1 The Construction of Names and Attributes↩︎

Each individual has a name and six attributes. The name is composed of three parts: first name, middle name, and last name. Each attribute and name component is selected independently and randomly from its corresponding pool, where values are uniformly distributed and reflect real-world data. While the language model parameters are randomly initialized before pre-training, we maintain the original tokenization rules. Instead of populating the pool with randomly generated strings, we use real-world data, as it shortens the tokenized length of each name component or attribute, thereby reducing training costs. Additionally, using real-world data improves the readability of both the dataset and model outputs.

Name

Each components of names are selected from a separate pool. For first names and middle names, we first randomly select 800 common first names from a UCI Machine Learning Dataset³, then divide them into two pools, corresponding to the pool of first and middle names. For last names, we select 1000 names from a Github repository⁴ to construct the corresponding pool. Rejection sampling is applied to ensure all \(N\) individuals have unique full names.

Birthday

An individual’s birthday consists year, month and day.Years range from 1900 to 2099, months are selected from the 12 months, and days are chosen between 1 and 28.

Birth City

An individual’s birth city is selected from a pool of the 200 most populous cities in the US⁵. Cities are identified with their respective state abbreviations, such as New York, NY and Los Angeles, CA.

University

An individual’s university attended is selected from a pool of the 300 well-known research universities⁶. Notably, many of these universities share the same prefix, such as University of California, Berkeley, University of California, Irvine, University of California, Davis and so on. Among the 300 universities, 115 begin with the prefix University of.

Major

The major that an individual pursued at the university is selected from 100 popular college majors, including Nursing, Liberal Arts, and Business Administration.

Company Name

The name of the company that an individual is employed by is selected from the top 263 companies on the Fortune 500 list of 2017⁷. Famous companies such as Walmart and Apple are included.

Company City

The company city is an attribute that depends on the company name. If two individuals share the same company name, they will also have the same company city attribute. The city for each company is determined based on information from the Fortune 500 list and is also identified with its respective state abbreviation.

Since each attribute is selected independently, the combinations of attributes may sometimes be unrealistic. For instance, an individual born in 1900 could be associated with a company founded in 2000. However, this would not confuse the language model which is trained from scratch.

9.1.2 The Construction of Templates↩︎

In Biography dataset, each sentence of a biography text entry illuminates a distinct attribute of the individual. The sentence is obtained by filling the full individual name and attribute in the corresponding templates. To increase the diversity of the dataset, each template used in an entry is selected from the corresponding template pool. We have 50 templates for each attribute.

We employ GPT-4o ([35]) to construct the templates. For each attribute, we first collect three template examples from [34], then use few-shot learning ([36]) to generate the template pool. Each individual implicitly has a gender attribute, which determines whether they should be referred to as his, him, or her in the template. Since in English, her can function as both a possessive and an object pronoun, if we generate templates assuming the individual is female, it becomes difficult to simply replace her with his or him when applying the template to a male individual. Therefore, during the template construction process, we assume the individual is male by default. Here is the prompt used to construct the templates pool for attribute Birthday.

9.1.3 The Construction of Biographical Text Entries↩︎

In Biography dataset, each individual corresponds to five biography text entries. For each entry of each individual, after getting the templates for six attributes, we can obtain sentences of entries by filling the templates by individual name and attribute value. The order of six sentences in an entry is determined randomly.

Below are the five biography text entries for the first individual of the Biography dataset. The attribute value of the individual in each sentence is highlighted by blue.

9.2 QA Dataset↩︎

The QA dataset is used to extract the knowledge of a language model which has been pre-trained on the Biography dataset. We perform the knowledge extraction using question and answer (QA) framework. For each individual, we pose six questions targeting their six unique attributes. Below are the QA pairs of the first individual. In each pair, the model is required to generate an answer conditioned on the prompt, which is made up of a and a prompt indicator (Answer:) where the model is expected to provide the correct response. The attribute value in each QA pair is highlighted by blue.

10 Pre-Training and Fine-Tuning↩︎

In the experiments mentioned in the main paper, the language model goes through three periods: it is first trained on the Biography dataset corresponding to the first 100k individuals, then trained on the QA dataset corresponding to the first 50k individuals, and finally trained on the QA dataset corresponding to the individuals from 100k to 120k.

The training process can be seen as the method of bestowing knowledge to the language model. In our experiment, since the model is trained from scratch, we assume that it initially contains no knowledge whatsoever. This knowledge can manifest in various forms, such as grammar rules or the location of a company. However, in this context, we are specifically concerned with the knowledge that captures the meaningful connections between an individual and their six corresponding attributes, which is defined in the Biography dataset. A biography text entry in the dataset can be viewed as a collection of connections between an individual and the corresponding six attributes. We will use the term knowledge to specifically refer to this type of connection. It should be noted that the QA dataset also contains knowledge, since a single QA pair also connect an individual to a particular attribute. We will discuss the method of assessing the model’s level of knowledge acquisition in Appendix 11.1.

In the following sections, we will divide the period into pre-training stage and fine-tuning stage. We also analyze the function of the datasets used in each period based on the knowledge contained within the language model and the datasets. The implementation details of two stages are included in 11.2.

10.1 Pre-Training Stage↩︎

We consider the first period as pre-training stage, where the language model is bestowed with knowledge from the Biography dataset. We use the standard language modeling objective function ([37]) to train the model from scratch. After the pre-training stage, the knowledge is encoded as model parameters. We regard this stage as pre-training not only because the training paradigm is consistent with traditional pre-training, but also because the model acquires a broad range of knowledge after completing this period.

10.2 Fine-Tuning Stage↩︎

We consider the second period as fine-tuning stage, where the language model is required to generate the answer conditioning on the prompt. In the second period, the language model learns to extract the knowledge from the Biography dataset and manipulate it for answering questions. The individuals involved in this period are the subset of the individuals involved in the first period, so the language model is not bestowed with any new knowledge at all. Instead, the QA dataset is used to align the knowledge already encoded in the model’s parameters with the QA format.

We want to emphasize that although the language model is pre-trained on 100,000 individuals in the first period, we do not to use QA pairs for all individuals to fine-tune the model. By fine-tuning the model with only a subset of the individuals’ QA dataset, we can utilize the remaining individuals’ QA dataset to investigate whether the model is aligning knowledge encoded or learning new knowledge. If the model performs well on remaining individuals’ QA dataset, it indicates successful alignment. Otherwise, it indicates that the model simply learns new knowledge from the QA dataset.

We also consider the third period as a fine-tuning stage, as the training paradigm is consistent with that of the previous period, which is also regarded as fine-tuning. While the QA dataset is used in both the second and third periods, its function differs between these stages. In the third period, the language model is fine-tuned on the QA dataset corresponding to individuals not involved in the previous two periods. The model is bestowed with knowledge from the QA pairs of new individuals.

11 Implementation Details↩︎

In this section, we are going to introduce the detail of evaluation and experiment.

11.1 Evaluation Details↩︎

In Appendix 10, we define knowledge as the meaningful connections between an individual and their six corresponding attributes. This definition enables us to quantitatively assess the model’s level of knowledge acquisition. We consider that the model has acquired a piece of knowledge (i.e., a connection between an individual and an attribute) if and only if it can effectively utilize that knowledge to answer questions in the QA dataset. Consequently, the model’s performance on the QA dataset serves as an indicator of its level of knowledge acquisition. To measure this performance, we use three metrics: soft first-token accuracy, hard first-token accuracy, and exact match accuracy. Additionally, we monitor the first two metrics during the pre-training process to ensure that the model is trained comprehensively. Below we will introduce the metrics in detail.

11.1.1 Soft First-Token Accuracy↩︎

We monitor the model’s next-token-prediction accuracy on the first token of each of the six attributes during the training process. To evaluate the model’s level of knowledge acquisition, we calculate the generation probability of the correct token. Soft-token accuracy captures the nuanced changes in the model’s knowledge acquisition. We assess the model’s soft-token accuracy on the training set during both the pre-training and Task 1 fine-tuning processes to ensure comprehensive training.

11.1.2 Hard First-Token Accuracy↩︎

The process for calculating hard first-token accuracy closely resembles that of soft first-token accuracy. During evaluation, we employ a greedy decoding strategy [38], and the model is considered to have acquired a piece of knowledge if the generation probability of the correct token is the highest among all tokens. In contrast to soft first-token accuracy, hard first-token accuracy provides a more accurate reflection of the model’s performance in real-world applications.

11.1.3 Exact Match Accuracy↩︎

We apply a greedy decoding strategy when calculating exact match accuracy as well. The model is deemed to have acquired a piece of knowledge if it correctly generates all tokens of an attribute.

While exact match accuracy provides a precise reflection of the model’s performance, its computational demands are significantly higher than those for soft first-token accuracy or hard first-token accuracy. In Figure 2 (a), we use soft first-token accuracy to evaluate the model’s performance during the fine-tuning process on Task 0 and Task 1, while exact match accuracy is employed to assess performance after recovery. Similar evaluation strategies are also used in the experiments detailed in Appendix 14.

11.2 Experimental Details↩︎

In this section, we will briefly introduce the implementation details of the pre-training stage and fine-tuning stage. All experiments are conducted using PyTorch.

11.2.1 Experimental Details of Pre-training Stage↩︎

For pre-training, we employed a conventional set of optimization parameters: the AdamW optimizer with a weight decay of 0.1, \(\epsilon = 10^{-6}\), an initial learning rate of 0.001, a 1000-step linear warmup, and cosine learning rate decay (from 0.001 decreasing to 0.0001). There are a total of 80,000 training steps in the pre-training stage and the batch size is set to 96. In each epoch, we first shuffle the order of the biographical text entries corresponding to all involved individuals, then concatenate these entries to create sequences of 512 tokens, using a standard <EOS> token to separate different entries. The pre-training experiments are executed on an NVIDIA A800 80GB GPU.

11.2.2 Experimental Details of Fine-tuning Stage↩︎

All parameters of the language model are updated during the fine-tuning stage. We employ the AdamW optimizer with a weight decay of 0.01, \(\epsilon = 10^{-6}\), an initial learning rate of \(5 \times 10^{-6}\), and cosine learning rate decay (from \(5 \times 10^{-6}\) to \(4.5 \times 10^{-6}\)). There are 62,500 training steps in the fine-tuning stage and the batch size is set to 48. The fine-tuning experiments are executed on NVIDIA RTX 3090 GPUs.

12 Formal Definition of Spurious Forgetting↩︎

Let \(\mathcal{D}_{A}\) and \(\mathcal{D}_{B}\) be the datasets corresponding to tasks \(A\) and \(B\), respectively, such that there is no knowledge overlap between them. Let \(\mathcal{M}_{A}\) be the model trained on \(\mathcal{D}_{A}\), and \(\mathcal{M}_{B}\) be the model obtained by finetuning \(\mathcal{M}_{A}\) on \(\mathcal{D}_{B}\). Additionally, let \(\mathcal{D}_{A'}\) be a dataset with no knowledge overlap with either \(\mathcal{D}_{A}\) or \(\mathcal{D}_{B}\), and let \(\mathcal{M}_{A'}\) be the model obtained by further finetuning \(\mathcal{M}_{B}\) on \(\mathcal{D}_{A'}\).

13 Theoretical Results and Proof↩︎

14 Additional Results on Biography Dataset↩︎

14.1 Spurious Forgetting under Performance Perspective↩︎

In Section 3.2, Our experiment reveals that the decline of the model’s performance on Task 0 is dramatic and can be recovered by using half of Task 0’s data. We are curious whether this phenomenon is commonly observed in other continuous learning experimental scenarios. To achieve the goal, we conduct additional experiments on the Biography dataset. Our experiments indicate that spurious forgetting occurs across various experimental settings with different the number of tasks (Appendix 14.1.1), individuals (Appendix 14.1.2), task types (Appendix 14.1.3), and optimizers and learning rates (Appendix 14.1.4). In the experiments described below, the learning rates and training steps for both pre-training and fine-tuning are consistent with those mentioned in Section 3.1, unless stated otherwise.

14.1.1 Extended Setting 1: More Tasks↩︎

In this experiment, we investigate whether spurious forgetting occurs consistently with an increasing number of tasks. The model is pre-trained on 100,000 individuals, then fine-tuned sequential on five tasks, each involves 20,000 individual that are unfamiliar to the model. The training steps for Tasks 2 and Task 3 are set to 62.5K, while the training steps for Task 4 and Task 5 are set to 80K. Compared to the previous tasks, the last two tasks require a higher number of training steps. This is because we found that, as the number of tasks increases, the same amount of training steps is inadequate for comprehensive training. The learning rates of all tasks are set to \(5\times10^{-6}\). The results of the experiment are shown in Figure 8. Our experiments show that with an increasing number of tasks, spurious forgetting still persists in subsequent tasks.

14.1.2 Extended Setting 2: Varying Numbers of Individuals↩︎

In this experiment, we investigate whether spurious forgetting occurs consistently when the number of individuals varies. The experiment setting is the same as Appendix 14.1.1 except that the number of individuals in each task. We conducted three sets of experiments, with the number of individuals in each set being 20, 200, and 2000, respectively. The results of the experiment are shown in Figure 9, 10. Our experiments show that when the number of individuals in each task is low, spurious forgetting is not pronounced. This occurs because the model has to learn fewer new individuals in a single task, which enables it to retain knowledge from the previous task more effectively, making the forgetting phenomenon less noticeable and consequently reducing the occurrence of spurious forgetting. However, as the number of individuals in each task increases, spurious forgetting becomes more pronounced along with the emergence of the forgetting phenomenon.

More specifically, the findings and analysis are summarized as follows:

• Task Size and Model Capability: When the new tasks are significantly smaller or simpler compared to the model’s capacity (e.g., pretraining on 100K individuals), the model can easily adapt to the new tasks by transferring existing knowledge without requiring large updates. For example, as illustrated in the trajectory of Task 1’s loss in Figure 3 (a), the model requires large updates to adapt to Task 1.

• Task Alignment Perspective: From the perspective of task alignment, a common direction between the task alignments of new and old tasks can be easily identified when the sample size is small or the task is simple. This will correspond to a larger intersection area in Figure 6.

• When Spurious Forgetting Occurs? Spurious forgetting is more likely to arise when new tasks are sufficiently challenging or have an adequate number of training samples relative to the model’s existing capacity. In such scenarios, learning a common direction for task alignment across tasks becomes less trivial, potentially leading to task misalignment and forgetting.

14.1.3 Extended Setting 3: Different Task Types↩︎

In this experiment, we investigate whether spurious forgetting occurs consistently across different task types. To achieve this, we introduce a new type of QA task in which each question comprises two sub-questions, each paired with its corresponding answer. We refer to these question-answer pairs as compound QA pairs and designate the dataset containing them as the Compound QA Dataset. Below are the compound QA pairs of the first individual. The attribute value in each QA pair is highlighted by blue.

In this experiment, the model is first pre-trained on a dataset of 100,000 individuals. Following pre-training, the model is fine-tuned on the corresponding QA pairs from the dataset detailed in Appendix 9.2. Next, the model undergoes a second fine-tuning on compound QA pairs related to an additional 20,000 individuals that are unfamiliar to the model. We refer to the initial fine-tuning as Task 0 and the subsequent fine-tuning as Task 1. The results of the experiment are illustrated in Figure 11. Our findings indicate that spurious forgetting still occurs when the types of tasks are different.

14.1.4 Extended Setting 4: Different Optimizers and Learning Rates↩︎

To demonstrate that spurious forgetting is not caused by instability at the beginning of finetuning, we experimented with different optimizers, including AdamW [41] and SGD, as well as various initial learning rates ranging from \(1 \times 10^{-4}\) to \(1 \times 10^{-7}\). Our findings reveal that SGD converges much slower than AdamW. Specifically, when using SGD with a learning rate of \(1 \times 10^{-5}\) or smaller, the training loss on Task 0 does not decrease even after 10,000 steps. In contrast, AdamW successfully facilitates training with learning rates as low as \(5 \times 10^{-7}\).

The results, presented in Figure 12, indicate that spurious forgetting occurs at different training steps under various combinations of optimizers and learning rates. For instance, when using SGD, spurious forgetting is observed after approximately 3,000 steps with a learning rate of \(1 \times 10^{-4}\) and after 6,000 steps with a learning rate of \(5 \times 10^{-5}\). In the case of AdamW, spurious forgetting occurs after 200, 600, and 1,000 steps for learning rates of \(5 \times 10^{-6}\), \(1 \times 10^{-6}\), and \(5 \times 10^{-7}\), respectively. These findings confirm that spurious forgetting is not attributable to initial instability during finetuning.

14.2 Spurious Forgetting under Loss Landscape Perspective↩︎

We visualize the loss landscape for sequential fine-tuning (SEQ), data replay using 20% old data, and data replay utilizing 50% old data in Figure 13. The key findings align with those discussed in Section 3.3.

The trajectory in the data replay setting (Figure 13 (b)(c) and Figure 3 (b)) can be interpreted as follows:

• Beginning of the Trajectory: Initially, the trajectory moves in the opposite direction of Task 0 alignment (Y-axis) during the first 150 steps. This phase corresponds to the undoing of Task 0 alignment.

• Middle of the Trajectory: The trajectory starts to shift along the direction of Task 1 knowledge (X-axis) while still moving in the opposite direction of Task 0 alignment. This is because data replay progressively encourages Task 0 re-alignment throughout the learning process of Task 1, as the model searches for a balance between Task 1 alignment and Task 0 alignment (illustrated in 6).

• Final Phase: The trajectory continues to move in the direction of Task 1 knowledge (X-axis) while gradually finding a common direction for Task 0 alignment and Task 1 alignment. This final phase reflects the model’s attempt to reconcile alignment for both tasks.

Additionally, as more old data is incorporated, the loss landscapes for both Task 0 and Task 1 exhibit increased flatness. This flattening can be interpreted from two perspectives: (1) Data replay promotes the identification of common directions where the alignments of Task 0 and Task 1 converge; (2) Data replay facilitates the model in discovering improved solutions for Task 1 that are situated near the flat minima of Task 0.

Additionally, we visualize the average loss for both Task 0 and Task 1. The results indicate that SEQ fails to uncover effective solutions within its loss landscape, with the optimal average loss is around 0.8. This observation elucidates why the Task Vector struggles to achieve satisfactory performance, as discussed in Section 5.

14.3 Spurious Forgetting under Model Weight Perspective↩︎

In Section 3.3, we visualized the loss landscape of the output matrix in the MLP. Here, we extend this analysis to other model components. We consider eight types of 2-dimensional matrices, including the output matrix in the MLP (denoted as mlp.dense_4h_to_h), the input matrix in the MLP (denoted as mlp.dense_h_to_4h), as well as the output, query, key, and value matrices in the self-attention layers (denoted as attention.dense, attention.query, attention.key, attention.value), along with the input and output embedding layers.

Focusing on the feature space of the residual stream, we apply Singular Value Decomposition (SVD) to extract the column spaces of mlp.dense_4h_to_h and attention.dense, while obtaining the row spaces for the others. For instance, the shape of mlp.dense_4h_to_h is \((3072,768)\), with a feature dimension (i.e., hidden states) of \(d\_model=768\). Applying SVD yields a column space composed of \(r\) 768-dimensional vectors that account for 99% of the total variance, where \(r\) denotes the empirical rank of the matrix. The remaining \(768-r\) vectors form the left-null space of mlp.dense_4h_to_h.

To evaluate the angle \(\theta(\Delta A, \Delta B)\) between weight updates at two training stages, denoted as \(\Delta A\) and \(\Delta B\), we first determine the vectors \(r_A\) and \(r_B\) in the column spaces of \(\Delta A\) and \(\Delta B\), respectively. For each vector \(u\) in column space of \(\Delta B\), we project it into the column space of \(\Delta A\), obtaining \(\tilde{u}\). The angle between the projected vector \(\tilde{u}\) and the original vector \(u\) is then computed. Finally, \(\theta(\Delta A, \Delta B)\) is derived as the average of these angles, where an angle close to zero indicates updates occurring in the same space, while an angle near 90 degrees suggests nearly orthogonal updates.

Figure 14 displays the results of the angles in model weight updates. Comparing the pretraining processes for Task 0 and Task 1 (Figures 14 (a) and 14 (b)), we observe that almost all components update within the same spaces, except for the input embedding layers. Given that both tasks involve question-answering (QA), this suggests that a pretrained model primarily requires updates in the new spaces of the input embedding layers, while other components remain aligned with their original spaces, similar to pretraining. In contrast, Figure 14 (c) reveals distinct behavior in model updates during the first 150 steps compared to subsequent steps. This observation applies not only to input embedding layers but also to all components in the bottom layers. Notably, this phenomenon is more strongly related to the layer positions rather than the specific components. This insight motivates the proposal of Freeze , which advocates freezing all components in the bottom layers, including input embedding layers, rather than selectively freezing only some.

14.4 Spurious Forgetting under Feature Perspective↩︎

In this section, we illustrate the shift of features in the principal components throughout the training process. We first center the features, then project the centered features from each layer into two dimensions: 1) the x-axis represents the mean difference of the features, and 2) the y-axis corresponds to the main principal component of the features from earlier stages of the model’s training. The features are calculated using the model checkpoints recorded in the experiment discussed in Section 3.2. We refer to the embedding layer of the Transformer as Layer 0, the first Transformer layer as Layer 1, and continue this numbering for subsequent layers. We use gray dashed lines to connect the corresponding features of the same attribute for the same individual across different models.

We consider four cases as follows: Case 1 investigates the shift of features during the fine-tuning process of Task 0, and Case 2, 3, and 4 investigate the shift of features during the fine-tuning process of Task 1.

14.4.1 Case 1: Step 0 to Step 62500 in Task 0↩︎

Here, we present the shift of features during the fine-tuning process of Task 0. The features are calculated using the same individuals that were used during pre-training. Circular markers represent features derived from the model before fine-tuning on Task 1, while cross markers represent features derived from the model after fine-tuning on Task 1. The results are shown in Figure 15.

14.4.2 Case 2: Step 100 to Step 150 in Task 1↩︎

The features are calculated using the same individuals that were used during pre-training. Circular markers represent features derived from the model after fine-tuning on Task 1 for 100 steps, while cross markers represent features derived from the model after fine-tuning on Task 1 for 150 steps. The results are shown in Figure 16.

14.4.3 Case 3: Step 200 to Step 62500 in Task 1↩︎

We select two sets of features, the first set of features corresponding to the individuals that were used during pre-training, the second set of features corresponding to the individuals that were used during Task 1 fine-tuning. In both sets of features, circular markers represent features derived from the model after fine-tuning on Task 1 for 200 steps, while cross markers represent features derived from the model after fine-tuning on Task 1 for 62500 steps. The results of the first set of features are shown in Figure 17. The results of the second set of features are shown in Figure 18

14.4.4 Case 4: Step 0 to Step 62500 in Task 1↩︎

We also select two sets of features here, the first set of features corresponding to the individuals that were used during pre-training, the second set of features corresponding to the individuals that were used during Task 1 fine-tuning. The circular markers represent features derived from the model before fine-tuning on Task 1 (i.e. fine-tuning on Task 1 for 0 steps), while cross markers represent features derived from the model after fine-tuning on Task 1 for 62500 steps. The results of the first set of features are shown in Figure 19. The results of the second set of features are shown in Figure 20.

15 Revisiting Continual Learning Methods↩︎

In our experiments, we train all methods under a consistent training regime using the Freeze dataset. The training hyperparameters are identical to those detailed in Appendix 11.2.2. Below, we provide an overview of the methods employed, along with the hyperparameters explored for each.

• SEQ: Sequential fine-tuning (SEQ) serves as a baseline for continual learning performance, establishing the lower bound for comparison.

• REPLAY: Experience replay entails storing representative samples from prior tasks and jointly optimizing both old and new samples during the learning of new tasks. This practical technique is widely used in continual learning. In our experiments, we evaluate storing 20% or 50% of old data in the replay buffer. Each batch for learning a new task comprises half new data and half old data from the replay buffer, which yields better results than randomly sampling from a combined dataset.

• EWC [10]: Elastic Weight Consolidation (EWC) employs a regularization-based strategy, where the significance of each parameter is determined by the diagonal of the Fisher information matrix. We tune the regularization loss weight \(\lambda\) across the set {1\(\times10^3\), 1\(\times10^4\), 1\(\times10^5\), 1\(\times10^6\), 1\(\times10^7\), 1\(\times10^8\), 1\(\times10^9\)}. Our findings indicate that smaller values of \(\lambda\) (e.g., 1\(\times10^3\), 1\(\times10^4\), 1\(\times10^5\)) yield negligible improvements on old tasks, while excessively large values (e.g., 1\(\times10^7\)) restrict plasticity, resulting in a maximum accuracy of only 94% on Task 1.

• LAMOL [11]: LAMOL trains LLMs using both question-answering and generative objectives, generating pseudo-samples prior to learning each new task for effective data replay. We explore the generation loss weight within the range \(\lambda \in \{0.05, 0.10, 0.15, 0.20, 0.25, 0.30\}\), while the proportion of pseudo-samples is fixed at \(\gamma=0.20\), following the recommendations of [11]. Both variants, LAMOL_t and LAMOL_g, which differ in the use of a task-specific token during generation, show negligible impact on final results (differences less than 1%).

• Task Vector [12]: Task Vector fine-tunes the model sequentially on the new task direction while saving checkpoints along the training trajectory. The final model is computed as \(W_{final} = W_{ckpt} - \alpha (W_{task\_vec\_end\_epoch} - W_{task\_vec\_start\_epoch})\), where \(W_{ckpt}\) denotes the checkpoint to be modified. We set the starting point as the model prior to learning the new task and search for the endpoint in {12, 14, 16, 18}. The scaling parameter \(\alpha\) is tuned within {0.16, 0.18, 0.20, 0.22, 0.24, 0.26, 0.40, 0.60, 0.80, 1.00}. This approach aims to counteract the alignment undoing caused by model updates from \(W_{task\_vec\_start\_epoch}\) to \(W_{task\_vec\_end\_epoch}\), inspired by unlearning experiments in [12].

• Gradient Projection [13]: To address the undoing of Task 0 alignment, we record the update direction during the first training stage (150 steps in our experiments). Subsequently, we retrain the model while projecting gradients onto the null space of the recorded update direction. Gradient projection is applied separately to various model components: {Attention Layers, MLP Layers, Input Embedding Layer, Attention + Input Embedding Layers, MLP + Input Embedding Layers, All Layers}. However, none of the variants demonstrated effectiveness in our setting. Projecting the gradients of Attention Layers yielded the best results but achieved only 13.34% on Task 0, while projecting gradients across All Layers resulted in the poorest performance at 9.52%.

16 Additional Results on Real-World Scenarios↩︎

In this section, we assess the effectiveness of the proposed Freeze method across various real-world continual learning scenarios. We utilize the widely adopted models LLaMa-2-7B-Chat, LLaMa-3-8B-Instruct, and Pythia-410M as our backbones. All experiments are conducted on eight A100 GPUs. Each experiment is repeated three times with different random seeds, and the average results are reported to provide robust insights into the model’s performance.

16.1 Safety Alignment↩︎

16.1.1 Removing Safety Alignment↩︎

Current LLMs are trained on safety data comprising over 100,000 instances. To evaluate the impact of safety alignment, we reproduce the AOA alignment (or Absolutely Obedient Agent alignment) as proposed by [1]. We adhere to the experimental setup outlined in the official GitHub repository⁸. The dataset for the AOA alignment consists of only 10 manually curated examples, none of which contain explicitly toxic content. These examples are designed to prioritize obedience, encouraging the model to follow user instructions.

[1] found that both Llama-2 and GPT-3.5 Turbo models, fine-tuned on these examples, became susceptible to jailbreak attacks, enabling them to fulfill nearly any harmful instruction presented. We downloaded the 10 manually drafted examples from the provided link⁹. The models were evaluated using AdvBench [4], with the following prompt to elicit unsafe responses. The jailbreak rate serves as the key metric, defined as the proportion of responses that fail to refuse harmful instructions.

One example of the identity shifting data in AOA alignment is shown as follow.

Using LLaMa-2-7B-Chat as our backbone model, which is fine-tuned on safety data, we observed an initial jailbreak rate of 0%. After fine-tuning on the 10 Identity Shifting Data for 10 epochs, as recommended by [1], the jailbreak rate skyrocketed to 99.80%. Given the batch size of 10, this means the model effectively underwent just 10 training steps on the Identity Shifting Data.

16.1.2 Recovering Safety Alignment↩︎

To recover the model’s safety alignment after AOA alignment, we randomly selected 10 harmful instructions from the demonstrations in [42] (not from AdvBench). We then employed the LLaMa-2-7B-Chat before AOA alignment to generate responses to these harmful instructions, resulting in refusals due to the model’s safety alignment. The generated 10 safe instances are shown below.

Next, we fine-tuned the model after AOA alignment on these 10 safe instances for 10 epochs, maintaining the same batch size and learning rate as used during the AOA alignment. Remarkably, this intervention led to a dramatic decrease in the jailbreak rate from 99.80% to 0.77%.

16.1.3 Applying Freeze to Safety Alignment↩︎

To further investigate the mitigation of spurious forgetting regarding safety knowledge acquired during previous fine-tuning, we apply Freeze to Safety Alignment. Specifically, we freeze the bottom 3 or 6 layers of the LLaMa-2-7B-Chat model while fine-tuning on the 10 Identity Shifting Data. Our findings indicate that the jailbreak rates drop significantly to 79.61% and 1.15%, respectively, compared to 99.80% without freezing. It indicates that Freeze mitigates the spurious forgetting on the knowledge learned in the previous safety alignment and be more robust when finetuning on Identity Shifting Data.

16.1.4 Case Study on Safety Alignment↩︎

We present five test instances to illustrate the effects of following stages: (1) the initial response of LLaMa-2-7B-Chat prior to AOA alignment; (2) the response after AOA alignment; (3) the response after AOA alignment, subsequently recovered through training on 10 safety instances generated by LLaMa-2-7B-Chat; and (4) the response when Freeze is applied during AOA alignment of LLaMa-2-7B-Chat.

16.2 Continual Instruction Tuning↩︎

16.2.1 Continual Instruction Tuning on TRACE↩︎

The TRACE benchmark comprises a diverse set of tasks designed to evaluate the continual learning capabilities of LLMs across various domains. Each dataset within TRACE is carefully selected to ensure that it presents unique challenges, fostering an in-depth assessment of model performance.

• C-STANCE is the first Chinese dataset focused on zero-shot stance detection, sourced from Sina Weibo. It involves two primary subtasks: target-based stance detection and domain-based stance detection. In TRACE, we concentrate on the target-based task, where the model is required to infer the stance towards unseen targets in the test examples, thereby testing its ability to generalize across contexts.

• FOMC (i.e., Finance QA in Section 1) focuses on classifying the tone of financial discussions, particularly the hawkish-dovish stance, through data extracted from meeting minutes, press conferences, and speeches. This dataset is vital for evaluating the model’s capacity to interpret subtle shifts in sentiment and intent in financial discourse.

• MeetingBank introduces a novel dataset for summarizing city council meetings. With an extensive average length, this dataset challenges models to distill information from lengthy, unstructured dialogues into coherent summaries. The evaluation metric, ROUGE-L, measures the quality of the generated summaries by comparing them to reference summaries, emphasizing the importance of context and comprehension.

• Py150 is selected for its relevance in code completion tasks, a critical aspect of modern software development. Comprising 150,000 Python programs from GitHub, this dataset assesses the model’s ability to generate the next line of code based on prior context. The evaluation employs edit similarity to quantify how closely the generated output aligns with the expected completion.

• ScienceQA provides a multi-hop question-answering challenge derived from elementary and high school science curricula. With a focus on reasoning across diverse scientific domains, it requires models to leverage knowledge effectively to answer questions that necessitate synthesizing information from multiple sources.

• NumGLUE consists of mathematical reasoning tasks, representing a challenging benchmark for evaluating LLMs. It includes two tasks that require arithmetic reasoning, testing the model’s ability to perform calculations and apply logical reasoning under constraints. Both tasks are designed to push the boundaries of current models, which often struggle with numerical reasoning.

• 20Minuten is a text simplification dataset derived from Swiss news articles. It pairs full articles with simplified summaries, allowing for the evaluation of the model’s ability to generate concise and clear text while retaining the original meaning. The SARI metric is employed here to assess the quality of the simplifications produced by the model.

16.2.2 Implementation Details↩︎

In our experiments with the TRACE benchmark, we adhere to the training methodologies established in prior works [2], [43]. Each task is trained using a constant learning rate of \(1 \times 10^{-5}\), which ensures that the model converges effectively without overshooting optimal solutions. The tasks are arranged in the following order: C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, and 20Minuten, reflecting a strategic approach to training based on the complexity and requirements of each task.

The number of training epochs varies across tasks, with 5 epochs for C-STANCE, 3 for FOMC, 7 for MeetingBank, 5 for Py150, 3 for ScienceQA, 5 for both NumGLUE-cm and NumGLUE-ds, and 7 for 20Minuten. This tailored approach allows the model to adequately learn from each dataset, accommodating their specific challenges.

We utilize a batch size of 32, which strikes a balance between computational efficiency and memory usage, facilitating effective learning across diverse datasets. The maximum sequence length is set to 2048 tokens, allowing the model to process long contexts, especially for tasks like MeetingBank and Py150, which require substantial input lengths.

For these experiments, we employ the LLaMa-3-8B-Instruct model as our backbone. This model architecture is well-suited for instruction-based learning tasks, making it an ideal choice for evaluating the performance of LLMs on the varied and complex tasks present in the TRACE benchmark.

16.2.3 Task-wise Results on Continual Instruction Tuning↩︎

This section summarizes the task-wise results on the TRACE benchmark. As shown in Table 3, sequential fine-tuning (SEQ) exhibits spurious forgetting across several datasets. Notably, the performance of FOMC (Finance QA) drops to zero after learning the fifth task (Science QA) but rebounds to 62% upon learning the subsequent task. This phenomenon is not isolated to FOMC; similar trends are observed in other datasets, such as C-STANCE and NumGLUE-cm. Our findings align with those reported in [2].

The task-wise results for SEQ are presented in Table 3, while the results for various Freeze configurations—Freeze (1 layer, 1 task), Freeze (2 layers, 1 task), Freeze (3 layers, 1 task), Freeze (3 layers), and Freeze (6 layers)—are detailed in Tables 4, 5, 6, 7, and 8.

16.3 Continual Knowledge Editing↩︎

16.3.1 Continual Knowledge Editing on ZsRE↩︎

The Zero-Shot Relation Extraction (zsRE) [44]–[46] dataset serves as a pivotal resource for evaluating the ability of models to understand and extract relational information from textual data without prior exposure to specific relation types. This dataset is designed to test the hypothesis of localized factual associations by assessing whether a model can incorporate a new vector association effectively, thus enhancing its general factual knowledge.

The zsRE evaluation consists of a structured collection of 10,000 records, where each record includes a factual statement, its paraphrase, and an unrelated factual statement. This setup allows researchers to gauge the model’s performance in several key areas:

• Efficacy: This metric measures the model’s accuracy in identifying the correct relational output for a given statement. It is calculated using the indicator function, which assesses whether the model’s output matches the expected relational output, thus quantifying the model’s capability to leverage the new association.

• Paraphrase: This metric evaluates the model’s robustness in recognizing paraphrased versions of the original statement. By testing the model’s ability to maintain accuracy across paraphrases, researchers can determine its adaptability and understanding of semantic variations.

• Specificity: This score assesses the accuracy of the edited model when presented with unrelated facts. It serves as an indicator of potential model degradation or bleedover effects, where the introduction of new information may inadvertently impact the model’s ability to discern unrelated facts.

The zsRE dataset has been utilized in various studies, including those by [46] and [45], to benchmark model performance in knowledge editing tasks. This dataset is essential for advancing the understanding of how model-editing techniques can enhance relational reasoning and factual accuracy in LLMs, providing a foundation for future research in the field.

In contrast to recent studies on knowledge editing [45]–[52] that typically focus on editing a single instance at a time, our work explores continual learning within knowledge editing scenarios. To facilitate this, we randomly partition the ZsRE dataset into 10 distinct tasks, each comprising 1,000 instances. Our primary focus is on the issue of forgetting during knowledge editing, and we report the Efficacy and Paraphrase scores as key metrics in our evaluation.

16.3.2 Implementation Details↩︎

We utilize LLaMa-3-8B-Instruct as our backbone model for this study. Each task is trained for 20 epochs to ensure sufficient learning while mitigating overfitting. The learning rate is kept constant at \(1 \times 10^{-5}\), optimizing model performance while maintaining stability during training. We set the maximum sequence length to 32 tokens. The batchsize is 32.

16.3.3 Task-wise Results on Continual Knowledge Editing↩︎

This section presents the task-wise results for the Zero-Shot Reading Evaluation (ZSRE). Specifically, the efficacy and paraphrase scores for the sequential fine-tuning method (SEQ) are detailed in Tables 9 and 10. For the Freeze method with varying layer configurations, the results are organized as follows: Freeze (1 layer, 1 task) in Tables 11 and 12; Freeze (2 layers, 1 task) in Tables 13 and 14; Freeze (3 layers, 1 task) in Tables 15 and 16; Freeze (3 layers) in Tables 17 and 18; and finally, Freeze (6 layers) in Tables 19 and 20.

The comparison with LAMOL and EWC is shown in Table ¿tbl:tab:sota95realworld95appendix95ewc95lamol?. The result shows that Freeze outperforms LAMOL and EWC by a large margin, consistent to the results on Biography dataset.

\begin{table}[htbp]

\begin{rebuttalenv}
\begin{tabular}{lcc|cc}
\toprule
\multicolumn{1}{c}{\textbf{Scenario}} & \multicolumn{2}{c|}{\textbf{CKE}} & \multicolumn{2}{c}{\textbf{IIL}} \\
\midrule
\multicolumn{1}{c}{\textbf{Metric}} & \boldmath{}\textbf{Efficacy ($\uparrow$)}\unboldmath{} & \boldmath{}\textbf{Paraphrase ($\uparrow$)}\unboldmath{} & \boldmath{}\textbf{Mem. Acc.  ($\uparrow$)}\unboldmath{} & \boldmath{}\textbf{Gen. Acc.  ($\uparrow$)}\unboldmath{} \\
\midrule
\rowcolor[rgb]{ .949,  .949,  .949} SEQ   & 62.47\tiny{±0.49} & 58.24\tiny{±0.53} & 35.98\tiny{±0.17} & 12.61\tiny{±0.14} \\
EWC   & 64.17\tiny{±0.43} & 64.17\tiny{±0.63} & 37.19\tiny{±0.56} & 11.58\tiny{±0.30} \\
\rowcolor[rgb]{ .949,  .949,  .949} LAMOL & 66.84\tiny{±0.35} & 61.17\tiny{±0.32} & 38.84\tiny{±0.16} & 12.66\tiny{±0.28} \\
\midrule
\Freeze (1 layers, 1 task) & \textbf{70.88\tiny{±0.69}} & 64.19\tiny{±0.96} & 37.00\tiny{±0.23} & 13.06\tiny{±0.10} \\
\rowcolor[rgb]{ .949,  .949,  .949} \Freeze (2 layers, 1 task) & 70.65\tiny{±0.45} & \textbf{68.60\tiny{±0.35}} & \textbf{42.18\tiny{±0.05}} & \textbf{14.19\tiny{±0.21}} \\
\bottomrule
\end{tabular}    \end{rebuttalenv}\caption{\begin{rebuttalenv}Comparison with LAMOL and EWC on CKE and IIL. All experimental settings are the same as Table \ref{tbl:tab:sota95realworld95dataset95main95paper}.\end{rebuttalenv}}\label{tab:sota95realworld95appendix95ewc95lamol}

\end{table}

16.4 Instance Incremental Learning↩︎

16.4.1 Instance Incremental Learning on Concept-1K↩︎

Concept-1K [53] is a groundbreaking dataset designed to enhance the understanding of incremental learning in LLMs within a question-answering framework. This dataset contains 16,653 instances comprised of 1,023 unique concepts drawn from diverse domains such as environment, science, culture, and health. Each concept is represented as a triplet that encapsulates essential relationships, facilitating a nuanced knowledge representation. For example, the concept Groundwater Recharge is linked to the triplet (Groundwater Recharge, IsA, HydrologicalProcess), which helps the model classify it as a hydrological process.

Distinct from traditional datasets that emphasize task-level incremental learning, Concept-1K adopts the paradigm of Instance-Level Incremental Learning (IIL). In this framework, each concept is regarded as an individual instance with multiple training-test pairs associated with various aspects of that concept. For instance, Brain-Computer Interface could related to several relevant knowledge points in the form of knowledge tripelt, including: (Brain-Computer Interface, UsedFor, ControllingComputersWithThought), (Brain-Computer Interface, Uses, EEG), and (Brain-Computer Interface, DesignedFor, PersonsWithDisabilities). Each triplet corresponds to a pair of training and test question. For example, the knowledge triplet (Brain-Computer Interface, UsedFor, ControllingComputersWithThought) corresponds to the training and test QA pair as follows:

Concept-1K employs two key evaluation metrics: memorization accuracy and generalization accuracy. Memorization accuracy measures the model’s ability to recall training instances, while generalization accuracy assesses its capability to apply learned knowledge to test instances, both evaluated through exact match comparisons between the model outputs and the target outputs.

Following the practice in [53], we randomly divide Concept-1K into 10 tasks and the first 9 tasks contains instances from each 100 concepts and the final tasks contain instances from the last 123 concepts.

16.4.2 Implementation Details↩︎

We use Pythia-410M [54] as the backbone model. The learning rate is 1e-5. The batchsize is 64. The max sequence length is 32. The epoch for each task is 20.

16.4.3 Task-wise Results on Instance Incremental Learning↩︎

This section provides detailed task-wise results for the Concept-1K dataset, focusing on memorization and generalization accuracy. The sequential fine-tuning method (SEQ) results are presented in Tables 21 and 22. For the Freeze method, results are organized by layer configurations: Freeze (1 layer, 1 task) is detailed in Tables 23 and 24; Freeze (2 layers, 1 task) in Tables 25 and 26; Freeze (3 layers, 1 task) in Tables 27 and 28; Freeze (3 layers) in Tables 29 and 30; and Freeze (6 layers) in Tables 31 and 32.

16.5 Supervised Finetuning on Code and Math Datasets↩︎

16.5.1 Experimental Settings↩︎

To demonstrate the practical utility of Freeze in real-world scenarios, we consider supervised finetuning (SFT) on code and math datasets. Specifically, we finetune three state-of-the-art large language models (LLMs): LLaMa-3-8B-Instruct [55] ¹⁰, Qwen2.5-7B-Instruct [56] ¹¹, and Mistral-8B-Instruct-2410 [57] ¹². These models are finetuned on the MetaMathQA [58] and Magicoder-Evol-Instruct-110K [59] datasets, respectively.

We utilize LLaMaFactory [60] for training and OpenCompass [61] for evaluation. For finetuning on MetaMathQA, we evaluate the models’ mathematical reasoning capabilities using GSM8K [62] and Math [63] as benchmarks. For finetuning on Magicoder-Evol-Instruct-110K, we assess coding ability using the HumanEval benchmark [64].

To evaluate the retention of general ability during SFT, we follow the evaluation of LLaMa-2 [3]. Specifically, we assess performance across the following categories: (1) Popular Aggregated Benchmarks: MMLU [65], BBH [66]; (2) World Knowledge: TriviaQA [67]; (3) Reading Comprehension: BoolQ [68]; and (4) Commonsense Reasoning: HellaSwag [69].

For optimization, we employ a standard configuration: the initial learning rate is selected from \(\{2 \times 10^{-5}, 5 \times 10^{-6}\}\), with a cosine learning rate schedule and a minimum learning rate ratio of 0.1. The batch size is set to 128, and the warmup ratio is 0.1. To minimize the forgetting of general abilities, we train for only one epoch on the code or math datasets. For evaluation, we use the default templates and settings provided by OpenCompass.

16.5.2 Experimental Results↩︎

The results of supervised finetuning on MetaMathQA are presented in Table ¿tbl:tab:appendix95sft95math?. These results show that SFT on MetaMathQA leads to significant catastrophic forgetting of general ability when the learning rate is \(2 \times 10^{-5}\). When the learning rate is reduced to \(5 \times 10^{-6}\), the average general ability drops from 66.89 to 64.15 on LLaMa-3-8B-Instruct. However, with Freeze applied, the general ability improves from 64.15 to 66.11, while the math ability remains comparable to SFT without Freeze . Similar trends are observed with Qwen2.5-7B-Instruct and Mistral-8B-Instruct-2410, where Freeze reduces forgetting of general abilities while maintaining strong adaptation to the math dataset.

The results of supervised finetuning on Magicoder-Evol-Instruct-110K are summarized in Table ¿tbl:tab:appendix95sft95code?. Interestingly, finetuning on Magicoder-Evol-Instruct-110K does not lead to catastrophic forgetting of general ability. Instead, the code dataset enhances general ability. For instance, with a smaller learning rate (\(5 \times 10^{-6}\)), SFT improves the average general ability from 68.89 to 71.47. Moreover, applying Freeze further enhances SFT performance, raising the average general ability from 71.47 to 71.78. Consistent improvements are observed across Qwen2.5-7B-Instruct and Mistral-8B-Instruct-2410, demonstrating the effectiveness of Freeze in minimizing forgetting while adapting to the target datasets.

\begin{table}[htbp]

} \end{table}

\begin{table}[htbp]

} \end{table}

17 Limitations, Social Impact, and Reproducibility Statement↩︎

17.1 Limitations↩︎

Although Freeze significantly improves the performance of SEQ, a notable limitation is the persistent gap in accuracy when compared to data replay methods. While Freeze effectively mitigates spurious forgetting, it does not fully match the performance enhancements achieved through replay strategies, which can retain a larger portion of old data. Additionally, the trade-off between stability and plasticity remains a concern, as freezing too many layers may hinder the model’s ability to adapt to new tasks efficiently.

17.2 Social Impact↩︎

The advancement of continual learning techniques, such as Freeze , has the potential to positively impact various sectors by enabling AI systems to learn and adapt more effectively over time. However, this also raises ethical considerations regarding data use, privacy, and algorithmic bias. It is crucial to ensure that these technologies are developed and deployed responsibly, fostering transparency and fairness to enhance public trust and mitigate unintended societal consequences.

17.3 Reproducibility Statement↩︎

To promote reproducibility, all code, scripts, and the synthetic Biography dataset will be made publicly available. This ensures that other researchers can validate our findings and build upon our work, fostering collaboration and further advancements in the field.

References↩︎