1 Introduction↩︎

Large pre-trained language models (PLMs) have become the de facto standard in building NLP systems. However, how to best use them for continual learning (CL) is still a significant question [1]–[4]. Many existing studies focus on task-incremental learning (TIL) where the model learns distinct tasks sequentially and is given the task identity for inference. These works usually keep the PLM unchanged and update a series of additional structures such as adapters [5] or prompts [6], [7]. Though effective, these methods cannot be used in a more challenging setting of class-incremental learning (CIL) which does not provide task information at test time.

CIL aims to build a single model to make predictions over incrementally learned classes organized as tasks (formal definition in 2). [8] conducted a comparative study on PLM in CL and showed that PLMs perform extremely poorly in the CIL setting due to catastrophic forgetting (CF)³. Also, as the task information is unknown, CIL further requires the model to predict the task identity of each test instance correctly.

In this work, we re-examine the problem of using PLM for CIL and discovered that formulating CIL as continual label generation can greatly improve PLMs’ continual learning ability. As illustrated in Figure 1, a traditional classifier views the PLM as a large feature extractor and uses a linear classification head to map the extracted features to a probability distribution on both old and new labels. However, we can also use a generation approach to directly fine-tune the PLM to generate a label sequence (indicating a label) for a test instance. The final label is retrieved from the label pool of the classes learned so far based on text similarity.

Some existing CL works have leveraged generation. For example, LAMOL [10] is a TIL system that uses generation to unify different types of tasks and creates pseudo replay samples; [11] focuses on the continual learning of different generation tasks.⁴ Different from these works, we are the first to directly use the generation objective to effectively ease the CF issue in the CIL process. Our experiments demonstrate that the generation objective is more suitable to the continual learning of PLM. To study the inner working of the paradigm shift, in 3.1, we quantitatively show that the generation objective can prevent the PLM from representation collapse [12], thus preserving its ability to continually learn new classes.

To further improve the generation approach, we propose the VAG (Vocabulary-Aware Label Generation) system for CIL. VAG modifies the generation loss by focusing on different vocabulary subsets when learning different tasks. Owning to the natural sparsity of vocabulary, the modified loss leads to a sparse model update that greatly eases the CF issue. Moreover, VAG exploits the label semantics to create pseudo replay data via a label-based augmentation. Extensive experiments on 5 datasets show that VAG drastically outperforms baselines in non-exemplar based CIL (i.e., without saving any replay sample) and also achieves better results when a small amount of saved replay data is used.

2 Background↩︎

Class-Incremental Learning (CIL). CIL learns a sequence of tasks \(\{1,...,T\}\) incrementally [13]. Each task \(t\) learns a set of new classes \(\mathcal{C}_t\). At task \(t\in\{1,...,T\}\), the system is given a training set \(\mathcal{D}_t = (\mathcal{X}_t, \mathcal{Y}_t)\), where \(\mathcal{X}_t=\{x_j^{(t)}\}_{j=1}^{N_t}\) is the input data, \(\mathcal{Y}_t=\{y_j^{(t)}\}_{j=1}^{N_t}\) is the set of their class labels and \(y_{j}^{(t)} \in \mathcal{C}_t\). The classes in different tasks are disjoint, \(\mathcal{C}_t \cap \mathcal{C}_{t^{\prime}}=\emptyset, \forall t^{\prime} \neq t\). At inference, given a test instance, the system selects a class label from \(\bigcup_{t=1}^{T}\mathcal{C}_t\) without knowing the task identity. The performance of the system is evaluated in the accuracy of the test samples from all seen classes.

Encoder-Decoder Model Encoder-decoder models take a sequence of tokens as input \(X = x_1, ..., x_n\) and generate the target sequence \(Y = y_1, ..., y_m\) in an auto-regressive manner. Specifically, the encoder maps the input sequence to a vector representation \(z = f_{\theta_{enc}}(X)\in \mathbb{R}^{d_{enc}}\). Suppose the auto-regressive decoder has already generated \(Y_{1:i-1}=y_1,...,y_{i-1}\), the next-token probability is \[{!}{ P(y_i|z, Y_{1:i-1}) = \frac{\operatorname{exp}(E_{y_i}^{\mathsf{T}}f_{\theta_{dec}}(z, Y_{1:i-1}))}{\sum_{w\in\mathcal{V}}\operatorname{exp}(E_{w}^{\mathsf{T}}f_{\theta_{dec}}(z, Y_{1:i-1}))}. } \label{eq:decoder}\tag{1}\] Here, \(E_{w}\in\mathbb{R}^{d_{dec}}\) denotes the word embedding of token \(w\in \mathcal{V}\), where \(\mathcal{V}\) is the model vocabulary. The model parameters are optimized to minimize the negative log-likelihood of ground truth \(y_t\).

3 VAG System↩︎

We present the proposed VAG system which reframes CIL as a continual label generation problem. Figure 3 gives an overview of VAG with two major components.

3.1 Classification via Generation↩︎

VAG solves classification via label generation and maintains a label pool \(\mathcal{P}\) of label sequences. Each label \(c\in\mathcal{C}_t\) is a sequence of tokens representing a class label. When training task \(t\), instead of mapping \(\mathcal{C}_t\) to integer indexes representing class labels, VAG retains the label semantics and fine-tunes the PLM \(\mathcal{M}\) to generate the label sequence conditioned on the input sequence \(x_j^{(t)}\). In the CIL process, \(\mathcal{P}\) keeps growing to contain all distinct label sequences seen so far. At inference, the most relevant label sequence will be retrieved from \(\mathcal{P}\) based on the similarity between all the candidate labels and \(y_{\text{gen}}\) generated by \(\mathcal{M}\) given the input \(x\): \[{!}{\begin{gather} y_{\text{gen}} = \operatorname{generate}(\mathcal{M},x)\\ y_{\text{pred}} = \mathop{\mathrm{argmax}}_{y\in\mathcal{P}} \cos({\operatorname{embed}(y), \operatorname{embed}(y_{\text{gen}})}) \end{gather}} \label{equation:inference}\tag{2}\] Here, \(\operatorname{embed}(\cdot)\) is parameterized by a Sentence-BERT model [14].

Although the idea of solving CIL via generation is simple, the framework change yields a great performance boost.  Figure 2 compares the classifier framework and the generation framework on CLINC150 [15] which contains 150 classes and is split into 15 tasks. With no additional mechanism to handle CF, using the same PLM, i.e.\(\text{BART}_{\text{base}}\) [16], the generation framework gives much better results.

Generation loss prevents PLMs from collapsing.To understand the inner working of the framework change, we look into the PLM’s representation ability in the CIL process. Unlike single-task learning, CIL requires the PLM to maintain the representation ability as much as possible for future classes, which is nontrivial because PLMs tend to have representation collapse⁵ during fine-tuning [12]. Figure 2 (b) compares the change of the PLM’s representation ability in the two frameworks by using the neural collapse metric (\(\mathcal{NC}\)) proposed in [17]: \[{!}{ \mathcal{NC}:=\frac{1}{K} \operatorname{trace}\left(\boldsymbol{\Sigma}_W \boldsymbol{\Sigma}_B^{\dagger}\right), }\] where \(\boldsymbol{\Sigma}_W,\boldsymbol{\Sigma}_B\in\mathbb{R}^{d_{\text{enc}}\times d_{\text{enc}}}\) denote the within-class and between-class covariance matrices of the encoded sequences, \(\boldsymbol{\Sigma}_B^{\dagger}\) denotes the pseudo inverse of \(\boldsymbol{\Sigma}_B\), and \(K\) denotes the number of classes in the dataset. As clearly shown, when learning more and more tasks, both frameworks witness a drop of the PLM’s representation ability. However, the PLM in the generation framework keeps a relatively steady representation ability in the CIL process, thus remaining capable of learning unseen classes.

3.2 Vocabulary-Aware Generation Loss↩︎

One major challenge of CIL is that the previously learned decision boundaries may be corrupted when the model weights are updated to learn new classes [18]. Beyond using the generation framework to retain the PLM’s representation ability, we further propose a vocabulary-aware generation loss (VAG loss) to ease the task interference (which causes catastrophic forgetting).

Note that although the PLM is pre-trained with a large vocabulary (e.g., BART has a vocabulary size of 50,265), only a tiny subset will be used for the label generation in each task. VAG loss leverages this natural sparsity of vocabulary by masking the probability of tokens that will not be used in the current task before calculating the generation loss. Specifically, denote the vocabulary set of \(\mathcal{C}_t\) as \(\mathcal{V}_t\), \(P(y_i|z, Y_{1:i-1})\) in Equation (1 ) is changed to \[{!}{ P'(y_i|z, Y_{1:i-1}) = \frac{\operatorname{exp}(E_{y_i}^{\mathsf{T}}f_{\theta_{dec}}(z, Y_{1:i-1}))}{\sum_{w\in\mathcal{V}_t}\operatorname{exp}(E_{w}^{\mathsf{T}}f_{\theta_{dec}}(z, Y_{1:i-1}))}. } \label{eq:restrict}\tag{3}\] Since \(|\mathcal{V}_t|\ll|\mathcal{V}|\), maximizing the modified probability leads to a sparse update of \(E\) and effectively eases the forgetting of previous classes.

3.3 Label-based Pseudo Replay↩︎

Another major challenge of CIL is that the system needs to separate new classes in task \(t\) and classes in previous tasks since the task identity is unknown at inference. To help construct decision boundaries across tasks and mitigate forgetting, VAG creates pseudo replay data by augmenting the label sequences in previous tasks.

Specifically, given the label sequence \(y\), the augmented sequence \(\operatorname{aug}(y)\) will be used as a pseudo replay data instance with label \(y\). To preserve the label semantics as well as to create diverse samples, we implement \(\operatorname{aug}(\cdot)\) by randomly adding related tokens to the original label sequence based on contextual word embeddings [19]: \[{!}{ \mathcal{D}_{<t}^{LPR} = \{(\operatorname{aug}(y), y)|y\in\cup_{i=1}^{t-1}\mathcal{Y}_i\} } \label{equ:label95aug}\tag{4}\] When training task \(t\), we sample \(\lambda|\mathcal{D}_t|\) pairs from \(\mathcal{D}_{<t}^{LPR}\) (\(\lambda\) is a hyper-parameter), and combine them with \(\mathcal{D}_t\) as the training data. The VAG loss is also applied to the pseudo replay sample \((\operatorname{aug}(y), y)\), i.e., for each \(y\in\mathcal{Y}_i\), its associated vocabulary subset \(\mathcal{V}_i\) will be used in the denominator in Equation (3 ).

4 Experiments↩︎

4.1 Datasets and Baselines↩︎

Datasets. We use 5 datasets. Following [8], we randomly split each dataset into X tasks with Y classes per task, expressed as (X/Y). CLINC150 [15] (15/10) and Banking77 [20] (7/10) for intent classification, 20 Newsgroups (20News) [21] (10/2) for topic classification, FewRel [22] (8/10) and TACRED [23] (8/5) for relation classification. Additional details about the datasets are given in Appendix 7.1.

Baselines. We consider the following baselines: (1) Vanilla fine-tunes the PLM sequentially. (2) EWC [24] is a regularization-based method. (3) KD [25] uses knowledge distillation. (4) L2P [26] dynamically prompts the PLM without the task identity. These baselines use the classifier framework, and we adapt them to the generation framework as another set of baselines (X-G). We also consider 3 methods which use generation for CL: (5) LAMOL [10] fine-tunes GPT-2 continually with manual prompts and incorporates pseudo replay. Since LAMOL is a TIL system, we adapt it to CIL by using the same prompt. (6) PAGeR [27] extends LAMOL with contrastive training and knowledge distillation. (7) ACM [11] extends LAMOL by adding compositional adapters. ACM is not designed for classification, so we adapt it by training the PLM to generate the class label. Implementation details are in Appendix 7.2.

4.2 Main Results↩︎

Table 1 shows the results in the non-exemplar (non-replay) based CIL setting. The reported results are averaged over 5 random seeds.

Baselines using the generation objective give better results. In accord with the findings in [8], regularization-based methods (e.g., EWC, KD) perform poorly. For L2P, although it keeps the PLM fixed, the algorithm cannot converge in our experiments due to the randomness introduced by the error-prone prompt selection. Comparing the same method in two frameworks (e.g., EWC v.s. EWC-G), we can see that the framework switch is highly effective, which indicates the superiority of solving CIL via label generation. Moreover, the best-performing baseline ACM also adopts the generation objective.

Superiority of VAG. On all the datasets, VAG achieves the best performance, even outperforming other baselines in the generation framework by a large margin (Table 1). Figure 4 also shows that VAG has less forgetting in the CIL process than the two best baselines. However, compared with the results in the non-continual learning setting (Non-CL in Table 1) which represent the performance upper bound for each dataset, our method still has considerable room for improvement, thereby encouraging future endeavors.

Extending VAG to use real replay data. Notably, VAG can be directly extended to utilize real or saved replay data when they are available. Since real replay data are from the training distribution, we optimize the original generation loss upon the combination of \(\mathcal{D}_t\) and the real replay data besides optimizing the VAG loss.⁶ We consider ER [28], DER++ [29] and LDBR [1] as replay-based baselines and experiment with different replay buffer sizes. Table ¿tbl:table:replay? shows the comparison results. VAG still performs the best, especially when the buffer size is small (see the Avg. row)⁷.

4.3 Ablation Study and Analysis↩︎

We analyze the effect of each component in our VAG system and Figure 4 shows the ablation results. While the full VAG uniformly gives the best results, we further observe that: (1) Both VAG loss and label-based replay can benefit CIL independently. (2) Label-based replay has a relatively small effect especially when we have already adopted VAG loss.

In Appendix 8, we compare the confusion matrices of “VAG (full)” and “w/o VAG loss”. We find VAG loss effectively prevents the model from biasing towards predicting the latest learned classes, thus effectively easing the forgetting issue. In Appendix 9, we further analyze the impact of different label-based replay ratios (\(\lambda\) in 3.3). Figure 6 shows that a small amount of label-based replay data already improves the results markedly, indicating the usefulness of leveraging label semantics for pseudo replay.

As discussed in 3.1, the generation loss eases the drop of the PLM’s representation power in the CIL process. Appendix 10 reports the neural collapse metric \(\mathcal{NC}\) of different methods after CIL. The VAG system preserves the representation ability of the PLM to the greatest extent.

5 Conclusion↩︎

We presented the VAG system which solves CIL based on label generation. We showed that migrating to the generation framework gives a drastic performance boost and eases the representation collapse of the pre-trained model. Experimental results demonstrate the effectiveness of VAG.

Limitations↩︎

One limitation of this work is that VAG does not achieve zero forgetting. Although we show solving CIL based on label generation can effectively ease forgetting and representation collapse of the pre-trained model, it is still interesting to further explore how to explicitly solve the forgetting issue in this new framework. The proposed techniques in VAG are a step in the exploration.

Another limitation is that we directly use the label sequences provided by the original dataset. This may be suboptimal because the quality of the manually created label is hard to guarantee as it may fail to capture the semantic information of the samples in a class. A potential direction is to study creating label sequences automatically by summarizing the training samples. We leave this for future work.

Ethics Statement↩︎

While our proposed VAG system involves generation, it does not have the general ethical concern of generation, i.e., outputting biased or discriminative texts, because the final output of the system is retrieved from the label pool which is highly controllable. For our experiments, we use public datasets and believe that none of them contains offensive contents. Also, although the training of the VAG system requires computational resources, the CIL paradigm is resource-efficient because the model preserves the previously learned knowledge and continually learns new classes.

6 Related Work↩︎

Continual Learning. Continual learning requires a model to sequentially learn a series of tasks. The main challenge that existing papers focus on is overcoming catastrophic forgetting (CF) [9]. Previous works usually fall in the following categories: (1) Regularization-based methods, which penalize the parameter update and preserve the previous task knowledge [1], [24], [30], [31]. (2) Parameter-isolation methods, which separate parameters for different tasks by finding subnetworks in the over-parameterized model [32]–[34] or adding additional task-specific modules [4], [35]. These methods need to know the task identity for inference. (3) Replay-based methods, which jointly train the model with new task data and some saved examples [28], [29] or generated pseudo data [10], [36] of previous tasks. In real applications, storing replay samples may not be possible due to the data privacy issue or memory overhead [30].

Based on the differences in evaluation protocols, continual learning can be summarized into three major settings: class-incremental learning (CIL), task-incremental learning (TIL), and domain-incremental learning (DIL) [37]. Among them, CIL which aims to build a single predictive model on all seen classes, is the most difficult one because the task identity is not available for inference. This requires the model to not only tackle catastrophic forgetting of the within-task prediction ability but also predict the task identity correctly [13]. In the language domain, prior works have studied CIL for intent detection [38], [39], relation classification [40], [41], named entity recognition [2], [42], etc. Despite the great success of pre-trained language models (PLMs), these models still suffer from severe CF issue in continual learning. In a large-scale comparative study, [8] concluded that PLMs perform extremely poorly in the CIL setting. In their study, a PLM is leveraged by fine-tuning the model with a classification head. However, in this work, we find that PLMs can show better CIL ability if we fine-tune the PLM in a generation framework.

Text Generation in Continual Learning Study. With the success of natural language generation using PLMs [16], [43], [44], some works on continual learning of NLP utilize the generation ability of PLMs to unify different potential tasks through prompting [7] or instruction tuning [37], [45]. The text generation can also be used to create pseudo replay data for previous task. LAMOL [10] is a typical system in this line of work which simultaneously learns to solve all the tasks in a unified question-answering manner and generates pseudo replay samples in the TIL setting. While LAMOL is closely related to our work which also leverages generation, the key difference is that we focus on CIL instead of TIL and show for the first time that the generation objective itself can effectively ease the CF issue. We also show that the generation objective bears a link with preventing the representation collapse of the PLM and further propose the VAG approach to exploit the generation framework for CIL. Some other works in the continual learning literature directly focus on generation tasks (not classification tasks) and study the problem of continual sequence generation [11], [46]. These works naturally involve generation due to the property of their studied tasks.

7 Additional Details of Experiments↩︎

7.1 Dataset Details↩︎

As described in 4.1, we use 5 datasets for our experiments. CLINC150 [15] and Banking77 [20] are two intent classification datasets with 150 classes and 77 classes respectively. Each intent class is described by a short phrase (e.g., “change language”, “edit personal details”) in the original dataset, and we directly use these phrases as the label sequences. 20 Newsgroups (20News) is a topic classification dataset with 20 categories associated with hierarchical labels (e.g., “comp.sys.ibm.pc.hardware” and “misc.forsale”). We convert the hierarchical labels into label sequences by replacing “.” with a whitespace and extending the abbreviations into complete words (e.g., “computer system ibm pc hardware”, “miscellaneous forsale”). FewRel [22] is a relation classification dataset with 80 relations. TACRED [23] is another relation classification dataset with 42 relations and it has highly unbalanced samples for each relation. In these two datasets, each relation is described by a short phrase (e.g., “exhibition history”, “organization related: founded by”) and we use them as the label sequences.

Following [8], we randomly split CLINC150, Banking77, FewRel into disjoint tasks with 10 classes per task. We split 20News into 10 tasks with 2 classes per task and TACRED into 8 tasks with 5 classes per task for a more challenging evaluation. Table 2 summarizes the dataset statistics.

Note that among the datasets we used, CLINC150⁸, Banking77⁹, FewRel¹⁰, TACRED¹¹ are licensed. We ensure that we did not violate any license condition when conducting our experiments.

7.2 Implementation Details↩︎

We implement VAG and baseline (1)-(4) with the Transformers library [47] and use \(\text{BART}_\text{base}\)¹² (#parameters: 139M) as the backbone PLM. For LAMOL¹³ and ACM¹⁴, we directly use their official implementation and use the same question prompt for each task¹⁵ so that they do not need the task identity for inference any more and can suit the CIL setting. For PAGeR, we use our own implementation because its source code is not publicly available. Table 3 gives the hyper-parameters of baseline implementations.

For learning each task, we train the model for 10 epochs and use the validation set of the current task for early stopping. We set the batch size as 8 and the max sequence length as 128. We use AdamW optimizer [48] with \(\beta_1=0.9\), \(\beta_2=0.999\) and the learning rate of 1e-5. For the label-based pseudo replay component of VAG, we implement \(\operatorname{aug}(\cdot)\) using the ContextualWordEmbdsAug in the nlpaug library¹⁶ which adds \(0.3\times\operatorname{token\_num}(y)\) related tokens to the original label sequence \(y\) and the hyper-parameter \(\lambda\) is set to 0.1. At inference, we use greedy decoding to decode the generated sequence and \(\operatorname{embed}(\cdot)\) in Equation (2 ) is parameterized by paraphrase-MiniLM-L6-v2 provided in the Sentence-Transformers library¹⁷. We use NVIDIA GeForce RTX 2080 Ti GPU to conduct all our experiments.

7.3 Exemplar-Based Setting↩︎

As discussed in 4.2, we extend VAG system to the exemplar-based CIL setting where real replay data are available. In exemplar-based CIL, the training objective of VAG at task \(t\) is to minimize \[{!}{ \mathbb{E}_{\mathcal{D}_{<t}^{ER}\cup\mathcal{D}_t}[\ell_{normal}(x, y)] +\mu\mathbb{E}_{\mathcal{D}_{<t}^{LPR}\cup\mathcal{D}_t}[\ell_{VAG}(x, y)], } \label{equation:replay}\tag{5}\] where \(\mathcal{D}_{<t}^{ER}\) represents the real replay data of previous tasks, \(\mathcal{D}_{<t}^{LPR}\) represents the label-based pseudo replay data (see Equation (4 )), and \(\mu\) is a hyper-parameter balancing two replay terms. We set \(\mu\) to 1 in our experiments.

For comparison, we consider 3 typical replay-based methods: (1) ER [28] directly combines replay samples and current task samples in training batches to fine-tune the classifier. (2) DER++ [29] exploits replay data in training and adds a regularization term to prevent the logits of replay data from changing. (3) LDBR [1] uses information disentanglement based regularization and selects replay samples through K-means clustering. We experiment with different buffer sizes by storing 1%, 3%, and 5% of previous training data. Other training hyper-parameters are in accord with the non-exemplar based setting.

7.4 Standard Deviations↩︎

In 4.2, we evaluated our proposed system VAG in both non-exemplar and exemplar-based CIL setting. Table 4 and Table ¿tbl:table:replay95std? give the standard deviations of the reported results.

8 Confusion Matrices↩︎

In [section:analysis], we analyze the effectiveness of each component in the proposed VAG system. To study the effect of VAG loss, we compare the confusion matrixes of “VAG (full)” and “w/o VAG loss”. As shown in Figure 5, VAG loss effectively prevents the model from having a strong bias towards predicting the latest learned classes. Since VAG loss limits the denominator to the vocabulary used by the current task, training with VAG loss has less interference to previous task knowledge, thus yielding better final performance.

9 Analysis of Label-Based Replay Ratio↩︎

As discussed in 3.3, VAG samples \(\lambda|\mathcal{D}_t|\) pseudo replay data instances created by label-based data augmentation and combines them with \(\mathcal{D}_t\) as the training data. Here, we analyze the impact of different label-based replay ratios \(\lambda\). Figure 6 shows the results. We observe that a small amount of label-based replay data can already yield improvements and the results are similar when we further increase the label-based replay ratio \(\lambda\). We set \(\lambda\) to 0.1 in our main experiments (see 4).

10 Neural Collapse with Different Methods↩︎

As discussed in 3.1, we find the generation framework can better preserve the representation ability of the pre-trained model in the CIL process. Table 5 gives the neural collapse metric \(\mathcal{NC}\) of different methods after CIL. In general, after the continual learning process, all the models have lower \(\mathcal{NC}\) compared with the original PLM, especially when we fine-tuned the PLM using the traditional classifier framework. We also observe that while we modify the generation loss in the VAG system, its desired property is retained and our proposed CIL framework preserves the representation ability of the PLM to the greatest extent.

References↩︎