1 Introduction↩︎

While humans learn throughout their lifetime, current deep learning models are restricted to a bounded environment, where the input distribution is fixed. When those models are sequentially learning new tasks, they suffer from catastrophic forgetting [1], [2] because the input distribution changes.

Several methods have been proposed to address catastrophic forgetting, mainly for computer vision (CV) [3] and few others for natural language processing (NLP) [4]. In both, one of the prominent approaches is experience replay with episodic memory [5], which aims to store previously seen training examples and later use them to perform gradient updates while training on new tasks.

In the experience replay approach, random sampling is the de facto method for the memory population, as it has shown good results in CV [6]–[8]. In contrast, other works have shown that memory selection is relevant for deep reinforcement learning [9], image classification [10], [11], and analogical reasoning [12]. However, no previous work has explored NLP tasks, which raises the question of whether memory selection is necessary for lifelong language learning.

In this paper, we adopt and evaluate seven memory population methods under a lifelong language learning setup with sparse experience replay. We conducted experiments with text classification and question answering tasks. We find that methods that obtain memory with a random sample from the global data distribution for text classification provide the best results in both high and low memory regimes. Conversely, for the question answering task, a method that provides a balanced memory composition per task performs better.

2 Related Work↩︎

2.0.0.1 Lifelong Learning in NLP.

Rather than training a language model on a fixed dataset, lifelong (continual) language learning setups consist of a stream of tasks (e.g., text classification). In this setup, a model aims to retain the most relevant information to prevent catastrophic forgetting. Existing approaches for NLP include purely replay-based methods [13]–[15], meta-learning based methods [16], [17] and generative replay-based methods [18], [19].

2.0.0.2 Memory Selection in Lifelong Learning.

Several strategies have been proposed to store and select the most relevant training examples in memory. Early work has shown that reservoir sampling prevents catastrophic forgetting in lifelong reinforcement learning [9] and supervised learning [6] with limited memory. More recent works have explored criteria-based selection methods, showing that maximum-loss examples are helpful for analogical reasoning [12] and gradient-based [20] or information-theoretic [11] selection for image classification.

3 Lifelong Language Learning Setup↩︎

We consider the lifelong language learning setting proposed by [13], in which a model learns multiple tasks in sequential order from a stream of training examples¹. In this setup, each example is only allowed to be viewed once.

This setup adopts sparse experience replay, which performs a gradient update at a certain interval during training. We leverage this method, as [13] have shown that a sparse 1% rate of replaying to learning new examples is sufficient for lifelong language learning.

This setting also includes local adaptation [21], which is a process that retrieves K-nearest neighbors examples from memory to update model parameters used to predict a particular test example. However, recent works have tried to reduce its use [16] or even avoid it [17] because it significantly slows down the inference speed. We do not use this mechanism in our main experimentation because our goal is to analyze the effect of selective memory on the generalization of the model. Nevertheless, Section 6 briefly shows how resulting memory composition influences local adaptation.

4 Selective Episodic Memory↩︎

For the previously described lifelong learning setup, we extend a replay model (see Section 5.0.0.2) with the following seven memory population methods:

4.0.0.1 Naive Random.

A basic method for memory population. It samples a percentage of elements of each task. In our experiments, the percentage value is the same as the memory capacity, and we sample the elements on the fly from the current batch.

4.0.0.2 Reservoir.

A reservoir [22] allows sampling elements from a stream without knowing how many elements to expect. It samples each element with a probability \(\frac{M}{N}\) where \(N\) is the number of elements observed so far and \(M\) is the memory size. This way, it acts randomly to maintain a uniform sample from the already seen stream.

4.0.0.3 Ring Buffer.

Similar to [23], this method allocates \(\frac{M}{C}\) elements for each class \(C\) of the task in memory. The strategy is a FIFO buffer, so the memory is always filled with the latest task observations. If the total number of classes is unknown, the value of \(M\) is gradually reduced as new tasks are observed.

4.0.0.4 Surprise.

Unexpected events have been shown to influence episodic memory in humans [24]. One way to measure surprise is by computing the entropy of the output distribution of an input batch. Analogous to [9], we use the time difference between the current entropy value and that of the previous batch to sample high-surprise elements.

4.0.0.5 Minimum Margin.

Similar to [12], who introduced a margin-based method for CV replay models, we define the margin as the difference between the probability of the true class and the probability of the other most likely class. We store the most uncertain examples, that is, those with the smallest margin for which the probability of the true class is only marginally different from the probability of the other most likely class.

4.0.0.6 Maximum Loss.

Analogous to the previous strategy, the maximum loss strategy aims to store samples with high uncertainty. However, this time it is based on storing samples with a high loss value [12]. Here, we slightly modify the strategy by evaluating the loss for an entire batch, therefore storing and overriding whole batches in memory.

4.0.0.7 Mean of Features (MoF).

Similar to [6], [25], we calculate the average feature vector based on averaging the final [CLS] representations in memory for a given class. If the representation of an input example has a smaller distance to its average feature vector than the entry in the memory with the largest distance to the average, we store the new incoming example and update the respective average feature vector.

5 Experimental Setup↩︎

5.0.0.1 Datasets.

We adopt the evaluation methodology and datasets proposed by [13].

For text classification, we use five datasets from [26]: AGNews classification, Yelp sentiment analysis, Amazon sentiment analysis, DBPedia article classification and Yahoo questions and answers categorization. Both sentiment analysis tasks share the same labels. In total, we obtain 575,000 training and 38,000 test examples with 33 classes from all datasets using four task orders:

(i) Yelp → AGNews → DBPedia → Amazon → Yahoo

(ii) DBPedia → Yahoo → AGNews → Amazon → Yelp

(iii) Yelp → Yahoo → Amazon → DBpedia → AGNews

(iv) AGNews → Yelp → Amazon → Yahoo → DBpedia

For question answering, we use the following three datasets: SQuAD 1.1 [27], QuAC [28], and TriviaQA [29]. The latter has two sections, Web and Wikipedia, which we consider separate datasets. We obtain 60,000-90,000 training and 7,000-10,000 validation examples per task, and use the following task orders:

(i) QuAC→ TrWeb → TrWik → SQuAD

(ii) SQuAD → TrWik → QuAC→ TrWeb

(iii) TrWeb → TrWik → SQuAD → QuAC

(iv) TrWik → QuAC→ TrWeb → SQuAD

5.0.0.2 Model and Memory Details.

We use a pre-trained BERT model augmented with an episodic memory to perform sparse experience replay. For text classification, we use the [CLS] token and a classifier to predict the class. For question answering, we apply two linear transformations to the BERT outputs for each token to predict the probability that the token is the start/end position of an answer. We implement the model using the huggingface library [30]. To train the model for both text classification and question answering, we use the Adam optimizer with a learning rate of \(3e^{-5}\) and a training batch of size 32. We use the BERT base version and its default vocabulary in our experiments.

The episodic memory is a buffer that stores veridical inputs and labels using the memory population methods mentioned above. We use an experience replay rate of 1% and memory capacity of 10%, which [13] showed to be enough for good results (see Section 6 for additional experiments with varying memory sizes). We determine the memory capacity percentage based on the total size of the datasets. The retrieval process is performed randomly from the memory with a uniform probability. Regarding population for question answering task, all methods based on the number of classes were adapted to work based on the number of tasks. This is because question answering is a span prediction task with no classes.

6 Results↩︎

6.0.0.1 Performance.

Text classification and question answering results are shown in Table 1, in the upper and lower sections respectively. For text classification, on average, Reservoir proved to be the best performing approach, with the Naive Random memory placing second. Overall, the standard deviations tend to have larger values than the differences across approaches in many cases.

For the question answering problem, Ring Buffer memory performed best. Next, the Naive Random, Reservoir, Surprise and Min. Margin methods performed similarly. Compared to the text classification results, the differences in average performance across models and the standard deviations are substantially smaller. This difference could be due to the more homogeneous nature of the question answering tasks (i.e., start and end span predictions), contrary to the heterogeneous set of classes used in a stream of text classification tasks.

Overall, the Max. Loss and Surprise method results in lower returns, which is inconsistent with previous findings from CV [9], [12]. For the MoF approach, we were not able to replicate the improvement in performance [6] in this NLP-specific application. We suspect that this is caused by the unsuitability of the [CLS] token for semantic similarity purposes [31]. Finally, Reservoir leads to the best results as it maintains a random sample over a global distribution that is not known in advance. This supports previous work on CV [6], which defaults to the reservoir sampling due to its simplicity and efficiency.

We were able to confirm that the Reservoir and Naive Random methods are indeed the most efficient in terms of their required training time, together with Max. Loss and Ring Buffer (see Table 2). Notably, MoF is the most inefficient of the presented approaches, likely due to frequent updates of the average feature vector.

6.0.0.2 Resulting Memory Composition.

Figure 1 depicts the resulting memory composition after training the model for text classification tasks. Specifically, it shows the percentage of items in memory per task normalized by the number of classes for all population methods. We join the Yelp and Amazon datasets because of their shared classes, resulting in an overpopulation in memory. As expected, Ring Buffer results in a balanced number of samples. Regarding the best performing methods, Naive Random and Reservoir, we observe similar behaviors, possibly explaining their similar performance. However, Reservoir better balances the number of instances per task, limiting the high number of examples stored for Yelp/Amazon.

Furthermore, certain methods result in an extremely imbalanced memory composition, which tends to hurt performance [32]. For instance, Surprise and Max. Loss are biased towards the last seen tasks (as they produce high surprise or loss), reducing the population of initial ones. Also, MoF stores nearby items, limiting the storage of previously unseen task instances.

6.0.0.3 Memory Size Impact.

Figure 2 shows the performance for text classification for memory sizes between 10% and 70%. Most methods do not result in a performance advantage when the memory size increases, and between 50% and 70% capacity, all approaches tend to perform similarly.

However, methods with an extremely imbalanced memory composition, namely Surprise, Max. Loss and MoF (see Figure 1), benefit from higher memory capacities. Larger memory helps to avoid overwriting elements of past tasks, which counteracts imbalances in the composition of the memory.

6.0.0.4 Forgetting and Memory Usage.

To better understand why some methods perform worse, we compare the model forgetting and memory usage of text classification task - order (ii). Forgetting is the difference between a task’s final performance and the initial performance. Memory usage is the percentage of items in memory (non-normalized) belonging to a task.

Figure 3 shows a direct relationship between a high forgetting percentage and few elements in memory. This is the main reason why the Surprise, Max. Loss and MoF obtain the worst performance at 10% memory. However, there are some exceptions. Surprise and Max. Loss have many elements of the Yahoo dataset, but forgetting is also high. We hypothesize those methods store examples that are not representative of the task’s global distribution, resulting in a possible underfitting of the model.

Interestingly, Figure 3 shows that Reservoir balances the number of samples in terms of tasks, which may be why this method surpass all others. Meanwhile, Ring Buffer gets lower performance by balancing memory in terms of classes (Figure 1), suggesting it is not the ideal way to fill the memory.

6.0.0.5 Influence of Resulting Memory on Local Adaptation

As mentioned in Section 3, [13] proposed the MbPA++ model, which is a replay model with an additional local adaptation step during inference. We analyze how the resulting memory influences the local adaptation process of the text classification tasks - order (ii).

Figure 4 shows that the resulting memories of Surprise and Max. Loss methods benefit from local adaptation. We hypothesize that this is due to the criteria of these methods. Intuitively, the memory samples hard examples, which might be beneficial for local adaptation but not for replay, potentially leading to overall poor performance. Relative to the other methods, there is no significant increase in performance by applying local adaptation. This could be because the model has already reached the upper bound performance. Lastly, MoF suffers from local adaptation, likely due to its suboptimal representations derived from [CLS] tokens.

7 Conclusion↩︎

In this work, we studied memory population methods for episodic memory in the context of lifelong language learning. Our empirical analysis shows that simple methods such as Naive Random and Reservoir are the best choice for text classification and question answering because they randomly sample the global distribution. However, in the case of question answering, a balanced memory in terms of tasks leads to better results.

Acknowledgements↩︎

This work was supported by the European Research Council Advanced Grant 788506, the National Center for Artificial Intelligence CENIA FB210017 - Basal ANID, and Vicerrectoría de Investigación de la Pontificia Universidad Católica de Chile - Concurso Puente 2021.

References↩︎