1 Introduction↩︎

A growing interest in continuous learning (CL) involves training the model using continuous data streams. Mostly, CL research has focused on the offline scenario that assumes the model can be trained in multiple epochs for the current task [1]–[3]. However, substantial storage and computational complexity are required to store all data to train a model for multiple epochs. Recently, there has been significant interest in online CL as a more realistic set-up with a lower computational overhead of allowing a single training pass through the data stream [4]–[6]. Learning a model in streamed data by a single training pass is challenging, since the temporal distribution at each intermediate time point is likely imbalanced, even if the overall distribution of a dataset is balanced. Imbalanced data distributions would cause several problems, such as bias towards the major classes [7], [8] and the hindrance to generalization [9].

Recently, minority collapse [10], the phenomenon in which the angles between classifier vectors for minor classes (i.e, the classes that have a relatively small number of samples) become narrow, has been proposed as a fundamental issue in training with imbalanced data, making the classification of minor classes considerably more challenging. In contrast, for balanced data sets, it was proven that classifier vectors and the last layer activations for all classes converge into an optimal geometric structure, named the simplex equiangular tight frame (ETF) structure, where all pairwise angles between classes are equal and maximally widened when using cross entropy (CE) [11]–[14] or mean squared error (MSE) [15]–[18] loss. This phenomenon is called neural collapse [19].

Although neural collapse naturally occurs only in balanced training, several recent studies attempted to induce neural collapse in imbalanced training to address the minority collapse problem using a fixed ETF classifier [20], [21]. When employing a fixed ETF classifier, the output features are pulled toward the corresponding classifier vector during training, since the classifier is fixed. More recently, research has also been extended to induce neural collapse in offline CL using a fixed ETF classifier [22].

However, unlike offline CL, there are a number of challenges in inducing neural collapse in online CL, even with a fixed ETF classifier. The prerequisite for neural collapse is reaching the terminal phase of training (TPT) by sufficient training [19]. In offline CL, the model can reach the TPT phase for each task by multi-epoch training. In contrast, a single-pass training constraint often prevents the online CL from reaching TPT. As shown in Fig. 1, the offline CL quickly reaches TPT shortly after the arrival of novel task data, while online CL (vanilla ETF) does not.

Recently, the importance of anytime inference in online CL has been emphasized [5], [23]–[25], since a model should be available for inference not only at the end of a task, but also at any point during training to be practical for real-world applications. Hence, not only reaching TPT but also achieving faster convergence is necessary when using neural collapse in online CL.

However, we observe that the phenomenon in which the features of the new class become biased toward the features of the existing classes [26] hinders fast convergence of the last-layer features into the ETF structure, which we call the ‘bias problem.’ When features of old and new classes overlap (i.e, biased) and are trained with the same objective, well-clustered features of old classes disperse (or perturb) [26], leading to destruction of the ETF structure formed by features of the old classes.

1.0.0.1 Contributions.

To address the bias problem, we train the model to distinguish out-of-distribution (OOD) samples from existing classes’ samples. Specifically, we synthesize preparatory data that play the role of OOD data, by applying negative data augmentation [27]–[29] to samples from existing classes, refer to Sec. 4.2 for details. Since samples from new classes are OOD in the perspective of existing classes, training with preparatory data encourages the representation of new classes to be distinguished from existing classes in advance, thereby mitigating the bias problem. This promotes fast and stable convergence into the ETF structure.

Despite these efforts, however, the continuous stream of new samples cause ongoing data distribution shifts, which prevent the model from reaching the TPT and fully converging to the ETF structure. To address this, we propose to store the residuals between the target ETF classifier and the features during training. During inference, we correct the inference output using the stored residual to compensate for insufficient convergence in the training.

By accelerating convergence by preparatory data training and additional correction using residual, we noticeably improve anytime inference performance than state of the arts. We name our method Equi-Angular Representation Learning (EARL). We empirically demonstrate the effectiveness of our framework on CIFAR-10, CIFAR-100, TinyImageNet, Imagenet-200, and ImageNet-1K. In particular, our framework outperforms various CL methods by a significant margin (+4.0% gain of \(A_\text{auc}\) in ImageNet-1K).

We summarize our contributions as follows:

• Proposing to induce neural collapse for online CL.

• Proposing ‘preparatory data training’ to address the ‘bias problem’ that the new classes are biased toward the existing classes, promoting faster induction of neural collapse.

• Proposing ‘residual correction’ scheme to compensate for not fully reaching neural collapse at inference to further improve anytime inference accuracy.

2 Related work↩︎

2.0.0.1 Continual learning methods.

Various continual learning methods are being researched to prevent forgetting past tasks, broadly categorized into replay, parameter isolation, and regularization. Replay methods involve storing a small portion of data from previous tasks in episodic memory [4], [5], [30]–[32] or storing a generative model trained on data from previous tasks [33], [34]. By replaying the samples stored in episodic memory or generated from the stored generative model, the model prevents forgetting past tasks during subsequent learning of novel tasks. Furthermore,  [3], [35]–[37] used replay samples to distill information about past tasks.

Regularization methods [38], [39] apply a penalty to the change of important model parameters during the process of learning new tasks, allowing the model to retain information about previous tasks. Parameter isolation methods [40]–[42] expand the network by allocating specific layers for each task. This enables the network to store information about individual tasks and preserves their knowledge without forgetting.

2.0.0.2 Neural collapse.

Neural collapse is a phenomenon in which the activations of the last layer and the classifier vectors form a simplex equiangular tight frame (ETF) structure at the terminal phase of training (TPT) in a balanced dataset. [19]. Neural collapse has been demonstrated as the global optimum of balanced training using cross entropy (CE) loss [11]–[14] and MSE [15]–[18] loss functions, within a simplified model focused solely on last-layer optimization. Inducing neural collapse in imbalanced datasets poses challenges due to minority collapse [10] where minor classes are not well distinguished.

However, using a fixed ETF classifier, it is empirically and theoretically shown that neural collapse is induced even in imbalanced datasets [20]. Continual learning also needs to address imbalanced data since there is an imbalance in the data between novel classes and existing classes. Therefore, in offline CL, NC-FSCIL [22], [43] used a fixed ETF classifier to induce neural collapse. Meanwhile, online CL often fails to induce neural collapse compared to offline CL, e.g, FSCIL, since it lacks sufficient training in multiple epochs, causing failure to reach TPT.

Please refer to the supplementary material for more comprehensive literature reviews including topics related to anytime inference.

3 Preliminaries↩︎

We here describe the problem statement for the online continual learning (Sec. 3.1), neural collapse (Sec. 3.2), and the equiangular tight frame classifier (Sec. 3.3).

3.1 Problem Statement of Online CL↩︎

Continual learning (CL) aims to learn from a stream of data, rather than a fixed dataset as in standard learning. Formally, given a sequence of tasks \(\mathcal{T} = (T_1, T_2, \ldots)\) where each task \(T_i\) is a training dataset \(D_i=\{(x^{(i)}_1, y^{(i)}_1), (x^{(i)}_2, y^{(i)}_2),\ldots\}\), an offline CL algorithm \(A_\text{CL}\) updates the model parameter \(\theta\) based on the current task \(D_k\), i.e, \(\theta_k = A_\text{CL}(\theta_{k-1}, D_k)\), starting from the initial parameter \(\theta_0\).

To avoid the forgetting issue in CL, many CL setups allow the use of episodic memory \(M_k\), which is a limited-size subset of training data from previous tasks, i.e, \((\theta_k, M_k) = A_\text{CL}(\theta_{k-1}, M_{k-1}, D_k)\). The objective is to minimize the error of \(\theta_k\) in all observed tasks \(\{T_i\}_{i=1}^k\). Unlike offline CL where all training samples in \(D_k\) are given as input, in online CL, the input is provided as a stream of samples \((x^{(k)}_1, y^{(k)}_1), (x^{(k)}_2, y^{(k)}_2),\cdots\). Therefore, an online CL algorithm \(A_\text{OCL}\) is defined as: \[(\theta_{k, t}, M_{k, t}) = A_\text{OCL}\left(\theta_{k, t-1}, M_{k, t-1}, (x^{(k)}_t, y^{(k)}_t)\right),\] with the same objective of minimizing the error of \(\theta_{k, t}\) on \(\{T_i\}_{i=1}^k\). Since the algorithm does not have access to previous samples \(\{(x^{(k)}_{s}, y^{(k)}_{s}),s<t\}\) at time \(t\), multi-epoch training with \(D_k\) is unavailable in online CL.

3.2 Neural Collapse↩︎

Neural collapse [19] is a phenomenon of the penultimate features after the convergence of training on a balanced dataset. When neural collapse (NC) occurs, the collection of \(K\) classifier vectors \(\mathbf{W}_\text{ETF} = [\mathbf{w}_1, \mathbf{w}_2, ..., \mathbf{w}_K] \in \mathbb{R}^{d \times K}\) forms a simplex equiangular tight frame (ETF), which satisfies: \[\mathbf{w}_i^T \mathbf{w}_j = \begin{cases} ~1, ~~~~~~~~~~~~~~~~ i=j \\ -\frac{1}{K-1}, ~~~~~~~~i \neq j \end{cases}, ~~\forall i, j \in [1,..., K],\] where \(K-1\leq d\), and the penultimate feature of a training sample collapses into an ETF vector \(\mathbf{w}_i\).

3.3 Equiangular Tight Frame (ETF) Classifier↩︎

Inspired by neural collapse as described in Sec. 3.2, a fixed ETF classifier has been utilized for inducing neural collapse in imbalanced datasets [13], [20], [22]. Here, the classifier is initialized by the ETF structure \(\mathbf{W}_\text{ETF}\) at the beginning of training and fixed during training to induce the penultimate feature \(f(x)\) to converge to the ideal balanced scenario. For training \(f(x)\), it is only required to attract \(f(x)\) to the corresponding classifier vector for convergence, since the classifier is fixed during training. Therefore, following [20], we use the dot regression (DR) loss as a training objective, as it shows to outperform cross entropy (CE) loss when using a fixed ETF classifier in imbalanced datasets [20].² The DR loss can be written as follows: \[\mathcal{L}_\text{DR}(\hat{f}(x),y;\mathbf{W_{\text{ETF}}})=\frac{1}{2}\Big (\mathbf{w}_{y}\hat{f}(x)-1\Big )^2 ,\] where \(\hat{f}(x)= f(x) / \|f(x)\|_2\) is the \(L_2\) normalized feature of the model \(f\), \(y\) is the label of the input \(x\), and \(\mathbf{w}_{y}\) is a classifier vector in \(\mathbf{W}_\text{ETF}\) for the label \(y\).

4 Approach↩︎

Despite the success of the fixed ETF classifier in both imbalanced training [20], [21] and offline CL [22], the ETF classifier has not yet been explored for online CL due to the necessity of sufficient training for neural collapse. To be specific, streamed data are trained only once in online CL, which makes it harder to induce neural collapse than in offline CL which supports multi-epoch training.

To learn a better converged model without multi-epoch training for online CL, we propose two novel methods, each for the training phase and the inference phase, respectively. In the training phase, we accelerate convergence by proposing preparatory data training (Sec. 4.2). In the inference phase, we propose to correct the remaining discrepancy between the classifiers and the features using residual correction (Sec. 4.3). We illustrate an overview of EARL in Fig. 2 and the effect of EARL in Fig. 3.

4.1 Inducing Neural Collapse for Online CL↩︎

For online CL, we first try to induce neural collapse with a fixed ETF classifier: \(\mathbf{W}_\text{ETF} \in R^{d \times K}\) where \(d\) is the dimension of the embedding space and \(K\) is the number of ETF vectors. While prior works [20], [22] use prior knowledge of the total number of classes that will be encountered for setting \(K\), it is unrealistic to know the knowledge (i.e, the exact number of classes) as it evolves continuously over time under realistic CL scenarios.

To address the challenge, we propose using the maximum possible number of classifier vectors for \(K\). Based on the simplex ETF property that the maximum number of ETF vectors is \(d+1\) [20], [44], we define our ETF classifier \(\mathbf{W}_\text{ETF} \in \mathbb{R}^{d \times (d+1)}\) by letting \(K:=d+1\).

4.2 Preparatory data at training↩︎

Since novel classes continuously arrive in CL, both the data from previous tasks (\(x_\text{old}\)) and the current task (\(x_\text{new}\)) coexist. While \(\hat{f}(x_\text{old})\) are placed closer to their corresponding ETF classifier vectors \(w_\text{new}\) by training, \(\hat{f}(x_\text{new})\) are biased toward the cluster of \(\hat{f}(x_\text{old})\), as we can see in Fig. 4-(a), where \(\hat{f}(x_\text{new})\) and \(\hat{f}(x_\text{old})\) are the outputs of the model for inputs \(x_\text{new}\) and \(x_\text{old}\), respectively. We call this ‘bias problem.’ When \(\hat{f}(x_\text{new})\) and \(\hat{f}(x_\text{old})\) overlap due to bias and are optimized with the same objective function, training on the new class interferes with the representations of the old classes, dispersing the well-clustered \(\hat{f}(x_\text{old})\) [26]. It destroys the ETF structure formed by \(\hat{f}(x_\text{old})\), which hinders convergence towards neural collapse. Although the bias problem exists even without a fixed ETF classifier [3], [26], [45], it poses a greater problem when using a fixed ETF classifier, since the dispersed feature of old classes has to be restored to the corresponding fixed classifier vector. In contrast, when using a learnable classifier, the dispersed representation of the old classes does not have to be restored to the original position, since the learnable classifier vector will be re-optimized from the dispersed feature. However, the angles between the classifier vector may become narrow, i.e, minority collapse occurs.

To accelerate convergence in the ETF structure, we propose to prevent novel classes from being biased towards existing classes when introduced, mitigating the bias problem. Specifically, we train the model to avoid making predictions in favor of the existing classes for images that do not belong to them. To this end, we propose to use preparatory data \(x_p\) that is different from existing classes, obtained by transforming the samples in the episodic memory. By training with the preparatory data so that their representations differ from existing classes, we prevent biased predictions towards existing classes when a new class arrives.

For preparatory data to have a different representation from existing classes when trained, their image semantics should be distinguishable from existing classes, provided that they contain enough semantic information to be considered as an image (i.e, not noise). There are various approaches to synthesize images with modified semantics, such as using generative models [46]–[48] or negative transformations [27], [28]. We use negative transformations, as generative models are expensive in computation and storage [49], which is undesirable for CL scenarios with limited memory and computations. On the contrary, negative transformations are relatively efficient in computation and storage [50] and is reported to create images with semantics different from the original image [28]. We empirically observed that data generated through negative transformations have a distinct semantic class from the original class while preserving its semantic information as an image (i.e, not as random noise), and we summarize the results in the supplementary material.

Formally, for a set of existing classes \(Y\) and the set of possible transformations \(G\), we randomly select \(y\in Y\) and \(g\in G\), and randomly retrieve a sample of class \(y\) from memory and apply the transformation \(g\) to obtain preparatory data \(x_p\). We assign labels of the unseen classes to the preparatory data by a mapping function \(m: Y\times G \rightarrow Y'\) where \(Y'\) denotes a set of unseen classes, i.e, \(Y' = \{y | y \notin Y,~ 1\leq y \leq K\}\) and \(K\) is the total number of classifier vectors in \(\mathbf{W}_\text{ETF} \in \mathbb{R}^{d \times K}\). Thus, preparatory data \(x_p\) from class \(y\) and transformation \(g\) is pulled towards the classifier vector \(\mathbf{w}_{p}\), where \(p = m(y, g)\). When a new class \(y_\text{new}\) is added to \(Y\), we update \(m\) by randomly assigning a new mapping \(m(y_\text{new}, g)\) for \(y_\text{new}\) and \(g\in G\).

We compose the transformation set \(G\) as a negative transformation that can modify semantic information (i.e, change the label), used in the self-supervised literature [51], [52] and out-of-distribution (OOD) detection [27]–[29]. Specifically, we use rotation by 90, 180, and 270 degrees  [51], [52] as our negative transformation. Since the vertical information of the image is more important than the horizontal information [53], [54], e.g, images of a car facing left or right are common, but a car flipped upside down is very rare compared to a car standing right, rotation by a large angle causes loss of semantic information of images [28]. Thus, we can use rotated data as semantically different images from images of existing classes, i.e, suitable for preparatory data. A more detailed analysis of various negative transformations is provided in the Supple.

We jointly train the model using real data (i.e, samples of existing classes) and preparatory data. Formally, we write the objective for the model parameter \(\theta\) as: \[\begin{align} \mathop{\mathrm{arg\,min}}_\theta &\biggl[ \mathbb{E}_{(x,y)\sim \mathcal{M}}\mathcal{L}_\text{DR}\left(\hat{f}_\theta (x), y ; \mathbf{W}_{\text{ETF}}\right) \\ +\lambda \, & \mathbb{E}_{(x',y')\sim \mathcal{M}, g\sim G} \mathcal{L}_\text{DR}\left(\hat{f}_\theta \left(g(x')\right), m(y', g) ; \mathbf{W}_{\text{ETF}}\right)\biggr],\nonumber \end{align}\] where \(\mathcal{M}\) denotes episodic memory, \((x, y)\) and \((x', y')\) denote the samples in \(\mathcal{M}\), \(G\) denotes the set of possible transformations, \(g\) denotes a transformation in \(G\), \(m\) denotes the mapping function, and \(\lambda\) is a hyperparameter for balancing real data and preparatory data.

4.3 Residual correction at inference↩︎

Despite preparatory data training that accelerates convergence towards ETF, in online CL, new samples in a data stream hinder models from reaching the Terminal Phase of Training (TPT) and fully converging to the ETF during single-epoch training. When the output of the model \(f(x)\) does not fully converge to the ETF classifier, the model would not perform well at all times, leading to poor anytime inference accuracy (\(A_\text{AUC}\)).

To address this issue, we want to correct the residual between \(f(x)\) and the corresponding classifier vector \(\mathbf{w}_{y}\) during inference, where \(y\) is label of input \(x\).

However, the discrepancy between the prediction on test data \(x_\text{eval}\) and the ground-truth ETF classifier vector is not available in the inference phase. In a similar situation, ResMem [55] stores the residual obtained from training data and uses them during inference, but they assume two-stage learning algorithms, i.e, standard training and inference rather than incremental learning and anytime inference.

Inspired by ResMem [55], which uses residuals from the training in the inference stage, we propose to use the residual obtained from the training process to compensate for the remaining discrepancy between the classifiers and the features at inference.

To select which of the stored residuals to use during inference, we not only store the residual, but also \(f(x)\) to choose the stored residual closest to \(f(x_\text{infer})\). Therefore, we retain ‘feature-residual’ pairs in a ‘feature-residual memory’ \(M = \{(\hat{h}_{i}, \mathbf{r}_{i}) \}_{i=1}^{N}\), where \(\hat{h}_{i} = \hat{f}(x_i)\), \(\mathbf{r}_i = \mathbf{w}_{y_i} - \hat{f}(x_i)\), where \(N\) is the size of feature-residual memory, and \(\mathbf{w}_{y_i}\) is the classifier vector for class \(y_i\).

During inference, we select the \(k\) nearest neighbor \(\hat{h}\)’s, i.e, \(\{\hat{h}_{n_1}, \hat{h}_{n_1}, \dots, \hat{h}_{n_k}\}\) from \(\{\hat{h}_i\}_{i=1}^{N}\), since using only the nearest residual for correction may lead to incorrect inference predictions if a wrong residual is selected from a different class. Finally, following ResMem [55], we calculate the residual correction term \(\boldsymbol{r}_\text{eval}\) by a weighted average of the corresponding residuals \(\{\boldsymbol{r}_{n_1}, \boldsymbol{r}_{n_2}, \dots, \boldsymbol{r}_{n_k}\}\), with weights \(\{s_1, s_2, \dots, s_k\}\) that are inversely proportional to the distance from \(\hat{f}(x_\text{eval})\), as: \[\mathbf{r}_\text{eval} = \sum_{i=1}^{k}s_i\mathbf{r}_i, \quad s_i = \frac{e^{-(\hat{f}(x_\text{eval}) - \hat{h_i})/\tau}}{\sum_{j=1}^{k} e^{-(\hat{f}(x_\text{eval}) - \hat{h}_j)/\tau}},\] where \(\tau\) is a temperature hyperparameter. We add the residual-correcting term \(\mathbf{r}_\text{eval}\) to the model output \(\hat{f}(x_\text{eval})\) to obtain the corrected output \(\hat{f}(x_\text{eval})_\text{corrected}\) as: \[\hat{f}(x_\text{eval})_\text{corrected} = \hat{f}(x_\text{eval}) + \mathbf{r}_\text{eval}.\]

5 Experiments↩︎

5.1 Experimental setup↩︎

We perform experiments on four datasets: CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-200, and ImageNet-1K. For all datasets, our experiments are conducted on both a disjoint setup [56] and a Gaussian scheduled setup [57], [58]. We report the average and standard deviation results in three different seeds, except ImageNet-1k due to computational resource constraints [5], [31].

To evaluate anytime inference performance, we use the area under the curve accuracy(\(A_{auc}\)) [5], [24], which measures the area under the accuracy curve. We also use last accuracy(\(A_{last}\)) which measures accuracy at the end of training. For detailed information about the experimental setup, refer to the Supplementary.

5.1.0.1 Baselines.

We compare EARL with various CL methods in different categories: regularization (EWC [38]), network expansion (MEMO [40]), replay (ER [59], ER-MIR [4], REMIND [30] , SCR [60], ODDL [35]), and distillation (DER++ [36], X-DER [35]). For more details on the implementation of these methods and a comparison with NC-FSCIL [22], which attempts to induce neural collapse in the context of few-shot class incremental learning, please refer to the supplementary material.

5.1.0.2 Implementation details.

We use three components to architect our model: a backbone network \(g(\cdot)\), a projection MLP \(p(\cdot)\), and a fixed ETF classifier \(\mathbf{W}_\text{ETF}\) (i.e, our model \(f\) can be defined as \(f(x) = p \circ g(x)\)). For the projection layer, we attach an MLP projection layer \(p_{\theta_p}\) to the output of the backbone network \(g_{\theta_g}\), where \(\theta_g\) and \(\theta_p\) denote the parameters of the backbone network and the projection layer, respectively (i.e, the model \(f\) can be defined as \(f(x) = p \circ g(x)\)), following  [22], [61], [62]. For all methods, we use ResNet-18 [63] as the backbone network.

Following [5], [64], we employ memory-only training, where a random batch is selected from memory at each iteration. Furthermore, for episodic memory sampling, EARL uses the Greedy Balanced Sampling strategy [65]. We describe the details about hyperparameters and the pseudocode of EARL in Supplementary for the sake of space.

5.2 Results↩︎

We first compare the accuracy of online continual learning methods, including EARL, and summarize the results in Table 1. As shown in the table, EARL outperforms other baselines on all benchmarks, both in disjoint and Gaussian-scheduled setups, except \(A_\text{last}\) in ImageNet-1K disjoint setup. In particular, high \(A_\text{AUC}\) suggests that EARL outperforms other methods for all the time that the data stream is provided to the model, which implies that it can be used for inference at anytime.

Furthermore, EARL, SCR, ER, MIR, DER, and ODDL do not use task boundary information during training, i.e, task-free, in contrast to EWC, X-DER, MEMO, and REMIND which use task-boundary information, i.e, task-aware. Nevertheless, EARL outperforms these methods even in the disjoint setup where utilizing task boundary information is advantageous due to abrupt distribution shifts at the task boundaries, except MEMO in ImageNet-1K disjoint. Since MEMO not only uses the task boundary information but also expands the network per task, this extra advantage of using a larger model is emphasized in large-scale datasets such as ImageNet which require a large learning capacity of the model. Please refer to the supplementary material for details of the computational budget comparison among the baseline methods.

5.2.0.1 Memory budget.

We consider not only the size of episodic memory, but also the size of the model, logits, and other components, following MEMO [40]. In the case of the memory budget for CIFAR-10 and the architecture of ResNet-18 (size of 42.6MB), EARL incurs an additional memory cost of 0.2MB to store feature-residual pairs. EWC requires an additional cost of 85.2MB due to the need to store Fisher Information per parameter and the previous model. MEMO, which expands the network per task, incurs an additional cost of 32MB per task. DER also requires an additional cost of 0.1MB to store logits during distillation.

5.3 Ablation study↩︎

We conduct an ablation study on the two components of EARL, preparatory data training and residual correction, and summarize the results in Table 3. Although both components contribute to performance improvement, preparatory data training shows a larger gain in performance. Training with preparatory data improves the baseline by 2.7% \(\sim\) 3.3% in \(A_{auc}\) and 2.1% \(\sim\) 4.4% in \(A_{last}\). When combined with residual correction, we observe further gains across all metrics in both disjoint and Gaussian-scheduled setups.

For further analysis, Fig. 6 visualizes the cosine similarity between the output features of 50 randomly selected test set samples of novel class ‘4’ and the classifier vectors \(\mathbf{w}_i\). In baseline (a), the features of the class 4 are strongly biased towards the classifier vectors of the old classes (i.e, \(\mathbf{w}_0\), \(\mathbf{w}_1\), \(\mathbf{w}_2\), \(\mathbf{w}_3\)), rather than the correct classifier, \(\mathbf{w}_4\). (c) When residual correction is used, some samples show high similarity with \(\mathbf{w}_4\) compared to the baseline.

However, since incorrect residuals can be added due to the bias problem, more samples have high similarity with \(\mathbf{w}_2\) than the baseline (i.e, wrong residuals are added). (b) When using preparatory data training, the bias toward \(\mathbf{w}_0\) and \(\mathbf{w}_1\) classes significantly decreases compared to the baseline. Fig. 5 also shows the effect of preparatory data, which reduces the similarity between the novel class features and existing classes. (d) Using both residual correction and preparatory data training shows a remarkable alignment with the ground truth classifier \(\mathbf{w}_4\). A more detailed analysis of the ablation results is provided in the Supple.

6 Conclusion↩︎

To better learn online data in a continuous data stream without multiple epoch training, we propose to induce neural collapse, which aligns last layer activations to the corresponding classifier vectors in the representation space. Unlike in offline CL, it is challenging to induce neural collapse in online CL due to insufficient training epochs and continuously streamed new data. We first observe that the bias of the new class towards existing classes slows the convergence of features toward neural collapse.

To mitigate the bias, we propose synthesizing preparatory data for unseen classes by transforming the samples of existing classes. Using the preparatory data for training, we accelerate neural collapse in an online CL scenario. Additionally, we propose residual correction to resolve the remaining discrepancy toward neural collapse at inference, which arises due to the continuous stream of new data. In our empirical evaluations, the proposed methods outperform state-of-the-art online CL methods in various datasets and setups, especially with high performance on anytime inference.

6.0.0.1 Limitations and Future Work.

Since our work uses ETF structure, it has an inherent limitation that the number of possible classifier vectors in the ETF classifier is limited by the dimension of the embedding space. Considering lifelong learning, where the number of new classes goes to infinity, it is interesting to explore the idea of dynamically expanding the ETF structure so that the model can continually learn the ever-increasing number of concepts in the real world.

6.0.0.2 Acknowledgement.

This work was partly supported by the NRF grant (No.2022R1A2C4002300, 20%) and IITP grants (No.2022-0-00077 (10%), No.2022-0-00113 (25%), No.2022-0-00959 (10%), No.2022-0-00871 (10%), No.2020-0-01361 (5%, Yonsei AI), No.2021-0-01343 (5%, SNU AI), No.2021-0-02068 (5%, AI Innov. Hub), No.2022-0-00951 (5%)) funded by the Korea government (MSIT) and CARAI grant funded by DAPA and ADD (UD230017TD) 5%. This work was done while J. Choi was with Yonsei University and partially while H. Lee, M. Seo and H. Kim was with LG AI Research.

Note: Blue characters denote the reference of the main paper. ³

7 Anytime Inference in Online CL (L164)↩︎

In online CL, new data continuously arrive in a stream rather than in a large chunk (e.g, task unit). Several previous works [2][66] train the model only after a large chunk of new data accumulates, which leads to poor inference performance on new data during data accumulation, since the model is not trained during the accumulation period [6, 27].

However, inference queries can occur at any time including the data accumulation, i.e, anytime inference, whose importance has been emphasized by recent research [67], [68] [17, 27, 36]. Consequently, we focus not only on \(A_{last}\), which is measured after the learning has finished for all data, but also on \(A_\text{AUC}\), which measures the average accuracy during training, i.e, ‘anytime inference’ performance [27].

8 Analysis about Negative Transformation (L332)↩︎

Negative transformation is a transformation that modifies semantic information of original images and is widely used in self-supervised learning [4, 5] and out-of-distribution (OOD) detection [25, 47, 50]. We consider various negative transformations such as negative rotation (i.e, rotation by 90, 180, and 270 degrees) [15, 18], patch permutation [69], negative CutMix, and Gaussian noise. Negative Cutmix is a kind of Cutmix [70] that finely divides an image into small patches and mixes it with another finely divided image. Gaussian noise refers to filling all the image pixels with a Gaussian random noise. Even though Gaussian noise is not a kind of transformation, it is a straightforward approach to generate synthetic input that can be distinguished from the existing images; thus, we also consider this.

Among them, we choose to use rotations 90, 180, and 270 degrees as the negative transform set \(T\), because the rotation transformation is simple to implement, widely applicable from low- to high-resolution images, and also outperforms other transformations in our empirical evaluations, as we can see in Tab. 4. Unlike rotation transformation, which preserves image continuity while modifying semantic information, patch permutation or Negative Cutmix could cause discontinuities at the boundary of patches [71], [72], leading to loss of image features.

Furthermore, to compare the effect of various negative transformations, we compare the cosine similarities between the features and the corresponding ground truth classifier as shown in Fig. 7. As we can see in the cyan highlighting box, the rotation transformation promotes the convergence of \(\hat{f}(x)\) for class \(0\) towards the ground truth classifier vector \(\mathbf{w}_0\).

9 Effect of Negative Rotation Transformation (L304)↩︎

We use negative rotation, which rotates images by 90, 180, and 270 degrees [15, 18]. Since the vertical information of the image is more important than the horizontal information [22, 49] as mentioned in Sec.4.2, rotation by a large angle changes the semantic information [50]. To empirically verify that the negative rotation transformation alters the semantic information of the image, we generate four datasets by rotating the original data by 0, 90, 180, and 270 degrees and train the model using each dataset. Subsequently, we performed inference on all four datasets to check whether they have different semantics.

Specifically, we used CIFAR-10-R90, CIFAR-10-R180, and CIFAR-10-R270, created by applying rotation transformations of 90, 180, and 270 degrees to CIFAR-10 dataset, respectively. We summarize the results in Tab. 5.

10 Prevent Forgetting with ETF Classifier↩︎

We compare the performance and forgetting between a fixed ETF classifier and a learnable classifier. We summarize the results in Tab. 6. The fixed ETF classifier not only effectively prevents forgetting, but also has high accuracy.

11 Comparison of Baselines with Computational Constraint↩︎

Following  [73][17], we measure the number of FLOPs required for each method to complete one iteration. We then calculate the relative FLOPs with respect to the ER [43], which is the simplest method, and adjust the total training iteration to align the total FLOPs used for the entire training process. With this computational constraint, we compare online CL methods on disjoint and Gaussian scheduled setup for CIFAR-10, CIFAR-100, TinyImageNet and ImageNet-200. Furthermore, we measure the Average Online Accuracy (\(AOA\))[7, 17], which uses the newly encountered streaming data for evaluation before incorporating it into the training process. We plot the online accuracy on TinyImageNet in Fig. 8.

12 Comparison of Baselines on CLEAR-10/100↩︎

To evaluate the setups in which the domain of classes changes over time, we compare CL methods with the CLEAR benchmark and summarize the result in Tab. 8. In our experiments, we address a more realistic scenario, called the domain-class-IL setup, where both novel classes and domain shifts take place. This contrasts with domain-IL setups, where all classes are initially provided and only the data distribution changes over time.

EARL significantly outperforms the baselines in CLEAR-10/100 benchmarks and achieves high performance. We perform experiments under the computational constraints mentioned in Sec. 11.

13 Comparison of Computational Budget (L)↩︎

EARL requires an additional computation for residual correction, which calculates \(k\) nearest features, and FLOPs of additional cost can be formulated as \(d \times N \times 3 + k \times N\), where \(d\) is the dimension of stored features, \(N\) is the total number of feature-residual pairs, and \(k\) denotes \(\text{top}_k\) in \(k\)-NN. The first term is for calculating the distances between the stored features and the feature of the inference image and is multiplied by 3 since it involves subtraction, squaring, and addition operations. The second term is the cost of selecting \(\text{top}_k\) features among \(N\) features. Compared to the cost incurred by the naive inference process, which requires forward flops of the model, it involves a very minimal amount of additional cost. Taking the example of ImageNet-200 with the ResNet-18 architecture that has \(\frac{1}{3}\) GFLOPs in the model forward, only 0.5% of additional cost is consumed, since we use \(N=2,000(=10\times200)\) and \(k=15\).

14 Details About Experiment Setup (L404)↩︎

This paper focuses on online class-incremental learning in two types of setups: disjoint [35] and Gaussian scheduled [74] [45, 51]. In the disjoint setup, each class is assigned to a specific task, i.e, tasks do not share any classes. On the other hand, in the Gaussian scheduled setup, the arrival time of each class follows a Gaussian distribution \(\mathcal{N}(\mu_i, \sigma)\). Since the class distribution is shifted every time step, the Gaussian scheduled setup is a boundary-free setup. We set \(\sigma\) to 0.1 and \(\mu_i\), mean of the class \(i\), to \(\frac{i}{N}\), where \(N\) is the number of classes.

15 Effect of \(k\)↩︎

\(k\), which is a hyperparameter that determines the number of the nearest features from the inference image used to weight-sum to calculate the residual, has a negligible impact, except for \(k=1\) as incorrect residual largely affects predictions. We illustrate the consistency of accuracy at various values of \(k\) in Fig. 9.

16 Hyperparameters (L428)↩︎

For all methods, we use Adam optimizer [75] with a constant learning rate (LR) of 0.0003. For data augmentation, we use RandAugment [76]. For hyperparameters such as iteration, memory size, and number of tasks for each dataset, we follow prior works [2, 27, 40]. Specifically, we set the number of iterations as 1, 3, 3, 0.25, and 0.25, and memory sizes as 500, 2,000, 4,000, 4,000, and 10000 for the CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-200, and ImageNet-1k datasets, respectively. To ensure a fair comparison among methods, we use the same batch size in all methods. Specifically, we use 16, 32, 32, 256, and 256 for CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-200, and ImageNet-1k datasets, respectively. Note that the number of preparatory data is included in the batch size for EARL, i.e, batch size = the number of retrieved samples from memory + the number of preparatory data.

To ensure that EARL does not depend on a specific dataset, we use the following hyperparameters for all datasets: \(\tau=0.9, k=15, d=4096\) and \(\lambda=1\), where \(\tau\) refers to the temperature parameter during residual correction, \(k\) refers to the number of samples for \(\text{top}_k\) in \(k\)-NN during residual correction, \(d\) refers to the output dimension of the projection layer, and \(\lambda\) refers to the loss balancing parameter between real data training and preparatory data training. In addition, for \(N\), the total number of stored feature-residual pairs in EARL, we used \(N=10*\left| C\right|\) for all dataset, where \(\left|C\right|\) is the number of classes seen so far.

17 Implementation of Baselines (L587)↩︎

Some of the baselines assumed multi-epoch training, i.e, offline CL, so we modified them to be used in online CL for comparisons with our proposed method and other baselines.

17.0.0.1 REMIND.

Remind is a feature-replay method and freezes front layers after base initialization, offline training phase using a subset of the data during the initial stage of training. We modified the base initialization of REMIND [20] to be suitable for use in online CL. In offline CL setup, REMIND freezes 7 layers after base initialization. However, as shown in Table 9, in online CL, it is not suitable to freeze many layers compared to offline CL, as the model is not sufficiently trained. We studied various numbers of frozen layers and revealed that freezing 6 layers (i.e, 3 blocks) is the optimal number for freezing in ResNet-18 in CIFAR-10 and CIFAR-100. Compared to freezing 7 layers, which is the original proposed freezing criterion, REMIND performed comparatively well when freezing 6 layers in the online CL, despite the decrease in the number of stored features due to the larger sizes of stored features. Regarding the hyperparameters used in REMIND, such as the size of the codebook and the number of codebooks for product quantization, we performed additional hyperparameter search experiments with CIFAR-100, as shown in Table 11, and used the same hyperparameter in all remaining datasets.

For CIFAR-10, CIFAR-100, and TinyImageNet, which have a relatively small number of images, we increased the proportion of the initialization sample base from 10\(\%\) to 60\(\%\) of the total samples, since 10\(\%\) images are not enough for the lower level layers to represent highly transferable features, leading to low performance. In ImageNet-200 and ImageNet-1k, we followed the original setting (i.e, 10\(\%\) base initialization samples and freezing 7 layers after base initialization), following [20].

17.0.0.2 MEMO.

MEMO [63] retrieves samples that are relatively close to the class mean feature at every task boundary and stores them in episodic memory for replay. However, in online CL, the model can not access all data for the current task, i.e, it continuously updates the memory using stream data from the current task. Therefore, the sampling strategy of MEMO is replaced by class-balanced random sampling, which is used to replace the sampling strategy of RM [2] for online CL in [27].

18 Comparison between NC-FSCIL and Vanilla ETF↩︎

NC-FSCIL [56] is a recently proposed offline CL method that attempts to induce neural collapse in few-shot class incremental learning (FSCIL), a CL setup with few training samples per class. NC-FSCIL uses the fixed ETF classifier, a backbone network \(f\), and a projection layer. It freezes the backbone network after training the base task and further fine-tunes the projection layer in the following incremental tasks by using the replay memory \(\mathcal{M}^{(t)}\), which stores the class-wise mean features \(\mathbf{h}_c\) retrieved from the backbone network for each old class \(c\) as follows: \[\mathcal{M}^{(t)} = \left\{\mathbf{h}_c|c\in\bigcup_{j=0}^{t-1}C^{(j)}\right\}, ~~ 1\le t\le T,\] where \(\mathbf{h}_c = \text{Avg}_i\left\{f\left(x_i, \theta\right)|y_i=c\right\}\), \(T\) refers to the total number of tasks, and \(C^{(j)}\) refers to the set of classes for task \(j\).

Considering that our setup is neither an offline CL nor a few-shot class incremental learning setup, for a fair comparison with online CL methods, we modify the freezing strategy of NC-FSCIL. Instead of freezing the whole backbone, we freeze only 6 layers, which achieved the best performance in online CL as shown in Table 9.

Despite the high performance of NC-FSCIL in offline CL, it has a lower performance than that of vanilla ETF (i.e, baseline of EARL without preparatory data training and residual correction) in online CL, as shown in Tab. 12.

19 Pseudocode for the Our Method (L428)↩︎

Algorithm [algo:algo95train] and Algorithm [algo:sdp] provide detailed pseudocode for EARL.

20 Detailed analysis of ablation results (L509)↩︎

Note that exclusively relying on residual correction without the training of preparatory data may result in the addition of incorrect residuals due to bias problem, as illustrated in Fig. 10-(a). The bias in CL causes the features of novel classes and old classes to overlap. When the features of multiple classes are clustered together, as in Fig.4-(a), residuals of one class can be added to the residual-correcting term of other classes in the cluster, since we select the residuals using \(k\) nearest-neighbors of corresponding features. Thus, the residuals of old classes are often added to novel class samples and vice versa, hurting the accuracy of both old and novel classes.

In this context, the use of preparatory data not only accelerates the convergence of ETF during training, but also promotes accurate residual addition during inference. In conclusion, the combination of residual correction and preparatory data training effectively aligns the model output with the corresponding ETF classifier, as demonstrated in Fig. 10-(b).

21 Properties of Neural Collapse↩︎

(NC1) Collapse of Variability: The last layer feature output of each data point collapses toward the class mean feature of its respective class. In other words, \(h_{k,i}\), last layer feature of sample \(i\) in class \(k\), collapse to \(\mu_k = \sum_{i=1}^{n_k}h_{k,i}\) for \(\forall k\in [1,K]\) where \(n_k\) is the number of samples for class \(k\). By considering within-class covariance and between-class covariance \[\begin{align} \Sigma_W = \frac{1}{K}\sum_{k=1}^{K}\frac{1}{n_k}(\sum_{i=1}^{n_k}(h_{k,i} - \mu_k)), \\ \Sigma_B = \frac{1}{K}\sum_{k=1}^{K}(\mu_k - \mu_G), \end{align}\] where \(\mu_G = \sum_{k=1}^{K}\mu_k\), empirical variability can be measured as \[NC1 := \frac{1}{K}trace({\Sigma_W \Sigma_B^\dagger}).\]

(NC2) Convergence to simplex equiangular tight frame (ETF): Class means \(\mu_k (k \in [1,K])\) centered by the global mean \(\mu_G\) converge to vertices of a simplex ETF structure, i.e., matrix \(M = [m_1~m_2~\cdots~m_K]\) where \(m_k=\frac{\mu_k - \mu_G}{\|\mu_k - \mu_G\|^2}\) satisfies the following equation: \[\mathbf{M}\mathbf{M}^T = \frac{1}{K-1}(K\mathit{I}_K - \mathbf{1}_K \mathbf{1}_K^T).\] The degree of convergence can be measured using: \[\frac{\mathbf{M}\mathbf{M}^T}{\|\mathbf{M}\mathbf{M}^T\|_F} - \frac{1}{\sqrt{K-1}}(\mathit{I}_K-\frac{1}{K}\mathbf{1}_K \mathbf{1}_K^T).\]

(NC3) Convergence to self-duality: Classifier \(\mathbf{W}\) converges to the simplex ETF \(\mathbf{M}\) formed by recentered feature mean, and during this convergence, the classifier vector \(w_k\) aligns with their corresponding feature mean \(m_k\) where \(w_k\) means classifier weight for class \(k\), \(k \in [1,K]\), i.e, \[\frac{\boldsymbol{M}}{~\|\boldsymbol{M}\|_F} = \frac{\boldsymbol{W}}{~\|\boldsymbol{W}\|_F}\] Duality can be measured by measuring: \[\frac{\mathbf{W}\mathbf{M}^T}{\|\mathbf{W}\mathbf{M}^T\|_F} - \frac{1}{\sqrt{K-1}}(\mathit{I}_K-\frac{1}{K}\mathbf{1}_K \mathbf{1}_K^T).\]

References↩︎