1 Introduction↩︎

Deep neural networks often forget previously learned knowledge when continually learning new tasks [1]. This is mainly because the network parameters are overwritten during the continual learning process, which in turn interferes with preserving prior knowledge of a network. To address this problem, conventional approaches adopt regularization techniques [2]–[5] or employ external memory to store previous data [6]–[8]. These methods, however, are often suboptimal for large-scale models, , vision transformers (ViTs) [9], for which the forgetting problem becomes more severe [10]. As an alternative, prompt-based continual learning methods have recently been proposed [11]–[14]. By introducing a small set of learnable prompts and optimizing the prompts while freezing the backbone, these methods encourage a network to effectively learn new tasks while retaining previously learned knowledge.

Current prompt-based approaches [11]–[14] introduce an additional prompt token into a ViT encoder by leveraging a set of key-value pairs storing learnable prompts. Specifically, given a query feature representing an input image, these methods first associate the keys with the query. The matched values (, prompts) are then aggregated using, , nearest neighbor selection [12] or weighted summation [13], to represent the optimal prompt token for each image. This suggests that the efficacy of prompt-based approaches relies on how well the prompts can represent the distribution of the query features extracted from input images. In continual learning scenarios, however, a single task inherently exhibits high intra-task variations due to the presence of multiple distinct object classes (see Fig. 1 (a)). The feature variations between images become even more pronounced as the continual learning process progresses and new tasks are incrementally introduced, making the distribution highly complex to capture. Moreover, all current methods learn a static embedding to obtain prompt tokens, that is, they predominantly exploit a set of deterministic prompt vectors. Because the fixed vectors are restricted to representing specific values, their limited expressive power fails to properly capture the highly diverse patterns present across the sequence of tasks.

In this paper, we first identify a prompt collapse problem that is prevalent across existing prompt-based approaches [11]–[14], in which different prompt tokens become highly correlated with one another and fail to capture the diverse data distributions in the continual learning process (Fig. 1 (b)). To address this issue, we propose a novel prompt-based framework exploiting a set of probabilistic prompts, in contrast to the deterministic ones used in previous works. By modeling prompts as distributions, our framework encourages the prompts to better represent diverse patterns of images while avoiding the prompt collapse problem. Consequently, our probabilistic formulation yields consistent performance advantages over existing methods (Fig. 1 (c)). We also introduce a regularization scheme that enforces smooth transitions in prompt distributions during training, ensuring a stable continual learning process and better mitigating the forgetting problem. Extensive experiments on standard benchmarks demonstrate that our framework consistently outperforms state-of-the-art approaches [11]–[14] in terms of final average accuracy (FAA) and cumulative average accuracy (CAA) with a negligible computational overhead. Our main contributions can be summarized as follows:

• We identify a prompt collapse problem among prompt-based continual learning frameworks, in which the prompt tokens fail to represent the diverse patterns present in a sequence of data. To address this limitation, we present a novel continual learning framework exploiting probabilistic prompts, enabling the model to better capture diverse query feature distributions.

• We introduce a distribution regularization loss that prevents abrupt changes in probabilistic prompts and better retains previously learned knowledge, thereby further mitigating the forgetting problem.

• We set a new state of the art on standard benchmarks, including CIFAR100 [16], ImageNet-R [15], and CUB200 [17], demonstrating the effectiveness and efficiency of our method with extensive experimental results and ablation studies.

2 Related Work↩︎

Continual learning aims to learn new knowledge progressively, while avoiding the forgetting problem [1]. It is generally divided into three settings: Task-incremental learning (TIL), domain-incremental learning (DIL), and class-incre-mental learning (CIL) [18]–[21]. In TIL, training and evaluation are performed on distinct tasks with explicit task identifiers [19], [22], [23], while DIL focuses on generalizing across various domains that share the same label space [19], [21]. On the other hand, CIL attempts to classify previously learned classes jointly, without relying on any task information [19]–[21]. To this end, CIL methods should preserve prior knowledge, while learning new classes continually, which is more challenging than other settings [11]–[14], [19]–[21]. A key challenge in CIL is thus balancing the plasticity and rigidity of a model, which are the abilities to learn new concepts and preserve previously learned knowledge, respectively [24]. CIL methods can further be categorized into three groups based on how they address this trade-off: architecture-based, regularization-based, and rehearsal-based approaches. Architecture-based methods expand [25] or adapt [26] networks to avoid forgetting, while acquiring knowledge from a new task. Regularization-based approaches [2], [8], [27] encourage parameters, pertinent to previous tasks, to be preserved during training, alleviating the forgetting problem. Rehearsal-based methods introduce a replay buffer to store past examples [28], [29] or latent features [30] explicitly, generally outperforming the other two approaches.

Prompt-based continual learning. With the remarkable success of large-scale models, such as ViTs [9], the works of [10], [31]–[33] explore adopting these models for continual learning. Specifically, they modify transformer architectures to retain previous knowledge [10], [31] or regularize attention scores [32] to preserve previously acquired information. Albeit effective, these approaches require optimizing parameters of the large-scale models, which is computationally demanding. To alleviate this problem, prompt-based continual learning methods [11]–[14], [34]–[37] have recently been introduced. They freeze pre-trained backbones to preserve their representation ability for alleviating the forgetting problem, and employ learnable prompts. Early works of [11], [12] present a task-conditioned prompting to capture task-relevant knowledge. In particular, they use a pool of learnable prompts, where each prompt is paired with a key. Given an input image, they first extract a query feature, and then select prompts based on a key-query matching process, encouraging the prompts to learn task-specific knowledge during training. However, these methods cannot optimize prompts end-to-end, as they select the prompts using the non-differentiable key-query matching process, e.g., nearest neighbor selection. To overcome this problem, recent approaches [13], [14] instead propose to learn prompt components, and generate prompt tokens for each input image dynamically by aggregating the components based on key-query similarities, encouraging an end-to-end optimization. Moreover, CODA-P [13] uses an element-wise attention mechanism to focus more on relevant queries in the aggregation process. VQ-Prompt [14] attempts to mimic the categorical perception strategy of the human brain, and proposes to use discrete component vectors. Another line of works [34], [35] generate prompts using features from intermediate layers with additional trainable modules. While these prompt-based methods are effective for continual learning, they rely on deterministic prompts, resulting in prompt collapse, which limits to represent diverse data distributions present in a sequence of data. In contrast to the prior works, we propose to sample prompts stochastically from learned distributions to better represent diverse query features across a task, alleviating the prompt collapse problem.

Probabilistic representation learning. Probabilistic representations have been exploited in various tasks in computer vision, including retrieval [38], classification [39], and segmentation [40]. Latent features are represented as probabilistic distributions, rather than deterministic ones, enabling more flexible modeling of uncertainty and semantic variations. Recently, the works of [41], [42] have explored probabilistic modeling for prompt learning. Specifically, ProDA [41] enhances CLIP [43] by representing each classifier weight as a probabilistic embedding. It estimates a distribution over weights using multiple textual prompts per class, trying to better capture the intra-class diversity of input images. PPL [42] models attributes of objects in images as probabilistic prompts for dense prediction tasks, such as instance segmentation, and samples multiple prompts to generate granular textual representations for capturing image details. These approaches are closely related to ours in that they use probabilistic prompts. In contrast to these works, our approach differs in the following aspects: (1) The prompts themselves in [41], [42] are modeled as learnable but deterministic vectors, with probabilistic distributions defined over their output representations. Such output-level modeling makes it difficult to address prompt-level collapse directly. In contrast, we parameterize the prompts themselves as probabilistic distributions and construct a query-conditioned mixture distribution from which we sample prompts, thereby addressing prompt collapse at the prompt level. (2) We do not exploit additional information to learn new knowledge incrementally, whereas the works of [41], [42] require a task identifier or prior knowledge for classes in the context of class-incremental learning. We instead propose a novel framework that selects prompts considering each input to capture the diversity of query features across the sequence of tasks. (3) We introduce a distribution regularization loss to avoid an abrupt change in prompts during training, which can occur when applying existing probabilistic methods [41], [42] to continual learning scenarios without considering the rigidity of a model. The regularization term helps to maintain the rigidity of the model, effectively mitigating the forgetting problem. (4) Our framework does not require additional trainable modules or networks as in [41], [42], except for the prompts and classifiers, and keeps pre-trained backbones frozen. This leads to greater efficiency and makes it more suitable for continual learning.

3 Method↩︎

In this section, we first identify the prompt collapse problem based on the empirical observation (Sec. 3.1) and describe our approach with an overview (Sec. 3.2). We then present a probabilistic prompting framework in detail (Sec. 3.3), including a training objective (Sec. 3.4).

3.1 Prompt collapse problem↩︎

Prompt-based CIL methods [11]–[14] optimize prompts without updating parameters in a backbone to preserve the representation capability of the pre-trained backbone, which is helpful for addressing the forgetting problem. Despite their effectiveness, we observe a critical limitation of existing prompt-based methods, the prompt collapse problem, namely that different prompt tokens become highly similar to each other (Fig. 2) and fail to represent the diverse data distributions across the sequence of tasks. Specifically, the feature variations between images become more pronounced as the continual learning process progresses, resulting in highly diverse data distributions. The existing methods fail to capture such complex distributions, since they set each prompt component as a fixed vector with limited representation capability. A straightforward way to mitigate this issue is to use more prompt components, but this still produces highly similar prompts, suggesting that prompt collapse cannot be alleviated by simply scaling up the number of prompt components. To address this, we propose a probabilistic prompt tuning framework that samples diverse prompts from learned distributions conditioned on query features.

3.2 Overview↩︎

We introduce a novel prompt-based CIL framework that samples prompt tokens from probabilistic distributions to address the prompt collapse problem in previous methods. We show in Fig. 3 an overview of our framework. We set each prompt component as a Gaussian distribution, parameterized by mean and covariance, which we call a prompt distribution. Given a query feature from an input image, we compute the distance between the query and each prompt distribution. We then form a mixture of the prompt distributions, from which we sample diverse prompts to capture the various patterns of input data in the continual learning process. In particular, we sample multiple prompts and aggregate them to obtain the final prompt token while improving learning stability. This stochastic sampling encourages exploring diverse prompts to represent the various patterns of images (Fig. 2 (e)). Following the standard protocol in [11], [13], [14], we adopt a prefix tuning (Pre-T) technique in order to provide representations of query features to a pre-trained model, by prompt tuning. Specifically, we prepend the sampled prompt tokens, concatenate the prompt tokens with input tokens, and input them into the pre-trained model. We optimize the prompt distributions end-to-end using a standard cross-entropy loss and our distribution regularization loss, over \(T\) tasks continually. We do not update parameters in a backbone directly to preserve the representation ability of pre-trained backbones, which is helpful for retaining the previous knowledge [11]–[14], [34], [35].

3.3 Probabilistic prompt tuning↩︎

Deterministic pools of prompts or component vectors in existing prompt-based CIL methods suffer from a prompt collapse problem, which limits their ability to capture diverse patterns across task sequences. To overcome the prompt collapse problem, we introduce a probabilistic prompting framework that samples prompts from learned distributions. To this end, we model each pool of prompts as a set of probabilistic distributions, and sample prompt tokens from a mixture of the distributions, allowing the prompt tokens to consider diverse patterns of the queries. Following [11]–[14], [34], [35], we assume that each input is associated with \(M\) prompt tokens, each of which is obtained from a corresponding pool of prompts. Each pool consists of \(N\) prompt distributions. For the \(m\)-th pool, we assume that its \(n\)-th prompt is normally distributed as follows: \[\label{eq:dist} \mathcal{N} \left( {\mu}(m, n), {\Sigma}(m, n) \right),\tag{1}\] where \(m \in \{1, \dotsc, M\}\) and \(n \in \{1, \dotsc, N \}\). We denote by \({\mu}\) a \(D\)-dimensional mean vector, and \({\Sigma}\) a \(D\times D\) diagonal covariance matrix with elements of variance, \({\sigma}^2(m, n) \in \mathbb{R}^D\), , \({\Sigma}(m, n) = \mathrm{diag}({\sigma}^2(m, n))\). In the following, we omit the index of the prompt token \(m\) for brevity, as the same process is applied to each pool independently.

Given a query feature \({q}\) from an input image, we form a mixture of the distributions to sample prompts for each prompt token. To be specific, we compute a relevance score \(s(n)\) for the \(n\)-th distributions of the pool w.r.t the query feature \({q}\) as follows: \[\label{eq:score} s(n) = \frac{\exp\left\{- S^2({q}, \mu(n), \Sigma(n))\right\}}{\sum_{n=1}^{N} \exp\left\{- S^2({q}, {\mu}(n), {\Sigma}(n))\right\}},\tag{2}\] where \[\label{eq:maha95dist} S({q}, {\mu}(n), {\Sigma}(n)) = \sqrt{ \left({q} - {\mu}(n)\right)^{\top} \left({\Sigma}(n) \right)^{-1} \left({q} - {\mu}(n)\right)}.\tag{3}\] Note that \({\Sigma}(n)^{-1}\) is computed in an element-wise manner, since \({\Sigma}(n)\) is diagonal. We then compute a mixture distribution for each input token using the relevance scores. That is, the mixture distribution for the pool is defined as follows: \[\label{eq:mixture} \mathcal{N}_{\mathrm{GM}} \left( {\mu}_{\mathrm{GM}}, {\Sigma}_{\mathrm{GM}} \right),\tag{4}\] where \[\label{eq:mix95mean} {\mu}_{\mathrm{GM}} = \sum_{n} s(n) {\mu}(n),\tag{5}\] and \[{\Sigma}_{\mathrm{GM}} = \mathrm{diag}({\sigma}_{\mathrm{GM}}^2),\] with \[\label{eq:mix95var} {\sigma}_{\mathrm{GM}}^2 = \sum_{n} s(n) \left( {\sigma}^{2}(n) + \left( {\mu}(n) - {\mu}_{\mathrm{GM}} \right)^{2}\right).\tag{6}\]

From the mixture distribution in Eq. 4 , we sample a probabilistic prompt \(\tilde{P}\) for a prompt token using the reparameterization trick [44], enabling training our model end-to-end: \[\label{eq:sampling} \tilde{P} = {\mu}_{\mathrm{GM}} + {\sigma}_{\mathrm{GM}} \odot {\epsilon}, \text{~where~} {\epsilon} \sim \mathcal{N}\left( 0, \mathbf{I} \right).\tag{7}\] To obtain the final prompt \(\hat{P}\) for each prompt token, we draw \(N_s\) independent samples \(\{\tilde{P}_k\}_{k=1}^{N_s}\) from the mixture distribution using Eq. 7 . This improves learning stability by reducing the variance introduced by the stochastic sampling process. However, naively aggregating them via a simple average may fail to reflect the diverse patterns of the query feature, as each sample contributes equally regardless of its relevance to \({q}\). Instead, we aggregate the samples in a way that preserves their diversity while reflecting the characteristics of the query feature, as follows: \[\label{eq:final95prompt} \hat{P} = \sum_{k=1}^{N_s} w_k \tilde{P}_k,\tag{8}\] with the aggregation weight \(w_k\) defined as follows: \[\label{eq:final95weight} w_k = \frac{\exp \left( \mathrm{sim}(\tilde{P}_k, q)\right)}{\sum_{j=1}^{N_s} \exp \left( \mathrm{sim}(\tilde{P}_j, q) \right)},\tag{9}\] where \(\mathrm{sim}(\cdot,\cdot)\) is the cosine similarity. Note that \(\hat{P}\) reduces to a simple average when \(w_k = 1/N_s\), which we empirically find to be suboptimal (Sec. 4.3). More detailed analyses on the aggregation process can be found in the supplementary material.

The \(M\) prompt tokens obtained from the corresponding pool are then concatenated: \[\hat{\mathbf{P}} = [\hat{P}(1), \dots, \hat{P}(M)] ^{\top} \in \mathbb{R}^{M \times D}.\] We prepend \(\hat{\mathbf{P}}\) to the input by using the Pre-T technique. Note that the sampling process requires only a single forward pass, as we aggregate the \(N_s\) samples before applying the Pre-T technique, thus incurring negligible computational and memory costs (Sec. 4.3). We provide an overall sampling algorithm of our approach in the supplementary material.

3.4 Training loss↩︎

We train our model with an overall objective, balanced by the regularization parameter of \(\lambda\), as follows: \[\label{eq:total95loss} \mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda \mathcal{L}_{\mathrm{DR}},\tag{10}\] where we denote by \(\mathcal{L}_{\mathrm{CE}}\) and \(\mathcal{L}_{\mathrm{DR}}\) cross-entropy (CE) and distribution regularization losses, respectively. The distribution regularization term alleviates an abrupt change in the distributions, while optimizing distributions (or prompts): \[\label{eq:loss} \mathcal{L}_{\mathrm{DR}} = \frac{1}{M} \sum_{m=1}^{M} \frac{1}{N} \sum_{n=1}^{N}\mathcal{L}_{\mathrm{DR}}(m, n),\tag{11}\] where \[\label{eq:loss95component} \mathcal{L}_{\mathrm{DR}} (m, n) = \text{KL} \left(\mathcal{N} \left( \hat{{\mu}}(m, n), \hat{{\Sigma}}(m,n) \right) \, \middle\| \, \mathcal{N} \left( {\mu}(m, n), {\Sigma}(m,n) \right)\right).\tag{12}\] We denote by \(\mathcal{N} ( \hat{{\mu}}(m, n), \hat{{\Sigma}}(m,n) )\) an \(n\)-th prompt distribution of an \(m\)-th pool, similar to 1 , but cached from the previous training step. This regularization term minimizes the Kullback-Leibler (KL) divergence between prompt distributions over successive training steps to prevent the distributions from changing drastically, which helps to mitigate the forgetting problem.

4 Experiment↩︎

In this section, we describe implementation details (Sec. 4.1) and compare our method with the state of the art on class-incremental image classification (Sec. 4.2). We also provide a detailed analysis of our framework (Sec. 4.3).

4.1 Implementation details↩︎

Dataset and evaluation. We perform experiments on ImageNet-R [15], CIFAR-\(100\) [16], and CUB-\(200\) [17] for class-incremental learning. ImageNet-R consists of \(24\)K training and \(6\)K test images for \(200\) classes. The classes in ImageNet-R are taken from ImageNet [45], but the images feature different styles, such as sketches, cartoons, and graffiti. Following the standard protocol [11]–[14], we split ImageNet-R randomly into 5, 10, and 20 disjoint tasks, where each task contains a subset of non-overlapping classes. CIFAR-100 [16] provides 50K training and 10K test images for 100 object categories, and CUB-200 [17] contains 200 bird categories, consisting of 11.8K images. We randomly divide the CIFAR-100 and CUB-200 datasets into 10 tasks, where each task consists of 10 and 20 mutually exclusive classes for CIFAR-100 and CUB200, respectively. Following [14], [46], [47], we report final average accuracy (FAA) and cumulative average accuracy (CAA) for evaluation. In particular, FAA is the average test accuracy across all tasks after all incremental steps, while CAA is the average FAA, computed at the end of each incremental step. We provide more detailed descriptions of the evaluation metrics in the supplementary material.

0.4pt

Implementation details. Following the works of [11]–[14], [34], [46], [47], we adopt the ViT-B/16 [9] as our backbone. We use the ViT-B/16 pre-trained on ImageNet-1K [48] for ImageNet-R [15] and CIFAR-100 [16], and the ViT-B/16 pre-trained on ImageNet-21K [49] for CUB-200 [17]. For numerical stability, we parameterize each prompt distribution using its log standard deviation. To train our model, we use the AdamW optimizer [50] with \(\beta_1\) of 0.9 and \(\beta_2\) of 0.999, respectively. The initial learning rate is set to \(2.5 \times 10^{-3}\) and scheduled using cosine decay [51]. We optimize the prompts and classifiers for 20 epochs across all benchmarks following [14]. We set the batch size as 128 for CIFAR-100 and CUB-200, and 64 for ImageNet-R. We set the number of prompts sampled per input \(N_s\) and the balancing parameter \(\lambda\) to \(30\) and \(1e-6\), respectively, by performing a grid search on separate cross-validation splits on each dataset. Following VQ-Prompt [14], we set the number of prompt tokens \(M\) and the number distribution \(N\) to \(8\) and \(10\), respectively. We provide a detailed description of hyperparameter settings in the supplementary material. For all experiments, we implement our method using PyTorch and use NVIDIA A5000 GPUs.

4.2 Results↩︎

ImageNet-R. We compare in Table 1 our method with other prompt-based continual learning methods for 5, 10, and 20 tasks on ImageNet-R [15]. We report average scores over five runs with standard deviations. We reproduce the results for APT [36] using the official code with the same pre-trained weight for a fair comparison, and the numbers for other methods are taken from VQ-Prompt [14]. From the table, we can see that our approach outperforms state-of-the-art methods using prompts [11]–[14], [46] across all tasks by significant margins in terms of FAA and CAA. This indicates that diverse prompts sampled from our method better capture diverse patterns of queries in the continual learning scenarios, which is crucial for prompt-based class-incremental learning. The results also confirm that our distribution regularization loss, preventing the distributions from changing abruptly, alleviates the forgetting problem effectively.

6pt

CIFAR100 and CUB200. We also provide in Table 2 quantitative results on CIFAR-100 [16] and CUB-200 [17]. We provide average values of FAA and CAA over five runs with standard deviations. For a fair comparison, we reproduce the results of DualPrompt [11], CODA-P [13], and APT [36] using the official codes with the same pre-trained weights described in Sec. 4.1, while the numbers for the other methods are taken from VQ-Prompt [14]. From Table 2, we can see that our method surpasses all previous methods by clear margins in terms of FAA and CAA on both datasets, showing similar trends as in Table 1. This demonstrates that our framework can handle highly diverse patterns of input images, even for CUB-200 [17], a fine-grained scenario, where object classes are similar to each other.

5pt

4.3 Discussion↩︎

Comparison of pre-training regimes. We show in Table 3 quantitative results for 10 tasks on ImageNet-R [15], adopting ViT-B/16 pre-trained on iBOT-1K [52] and DINO-1K [53]. From Table 3, we can see that our method outperforms previous methods by clear margins in terms of FAA and CAA for both pre-trained weights, suggesting that our approach works favorably for different pre-training paradigms and it can generalize well across diverse initialization settings.

Per-task accuracy over incremental steps. We show in Fig. 4 per-task accuracy for the 10 tasks split on ImageNet-R [15]. From the figure, we can see that our method achieves higher accuracy across most tasks over the incremental steps. This is because diverse prompts enable the model to capture diverse image patterns in a sequence of data effectively. Additionally, the accuracy drop after incremental steps in our method is generally smaller than that of other methods, indicating that the distribution regularization loss addresses the forgetting problem effectively. These two observations demonstrate that our method learns new tasks effectively, while retaining previously acquired knowledge.

r0.6

4pt

Ablation study. We provide in Table 4 an ablation study on each component of our method. As a baseline in the first row, we use the mean vector \(\mu\) alone with unit variance, e.g., \(\Sigma=\mathbf{I}\), where \(\mathbf{I}\) is a \(D \times D\) identity matrix. This baseline corresponds to a deterministic variant of our method, where the sampling process is removed while the other components remain unchanged. For the variant in the second row, we aggregate prompts, sampled from each distribution independently, using the relevance scores in Eq. 2 without forming a mixture. From the table, we have three findings as follows: (1) The first and second rows show that modeling prompts as probabilistic distributions provides better results. A plausible reason is that the stochasticity of the sampling process allows us to facilitate diverse prompts, which enables our model to capture various patterns in input images. (2) From the second and third rows, we can observe that our framework, that leverages a mixture of prompt distributions, performs better. This is because aggregating prompts, sampled from each distribution independently, may neglect the relations between distributions and input tokens. (3) From the last four rows, we can see that our distribution regularization loss mitigates the forgetting problem effectively by avoiding an abrupt change in distributions.

r0.5

6pt

Effect of irrelevant distributions. The relevance scores in Eq. 2 assign non-zero weights to all prompt distributions, which could, in principle, allow irrelevant distributions to affect the mixture. However, the softmax in Eq. 2 can suppress such distributions. To verify this, we show in Table 5 the results of constructing the mixture using only the \(K\) most relevant distributions, while discarding the others, on ImageNet-R [15]. Note that \(K=N\) corresponds to our method in Eq. 4 . The results show that discarding low-relevance distributions provides only marginal gains over our soft mixture, suggesting that the softmax suppresses them effectively. Meanwhile, the optimal \(K\) varies across task splits even within the same dataset. This introduces an additional scenario-dependent hyperparameter, which is particularly impractical in continual learning, where task distributions change over time and future tasks are unknown. We thus retain the soft mixture to avoid scenario-specific tuning, while achieving comparable performance.

r0.55

6pt

Effect of \(N_s\). To train our model, we sample multiple prompts from the learned mixture distribution and combine them using Eq. 8 . We show in Table 6 the effect of the number of sampled prompts, denoted by \(N_s\), in terms of FAA and CAA. From the table, we have three findings as follows: (1) From the first three rows, we can see that training becomes unstable when \(N_s\) is small, e.g., \(N_s = 1\), \(5\), and \(10\). This is because the randomness from insufficient sampling hinders our model from learning task information. In particular, the model fails to converge if we use a single prompt. (2) The fourth to sixth rows show that using a moderate number of prompts, such as \(N_s=20\), \(30\), or \(40\), enables our model to achieve stable training and better performance than the previous methods [11]–[14], [34], [46]. This indicates that the prompts from our method benefit from stochasticity in the sampling process, which helps to learn rich information of data distributions in the continaul learning process (3) We can see from the last row that using \(\mu_{\mathrm{GM}}\) yields reasonable, but worse results, compared to the cases of \(N_s\) of 20, 30, and 40, as the model is still exploring diverse prompts to represent various input images during training. This suggests that exploiting \(\mu_{\mathrm{GM}}\) at training time is sub-optimal, confirming once more the importance of stochastic sampling during training. We provide a more detailed analysis on the effect of \(N_s\) over a wider range of values in the supplementary material.

2pt

r0.55

2pt

Aggregation scheme. We aggregate multiple samples using Eq. 9 to obtain prompts. To verify the effectiveness of the aggregation strategy, we compare it with a baseline that averages the samples, that is, \(w_k=1/N_s\). We show in Table 8 the performance comparison between the two sampling strategies in terms of FAA and CAA without \(\mathcal{L}_{\mathrm{DR}}\). From the table, we can observe that the aggregation using Eq. 9 outperforms the simple average counterpart by significant margins. This suggests that a simple average, which treats all samples equally, fails to consider the query feature properly due to the inherent stochasticity of the sampling process. In contrast, our aggregation scheme in Eq. 9 prioritizes each sample adaptively based on the relation between the sampled prompts and a given query feature, allowing the final prompt to reflect the diverse patterns present in a sequence of data while focusing on the query feature.

Computational and memory costs. We show in Table 7 the computational and memory costs, including the memory incurred by each prompt during a single forward pass, inference speed, and training time per epoch of ours and previous methods [11]–[14]. From the table, we can see that our method has comparable computational and memory costs to the previous works, while outperforming them by a clear margin. This is because we average sampled prompts before applying the Pre-T technique, enabling a single forward pass per input, rather than \(N_s\) forward passes, and adding only negligible computational and memory cost. Furthermore, our framework maintains a memory footprint comparable to previous works [11]–[14], because we do not use key [11]–[14] or learnable attention matrices [13], while introducing additional parameters for \(\Sigma\).

4.4 Limitations and Future Work↩︎

Our framework effectively addresses the prompt collapse problem and outperforms existing methods across all benchmarks. Nevertheless, our method is currently evaluated using ViT-based pre-trained backbones on standard class-incremental learning benchmarks. Extending it to more realistic settings, such as open-world continual learning, and to diverse network architectures would be a promising direction for future work.

5 Conclusion↩︎

We have identified the prompt collapse problem prevalent in existing prompt-based continual learning methods, where the prompts tend to be highly similar to each other and fail to represent the diverse data distributions in the continual learning process. To address this, we have introduced a novel prompt-based framework that models each prompt as a probabilistic distribution and samples diverse prompts from a query-conditioned mixture of the learned distributions, enabling the model to effectively capture the diverse data distributions in the sequence of tasks. We have also presented a distribution regularization loss that prevents the prompt distributions from changing abruptly during the continual learning process, mitigating the forgetting problem. Extensive experimental results demonstrate that our framework better captures the diverse patterns of images than existing prompt-based methods, consistently outperforming them across all benchmarks by clear margins.

Acknowledgements↩︎

This work was partly supported by IITP grant funded by the Korea government (MSIT) (No.RS-2025-09942968, AI Semiconductor Innovation Lab (Yonsei University), No.RS-2022-00143524, Development of Fundamental Technology and Integrated Solution for Next-Generation Automatic Artificial Intelligence System, and No.2022-0-00124, RS-2022-II220124, Development of Artificial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities).

References↩︎

---

1. Corresponding author.↩︎