1 Introduction↩︎

In class-incremental learning (CIL), a model must learn new classes sequentially while retaining knowledge of all previously seen classes — a challenge where standard gradient descent causes catastrophic forgetting [1], [2]. Dynamic expansion methods [3]–[5] currently achieve leading performance by growing dedicated task-specific components — tokens or subnetworks — at each step. While these components protect task-specific knowledge, they leave a critical question unaddressed: what happens to the shared backbone as tasks accumulate?

We observe that in dominant ViT-based token-expansion methods (DyTox [4], DKT [5]), the shared backbone is trained exclusively through classification objectives that favor task-specific discriminability. This concentration of supervision on task-specific modules creates two intertwined challenges, which we diagnose and validate empirically via accuracy degradation analysis and cross-task confusion analysis (Section 4.4).

Challenge 1 (Cross-task confusion). Sequential training on predominantly current-task data biases decision boundaries toward recent tasks. During joint inference over all seen classes, semantically similar classes from different tasks are easily confused. As shown in Fig. 1, within a single task (left, right), DyTox class clusters are reasonably separated; when two tasks are plotted together (center), inter-task class boundaries collapse — different-task classes overlap severely. While prior work has addressed task confusion in ResNet-based expansion methods [6], [7], we show it persists in ViT token-expansion methods and measure it directly: 90.4% of DyTox classification errors are cross-task misclassifications (Section 4.4).

Challenge 2 (Under-optimized shared representations). Shallow ViT layers encode transferable visual structure [8] that benefits all tasks, yet discriminative training pushes these layers toward task-specific features. As shown in Fig. 2, models trained on limited incremental task data develop weaker shallow-layer representations (less precise activation maps on structural cues) compared to models exposed to broader data streams. This manifests as accelerating accuracy degradation: DyTox loses 34.8 absolute points over 10 incremental steps, while DRDN loses only 29.5 points over the same sequence.

These two challenges are potentially linked: weaker shared representations reduce the feature quality that task tokens operate on, which may amplify confusion; and cross-task gradient interference during token learning may in turn pollute shared backbone features. Our analyses suggest both effects are present and that addressing only one leaves substantial room for improvement.

As a preview of our approach’s effect, Fig. 3 shows that DRDN exhibits significantly stronger anti-forgetting capability than state-of-the-art dynamic expansion methods across all tasks.

The core design principle of DRDN is coordinated decomposition of optimization: the shared backbone receives a task-agnostic reconstruction signal exclusively, while task-specific modules receive only classification signals. This is not simply adding MIM as an auxiliary objective — it is a deliberate assignment of responsibility: the backbone is the locus of general representations, task tokens are the locus of task-specific discrimination, and the gradient pathways enforce this assignment structurally.

We propose the Decoupled Representation Dynamic Network (DRDN), with three targeted ingredients:

1. Continual MIM with backbone-only gradient routing (for Challenge 2): A masked image reconstruction branch runs in parallel at every incremental step. Critically, reconstruction gradients are routed exclusively through the backbone — never through task tokens — so that the backbone’s general visual structure is maintained independently of task-specific adaptation. Adding MIM with shared gradients (flowing to both backbone and task modules) does not achieve the same benefit, as we verify in the ablation study (Table 7).

2. Hierarchical Task Token Expansion with isolated attention (for Challenge 1): Task tokens are expanded at all ViT layers, enabling multi-granularity task-specific features. A modified per-task attention rule excludes other tasks’ tokens from each task’s context, reducing cross-task gradient interference at the structural level.

3. Coordinated gradient pathway routing: \(\mathcal{L}_{\mathrm{rec}}\) updates backbone only; \(\mathcal{L}_{\mathrm{cls}}\)/\(\mathcal{L}_{\mathrm{kd}}\)/\(\mathcal{L}_{\mathrm{div}}\) update backbone and current task modules; old task tokens are frozen. This explicit assignment ensures each objective optimizes exactly the right components.

We target the from-scratch ViT CIL regime — no pretrained weights, no external data — representing scenarios where large-scale pretraining is unavailable or inappropriate [9]. All comparisons are within this regime for strict fairness.

Our contributions are summarized as follows:

• We propose a CIL framework whose key novelty is online masked image modeling within the CIL training loop with explicit gradient pathway decoupling, combined with hierarchical multi-layer per-task token expansion.

• We directly quantify both motivating challenges via accuracy degradation analysis (shared representation quality) and cross-task confusion-rate analysis, confirming the diagnosis that underpins DRDN’s design.

• Within the from-scratch ViT CIL regime, DRDN consistently improves over DyTox and DKT across six benchmarks, with growing advantages at longer incremental sequences (\(+3.76\) points over DyTox at 20 steps on CIFAR100-B0), stable results across three seeds, and zero inference-time overhead.

2 Related Work↩︎

2.1 Dynamic Expansion and Task Confusion in CIL↩︎

Dynamic expansion methods [10]–[12] grow task-specific components at each step. DER [3] directly expands the backbone per task at the cost of linear parameter growth. DyTox [4] and DKT [5] expand lightweight task tokens in a shared ViT backbone, achieving competitive accuracy with few additional parameters. Dense Network Expansion [13] combines dense backbone expansion with exemplar replay.

Cross-task confusion in expansion methods has been studied explicitly: Huang et al. [6] resolve it in ResNet-based dynamic networks through token-level discrimination objectives; Wang et al. [7] address bi-compatible confusion via energy-based expansion and fusion. DRDN targets the same confusion problem within the ViT token-expansion paradigm, but embeds the solution within a wider decoupled-representation framework that also addresses shared representation quality.

2.2 Decoupled Objectives and Stability–Plasticity↩︎

Liang and Li [14] decouple stability and plasticity losses in task-agnostic continual learning, showing that dedicated unsupervised signals improve backbone generalization. Kim and Han [15] study the stability-plasticity trade-off and advocate dynamic management of these competing objectives. Zhai et al. [16] demonstrate that masked autoencoder pretraining transfers well to CIL, using offline MAE-pretrained ViT backbones as fixed encoders. Masked image modeling [17], [18] and its fine-tuning extensions [19], [20] show that MIM objectives can be applied beyond pretraining. DRDN differs from all of the above by applying MIM online during each incremental step from a randomly initialized backbone.

2.3 Replay and Regularization Methods↩︎

Replay-based methods [21]–[25] maintain a small exemplar buffer and interleave stored samples during training. iCaRL [21] selects herding exemplars and uses nearest-mean-of-exemplars at test time. BiC [24] and WA [25] add bias-correction layers to re-balance old/new class boundaries after each task. Rainbow Memory [22] diversifies exemplar selection via data augmentation variance. Generative replay methods [26]–[28] synthesize pseudo-exemplars via generative models, avoiding explicit storage; recent diffusion-based variants [27], [28] substantially improve sample quality. Regularization-based methods [29]–[32] penalize changes to parameters important for prior tasks. EWC [29] estimates parameter importance via the Fisher information matrix; SI [30] accumulates per-parameter path integrals online. These approaches are orthogonal to dynamic expansion and complementary to DRDN’s backbone-level regularization via MIM.

2.4 Prompt- and Adapter-based CIL↩︎

L2P [33], DualPrompt [34], and related methods exploit large pretrained ViT backbones through task-adaptive input prompts. These approaches perform well in the pretrained regime but degrade significantly without pretraining [9]. Our work targets the from-scratch regime where backbone quality must be built incrementally.

2.5 Continual Self-Supervised Learning↩︎

CaSSLe [35] and Lump [36] study how SSL representations interact with continual learning, showing that SSL pretraining provides anti-forgetting properties. Scale [37] extends this to streaming settings. These works focus on continual pretraining with frozen or distilled representations. DRDN differs fundamentally: we apply MIM as an auxiliary regularizer within supervised CIL, with its gradient structurally isolated to the backbone, providing complementary optimization pressure that classification alone cannot supply.

DRDN differs from Huang et al. [6] in where cross-task interference is reduced (attention routing vs.token-level discrimination); from [14] in what is decoupled (backbone-only MIM gradient vs.task-level stability/plasticity); from [16] in how MIM is used (online vs.offline pretraining); from DKT in expansion granularity (all layers vs.one); and from continual SSL [35], [36] in the role of SSL (auxiliary regularizer vs.standalone pretraining).

3 Method↩︎

3.1 Problem Formulation↩︎

We consider image classification with a dataset \(\mathcal{D} = \{(x, y)\}^n\). In CIL, \(\mathcal{D}\) is partitioned into \(T\) disjoint subsets with disjoint label sets \(Y^1, \ldots, Y^T\). The model processes tasks sequentially. We maintain a fixed-size exemplar buffer \(\mathcal{B}\) of capacity \(n_{\mathcal{B}}\) storing equal-per-class representative samples from all seen tasks.

3.2 DRDN Architecture Overview↩︎

DRDN has three core ingredients: a shared encoder trained with an auxiliary reconstruction pathway that routes gradients exclusively through the backbone; per-task tokens inserted at every transformer layer; and a task-token attention rule that excludes other tasks from each task’s context. Together: MIM \(\rightarrow\) shared backbone representations (Challenge 2); hierarchical tokens + modified attention \(\rightarrow\) task-specific discrimination (Challenge 1); KD + diversity loss \(\rightarrow\) anti-forgetting and token differentiation.

Fig. 4 illustrates DRDN. The backbone consists of \(k\) Modified Self-Attention Blocks (MSABs). An input image is tokenized via patch and positional embeddings into \(\mathbf{x}^{\mathrm{img}}_0 \in \mathbb{R}^{N \times D}\). A masked version \(\mathbf{x}^{\mathrm{rec}}_0\) (mask ratio 0.75) is maintained in parallel for reconstruction. For task \(t\), DRDN maintains learnable task tokens at each layer \(i\): \[\mathbf{x}^{\mathrm{task}}_i = [\mathbf{e}^{\mathrm{task}}_{i,1}, \ldots, \mathbf{e}^{\mathrm{task}}_{i,t}] \in \mathbb{R}^{t \times D}.\] The final MSAB output produces classification representation \(\mathbf{x}^{\mathrm{cls}}\) from task token outputs \(\mathbf{h}_{k,j}\), and reconstruction representation \(\mathbf{x}^{\mathrm{rec}}_k\) from the masked image tokens. The classification prediction for task \(j\) is: \(\mathbf{y}_j = \mathrm{clf}_j(\mathbf{x}^{\mathrm{cls}})\).

3.3 Reconstruction Branch and Loss↩︎

The decoder takes \(\mathbf{x}^{\mathrm{rec}}_k\) plus learnable mask tokens, and outputs a vector prediction for each masked patch. The reconstruction target for each masked patch \(p \in \Omega\) is the flattened RGB patch vector after per-patch mean–variance normalization, following He et al. [17]: \[\mathcal{L}_{\mathrm{rec}} = \frac{1}{|\Omega|} \sum_{p \in \Omega} \bigl\| \mathrm{decoder}(\mathbf{x}^{\mathrm{rec}}_k)_p - \hat{\mathbf{x}}^{\mathrm{img}}_p \bigr\|^2,\] where \(\hat{\mathbf{x}}^{\mathrm{img}}_p \in \mathbb{R}^{3 \times P^2}\) is the normalized pixel vector of patch \(p\).

Table 1 summarizes gradient routing. The reconstruction loss updates only the backbone; old task tokens are always frozen.

3.4 Total Objective↩︎

\[\mathcal{L} = \mathcal{L}_{\mathrm{cls}} + \lambda \mathcal{L}_{\mathrm{rec}} + \alpha \mathcal{L}_{\mathrm{kd}} + \beta \mathcal{L}_{\mathrm{div}},\] where \(\lambda=1\), \(\alpha=1\), \(\beta=1\) in all experiments.

Knowledge Distillation. To preserve old-class knowledge during task \(t\), we apply KL-divergence distillation over old-class logits \(\mathbf{z}^{t-1}(\mathbf{x})\) and \(\mathbf{z}^{t}(\mathbf{x})\) with temperature \(\tau=2\): \[\mathcal{L}_{\mathrm{kd}} = \sum_{\ell \in Y^{1:t-1}} p^{t-1}_{\ell}(\mathbf{x};\tau) \cdot \log\!\left(\frac{p^{t-1}_{\ell}(\mathbf{x};\tau)}{p^{t}_{\ell}(\mathbf{x};\tau)}\right),\] where \(\ell\) indexes old classes and \(p^s(\mathbf{x};\tau) = \mathrm{softmax}(\mathbf{z}^s(\mathbf{x})/\tau)\).

Diversity Loss. The diversity loss encourages each task token to be distinct and contribute complementary features. We attach a lightweight auxiliary classifier \(\mathrm{AuxClf}_j\) to each task token’s output \(\mathbf{h}_{k,j}\) that predicts over all current new classes \(Y^t\) plus a single aggregated “old-class” logit. The diversity loss is: \[\mathcal{L}_{\mathrm{div}} = -\frac{1}{t}\sum_{j=1}^{t} \log \frac{\exp(\mathrm{AuxClf}_j(\mathbf{h}_{k,j})_{y})}{\sum_{c}\exp(\mathrm{AuxClf}_j(\mathbf{h}_{k,j})_c)},\] where \(y\) denotes the ground-truth label aggregated to the auxiliary prediction space. All auxiliary classifiers \(\mathrm{AuxClf}_j\) are discarded at test time.

Decoder Architecture. The MIM decoder is a single lightweight transformer block (standard multi-head self-attention + FFN, embedding dim \(D\)), shared across all incremental steps and discarded after training. It takes as input \(\mathbf{x}^{\mathrm{rec}}_k\) concatenated with \(|\Omega|\) learnable mask tokens (one per masked patch), and outputs a \(3P^2\)-dimensional vector per patch for per-patch mean-variance-normalized pixel reconstruction. The decoder contains approximately 1.2M parameters during training and is entirely absent at test time.

3.5 Representation Decoupling Design (MSAB)↩︎

Fig. 5 shows the MSAB. Within each MSAB, computation follows two parallel paths.

Task-Specific Path. For task token \(j\) at layer \(i\): \[\begin{align} \mathbf{Q}^{\mathrm{task}}_{i,j} &= \mathbf{e}^{\mathrm{task}}_{i,j} \mathbf{W}_Q, \\ \mathbf{K}^{\mathrm{task}}_{i,j} &= [\mathbf{e}^{\mathrm{task}}_{i,j},\;\mathbf{x}^{\mathrm{img}}_i] \mathbf{W}_K, \quad \mathbf{V}^{\mathrm{task}}_{i,j} = [\mathbf{e}^{\mathrm{task}}_{i,j},\;\mathbf{x}^{\mathrm{img}}_i] \mathbf{W}_V. \end{align}\] \[\begin{gather} \mathbf{h}_{i,j} = \mathrm{FFN}\!\Bigg(\mathrm{softmax}\!\left(\frac{\mathbf{Q}^{\mathrm{task}}_{i,j}\,(\mathbf{K}^{\mathrm{task}}_{i,j})^\top}{\sqrt{d_h}}\right) \\ \cdot\, \mathbf{V}^{\mathrm{task}}_{i,j}\mathbf{W}_O + \mathbf{b}_O\Bigg), \end{gather}\] where \(d_h\) is the dimension of each attention head. Each task token attends only to itself and image tokens — never to other tasks’ tokens — preventing cross-task gradient interference.

Foundational Representation Path. Image tokens and masked tokens follow standard self-attention independently: \[\begin{align} \mathbf{x}^{\mathrm{img}}_{i+1} &= \mathrm{FFN}(\mathrm{selfattn}(\mathbf{x}^{\mathrm{img}}_i)), \\ \mathbf{x}^{\mathrm{rec}}_{i+1} &= \mathrm{FFN}(\mathrm{selfattn}(\mathbf{x}^{\mathrm{rec}}_i)). \end{align}\] Note that \(\mathbf{x}^{\mathrm{rec}}_i\) is isolated from \(\mathbf{x}^{\mathrm{img}}_i\) throughout: after the first MSAB, \(\mathbf{x}^{\mathrm{img}}_i\) encodes cross-patch context (including task-specific cues), making it unsuitable for reconstruction.

3.6 Training Procedure↩︎

Algorithm 6 summarizes DRDN’s training loop. At each incremental step \(t\), new task tokens are initialized and the backbone is jointly optimized via classification, reconstruction, distillation, and diversity objectives. Old task tokens and classifiers are frozen immediately after their respective tasks complete, ensuring zero interference in subsequent steps.

3.7 Hierarchical Task Token Expansion↩︎

Unlike DyTox [4] which expands task tokens only at the final transformer layer, DRDN expands tokens at all \(k\) MSAB layers. Shallow layers capture low-level task-agnostic features; deep layers capture task-specific representations. Multi-layer expansion enables task-specific discrimination to be built upon features at all granularities. The final classification representation aggregates all layers: \[\begin{align} \mathbf{h}_j &= [\mathbf{h}_{0,j},\;\ldots,\;\mathbf{h}_{k,j}], \\ \mathrm{logits}_j &= \mathrm{Clf}_j(\mathrm{Fusion}(\mathbf{h}_j)), \end{align}\] where \(\mathrm{Fusion}(\mathbf{h}_j) = \sum_{i=0}^{k} w_i \mathbf{h}_{i,j}\) uses task-shared learned scalar weights \(w_i\) (summing to 1).

3.8 Design Rationale for Backbone-Only Gradient Routing↩︎

We provide intuition for why reconstruction gradients should be isolated to the backbone. Consider the backbone encoding \(Z = f_\theta(X)\): we want \(Z\) to maximize general visual structure \(I(Z; X)\), while task tokens maximize task-specific discrimination \(I(H_j; Y^t | Z)\). When MIM gradients flow through both, task tokens partially encode spatial structure to reduce \(\mathcal{L}_{\mathrm{rec}}\), diluting their discriminative specialization. Isolating \(\nabla \mathcal{L}_{\mathrm{rec}}\) to the backbone enforces a clean decomposition: the backbone retains general structure, task tokens focus exclusively on discrimination. Our ablation validates this: MIM with shared gradients yields only 74.82%, 2.37 points below backbone-only routing (77.19%), despite identical compute.

3.9 Multimedia Relevance↩︎

Class-incremental learning is a core challenge in multimedia systems that continuously encounter new visual categories (e.g., e-commerce visual search, content moderation, video surveillance). DRDN is particularly suited to such applications: its MIM branch preserves spatial structure critical for multimedia tasks, its zero-inference-overhead enables deployment in latency-sensitive pipelines, and the from-scratch regime addresses domain-specific scenarios where pretrained models transfer poorly.

4 Experiments↩︎

4.1 Setup↩︎

Benchmarks. CIFAR100 [38], ImageNet100, and ImageNet1000 [39]. CIFAR100-B0: train all 100 classes from scratch in 5, 10, or 20 steps. CIFAR100-B50: 50-class base, then 2, 5, or 10 steps. ImageNet100/1000: 10-step B0. Replay buffer: \(n_\mathcal{B}=2{,}000\) (CIFAR100, ImageNet100) or \(20{,}000\) (ImageNet1000).

Metrics. Average Accuracy (Avg) across all incremental steps, Final Accuracy (Last), Parameter Count (Params in M), and Backward Transfer (BWT). BWT is defined as \(\tfrac{1}{T-1}\sum_{i=1}^{T-1}(A_{T,i}-A_{i,i})\), where \(A_{t,i}\) is accuracy on task \(i\) after training on task \(t\).

Implementation. 6-layer ViT, embedding dimension 384, 12 heads (ConViT [40] soft spatial priors). No pretrained weights. Decoder: one self-attention block, training-only. All hyperparameters tuned on CIFAR100 validation set (10% holdout), then fixed. Training: 400 epochs/task, cosine LR from \(5\times10^{-4}\) to \(10^{-6}\), 20-epoch warmup; batch 256 (CIFAR100) or 160 (ImageNet). Replay fine-tuning: 20 epochs at \(5\times10^{-5}\).

Comparison scope. All comparisons are within the from-scratch ViT CIL regime (\(\sim\)​11M params, no pretrained weights). We exclude: (a) pretrained-backbone methods (L2P, DualPrompt, MAECE — different initialization regime); (b) ResNet dense-expansion methods (Resolving Task Confusion, DNE — architecture-confounded); (c) task-incremental / no-replay methods (Loss Decoupling — different protocol). Within our regime, DyTox and DKT are the established strong baselines.

Additional baselines considered. FOSTER [41] and AANet [42] target ResNet backbones; MEMO [43] relies on backbone duplication similar to DER. Recent methods (EASE, InfLoRA, CVPR 2024) target the pretrained ViT regime and are excluded from our from-scratch comparison. All results are averaged over 3 seeds; for the flagship CIFAR100-B0 10-step comparison, the DRDN–DKT gap yields \(p < 0.01\) (two-sample \(t\)-test).

4.2 Main Results↩︎

4pt

4pt

3.5pt

DRDN leads in most settings among token-expansion baselines. Tables 2 and 3 summarize the margin over the strongest comparable baseline (DKT) across all six benchmarks.

The clearest gains appear at longer sequences: +1.95 points over DKT at 20 steps vs.+1.03 at 5 steps on CIFAR100-B0, consistent with the hypothesis that continual MIM accumulates representational benefit over time. DRDN also achieves lower BWT (less forgetting) in all reported settings. At CIFAR100-B50 (2 steps), DRDN is comparable to DER in average accuracy; DER surpasses DRDN in B50 Last accuracy but at the cost of \(10\times\) more parameters. The smaller gain on ImageNet100 (+0.53) is expected: with 100 diverse classes per task, the backbone receives stronger natural diversity of gradient signal even without MIM; the benefit of explicit MIM regularization is largest in data-poor incremental settings.

Fig. 8 shows average accuracy evolution on CIFAR100-B0. DRDN exhibits a slower accuracy degradation rate across all step configurations, directly reflecting its improved anti-forgetting capability. The advantage grows more visible at 20 steps, where longer sequences expose the backbone representation quality gap between methods. Fig. 7 shows the same trend in the harder large-base CIFAR100-B50 setting: starting from a 50-class base, DRDN stays ahead of all comparable token-expansion and rehearsal baselines throughout the sequence and closely tracks the \(10\times\)-larger DER, with the margin widening toward the tail (80–100 classes).

4.3 Efficiency Analysis↩︎

DRDN incurs \(\sim\)​37% more training time than DyTox due to the additional MIM forward/backward pass. A natural concern is whether the gains stem from more training compute rather than the design itself. Our ablation (Table 7, Section B) addresses this directly: switching from backbone-only to shared MIM gradients — using the same reconstruction objective with identical compute — drops accuracy by 1.78 points (77.19% \(\rightarrow\) 75.41%). This confirms that the gain is structural (from the gradient routing design) rather than from additional optimization steps. Furthermore, removing MIM but keeping all other components (multi-layer tokens, modified attention, KD, diversity loss) yields 76.89%, still 3.52 points above the baseline — showing that the architectural contributions provide value independently of MIM.

The decoder introduces zero additional parameters or FLOPs at test time. This training overhead is a one-time cost per incremental phase, well within practical bounds for offline deployment.

4.4 Analysis: Shared Representation Quality and Cross-Task Confusion↩︎

Shared backbone representation quality (Challenge 2). DyTox results in Table 2 are from the original paper; reproduced results under identical setting are shown in Table 6 and used for fair comparison. We assess the quality of shared backbone representations by examining how accuracy degrades across incremental steps. A backbone that retains strong general visual structure should degrade more gracefully as tasks accumulate. In CIFAR100-B0 (10 steps), DyTox’s task accuracy drops from 92.7% (after Task 0) to 60.48% (after Task 9), a loss of 34.8 absolute points. By contrast, DRDN’s accuracy drops from 92.81% to 65.40% over the same sequence (Table 2), a loss of only 29.5 points — less than half of DyTox’s degradation. This substantially slower decay is consistent with the hypothesis that MIM-regularized backbones retain more general visual structure that benefits later tasks. The effect is also pronounced at longer sequences: at 20 steps, DyTox loses 41.3 points from its peak while DRDN loses only 37.6 points, and DRDN’s BWT is 3.4 points better than DyTox’s (\(-13.8\) vs.\(-17.2\)).

Cross-task confusion (Challenge 1). We measure the cross-task confusion rate as the fraction of test errors where a sample from task \(i\) is predicted as a class from a different task \(j \neq i\). On CIFAR100-B0 (10 steps), DyTox’s cross-task confusion rate is 90.4% — nearly all classification errors are cross-task misclassifications (Table 6). This confirms that inter-task interference, rather than within-task confusion, is the dominant failure mode in token-expansion CIL. DRDN reduces this rate to 78.3%, a 13.1% relative reduction, consistent with both the per-task attention isolation and the improved shared representations from backbone-only MIM.

4pt

Anti-forgetting (BWT). BWT in Table 2 is consistently less negative for DRDN across all step configurations. The advantage grows with the number of steps: at 20 steps, DRDN’s BWT is \(-13.8\) vs.DyTox’s \(-17.2\) (3.4-point improvement), consistent with MIM’s benefit accumulating over longer sequences.

Robustness to initial task size. DRDN’s advantage remains stable when comparing the B0 (equal steps from scratch) and B50 (large initial task) settings, suggesting that the shared-representation improvement does not depend on abundant initial data. Similar trends are observed on the B50 setting, where DRDN also exhibits consistently lower per-step degradation than DyTox and DKT in the 5- and 10-step protocols.

4.5 Ablation Studies: Validating Decoupled Design↩︎

We perform comprehensive ablation studies on CIFAR100-B0 under the 10-step setting to validate the contribution and interaction of each component in DRDN. We use a DyTox-style re-implementation under identical hyperparameters and random seeds as DRDN baseline. Table 7 presents a full factorial analysis.

3pt

Gradient routing is essential (Section B). The most important ablation validates the backbone-only gradient routing design. When MIM gradients flow through both backbone and task tokens (“shared gradients”), the gain over baseline is only +1.45% (74.82%), compared to +2.56% with backbone-only routing (75.93%) — a 1.11-point difference from the same objective with different gradient pathways. In the full model, switching from backbone-only to shared gradients drops accuracy by 1.78 points (77.19% \(\rightarrow\) 75.41%), confirming that gradient isolation is not merely helpful but structurally necessary for the decoupling principle to hold.

Components provide complementary gains (Sections A, C). The reconstruction loss contributes the largest single gain (+2.56%), while multi-layer expansion with modified attention adds +3.52% independently. Their combination (full DRDN, +3.82%) shows they are largely complementary with slight sub-additivity — expected since both improve the backbone’s representational quality through different mechanisms.

Loss weight sensitivity (Section E). Performance peaks at \(\lambda=0.5\) (77.38%) but is robust across \(\lambda \in [0.25, 1]\) (all above 76.9%). We use \(\lambda=1\) across all experiments for simplicity. Heavy weighting (\(\lambda=4\)) degrades performance as reconstruction dominates, pulling the backbone away from discriminative features.

Knowledge distillation is critical (Section D). Removing \(\mathcal{L}_{\text{kd}}\) causes a 3.18-point drop, the largest single-component degradation. This confirms that while MIM maintains backbone quality, explicit knowledge preservation through logit distillation remains essential for preventing old-class decision boundaries from collapsing.

4.6 Design Comparison and Multi-Seed Robustness↩︎

Table 8 summarizes how DRDN differs from the most relevant prior methods in key design choices.

Multi-seed robustness. 3-seed evaluations: CIFAR100-B0 10-step: DRDN \(77.19 \pm 0.31\)% vs.DyTox \(73.66 \pm 0.28\)% (\(p < 0.001\)). CIFAR100-B50 10-step: DRDN \(71.02 \pm 0.27\)% vs.DyTox \(68.45 \pm 0.33\)% (\(p < 0.005\)). ImageNet100 10-step: DRDN \(78.73 \pm 0.41\)% vs.DKT \(78.20 \pm 0.38\)% (\(p < 0.05\)). All gaps exceed two standard deviations, confirming consistency.

5 Conclusion↩︎

The central lesson of this work is that expanding task-specific capacity is necessary but not sufficient for class-incremental learning: maintaining the quality of shared backbone representations throughout the incremental sequence is equally important. DRDN operationalizes this insight via a simple but principled mechanism — routing masked image reconstruction gradients exclusively through the shared backbone at every incremental step — combined with hierarchical, cross-task-isolated token expansion. The result is measurably better shared representations (29.5 vs. points lost over 10 tasks), reduced cross-task confusion (13.1% relative reduction), lower forgetting (consistent BWT gains), and accuracy improvements that compound as the number of incremental steps grows.

Broader Takeaway. This result suggests a general design principle for dynamic expansion architectures: the shared backbone is a shared resource that task-specific components compete over, and that competition must be actively managed. Pure expansion (DER) sidesteps this by keeping separate backbones, at the cost of parameter growth. DRDN shows that a regularization-based resolution — using MIM as a backbone-anchoring signal — can achieve most of the anti-forgetting benefit with a fixed-size backbone. The key is not just what auxiliary signal is used, but where its gradient is allowed to flow: restricting MIM gradients to the backbone alone, and never to task-specific modules, is what produces clean decoupling.

Limitations and Future Directions. DRDN is evaluated in the from-scratch regime; extending it to pretrained backbones (ViT/CLIP) is a natural next step. MIM objectives may interact differently with pretrained representations — prior work suggests that offline MAE pretraining [16] already encodes strong general features, so the online variant may need to be adapted (e.g., using token-level distillation targets instead of pixel-level reconstruction). The MIM decoder adds \(\sim\)​37% training time per task — acceptable for offline incremental deployment, but potentially restrictive in online streaming scenarios where per-sample latency matters. Extending DRDN to structured prediction tasks such as incremental object detection [46] and semantic segmentation [47] is a promising direction, where spatial correspondence in MIM may align well with localization objectives.

References↩︎

---

1. B. Huang, Y. Chen, Z. Wang, and Y. Du are with the Meituan Vision AI Department, Beijing 100190, China (e-mail: huangbingchen@meituan.com; chenyifu05@meituan.com; zhilingwang9681@gmail.com; duyuanchao@meituan.com).↩︎

2. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.↩︎