1 Introduction↩︎

Multimodal large language models (MLLMs) have become a central interface for visual question answering, document understanding, chart reasoning, embodied perception, and open-ended image–language interaction. In realistic deployments, however, these models rarely face a fixed task distribution: new domains, instructions, visual formats, and reasoning skills arrive sequentially, making continual adaptation essential. The key challenge is that improving an MLLM on new multimodal tasks can silently degrade previously acquired abilities, limiting its reliability in long-lived and safety-sensitive applications.

Recent studies have begun to formalize this problem as continual vision–language learning and continual multimodal instruction tuning. CLiMB highlights that vision–language systems suffer nontrivial forgetting across sequential multimodal tasks [1], while ZSCL and CTP show that continual adaptation can damage the zero-shot transfer and cross-modal topology of pretrained vision–language representations [2], [3]. More recent MLLM-oriented benchmarks, such as CoIN and Continual LLaVA, further demonstrate that instruction-tuned MLLMs are vulnerable to capability-level forgetting when exposed to evolving multimodal instruction streams [4], [5]. These works establish continual multimodal learning as an important problem and motivate a growing set of methods for mitigating forgetting.

Existing approaches have made substantial progress by preserving model parameters, output behavior, or task-specific adaptation modules. For example, Model Tailor mitigates forgetting by identifying sparse parameter modifications that preserve pretrained capabilities [6], while HiDe-LLaVA, BranchLoRA, and ProgLoRA organize task-specific or progressive LoRA modules to reduce interference during continual instruction tuning [7]–[9]. Other methods improve continual multimodal adaptation through prompt routing, expert allocation, or synthetic replay signals, as seen in ModalPrompt, CL-MoE, and GIFT [10]–[12]. SEFE further argues that forgetting in MLLMs should not be treated as a single phenomenon, distinguishing superficial response-format degradation from more essential knowledge loss [13]. Despite these advances, most methods still evaluate and constrain forgetting mainly through final answers, output distributions, or parameter changes.

This answer-centric view leaves a critical gap. In multimodal reasoning, two checkpoints may produce the same answer while relying on different evidence sources, such as visual regions, OCR tokens, or language priors. Such hidden shifts are especially problematic because they can make a model appear stable under standard accuracy metrics while its grounding behavior becomes less faithful and more shortcut-driven. Prior work on language bias in VQA has shown that correct predictions may arise from dataset priors rather than genuine visual evidence [14], and counterfactual analysis has been used to reveal causal effects of biased evidence in static VQA settings [15]. However, this perspective has not been fully integrated into replay-free continual learning for MLLMs, where the central question is not only whether old answers are retained, but whether the model continues to use the appropriate multimodal evidence to obtain them.

In this paper, we ask: Can continual MLLM adaptation preserve not only what the model answers, but also how it uses multimodal evidence? We address this question from the perspective of evidence-use stability. At a high level, our framework treats the previous model as a behavioral reference, estimates how both the previous and current models depend on available evidence channels, and constrains continual adaptation to preserve this reliance pattern while learning new tasks. This design complements existing output- or parameter-preservation strategies by targeting a hidden dimension of forgetting that standard answer-level evaluation can miss.

Our contributions are summarized as follows:

• We identify and formalize hidden evidence-use forgetting: an MLLM may retain the correct answer while drifting toward different or less grounded evidence sources during continual adaptation.

• We propose a replay-free reliance-preserving continual learning framework that constrains evidence-use drift without storing previous multimodal examples or increasing inference-time cost.

• Through continual multimodal instruction streams, we show that preserving reliance profiles improves both answer retention and evidence-grounded stability, providing a stronger notion of continual multimodal robustness than answer accuracy alone.

2 Related Work↩︎

2.0.0.1 Continual vision–language representation learning.

Early multimodal continual learning studies moved beyond unimodal classification by evaluating sequential vision–language tasks and cross-modal representation stability. CLiMB introduced a benchmark for continual vision-and-language tasks, while Mod-X, ZSCL, and CTP showed that sequential adaptation can distort intra-modal geometry, cross-modal alignment, and zero-shot transfer in CLIP-like models [1]–[3], [16]. More recent work scales this direction to practical multimodal streams and stronger VLM backbones: FoMo-in-Flux studies continual multimodal pretraining under realistic compute and data-mixture constraints [17]; C-CLIP evaluates continual image–text retrieval and zero-shot retention [18]; LADA introduces label-specific CLIP adapters [19]; and GIFT uses synthetic image–text data for replay-style preservation [12]. These methods substantially improve representation-level stability, but they mainly protect global embedding spaces, retrieval/classification performance, or pretrained zero-shot ability. In contrast, our work studies autoregressive MLLMs, where forgetting may occur not only in the final answer but also in the instance-specific allocation of reliance over visual regions, OCR tokens, chart elements, and language context.

2.0.0.2 Continual instruction tuning for MLLMs.

Continual instruction tuning has recently become the dominant setting for adapting MLLMs to evolving tasks. CoIN formulates sequential instruction tuning over diverse task categories and shows that MLLMs suffer severe forgetting of instruction following [4]. Continual LLaVA further introduces COAST, covering domain-, capability-, and dataset-incremental LVLM adaptation [5]. Recent methods then focus on stronger parameter organization and routing: Model Tailor identifies sparse parameter patches to preserve pretrained capabilities [6]; HiDe-LLaVA uses task-specific expansion and task-general fusion based on layer-wise similarity [7]; BranchLoRA and ProgLoRA improve LoRA allocation under MCIT [8], [9]; ModalPrompt uses dual-modality guided prompt fusion and selection [10]; CL-MoE introduces dual momentum experts for continual VQA [11]; and MLLM-CL proposes domain/ability continual settings with MLLM-based routing [20]. SEFE is particularly relevant because it separates superficial format forgetting from essential knowledge forgetting [13]. However, these direct competitors primarily preserve answer accuracy, response format, parameter specialization, or output distributions. Our preliminary observation suggests that such preservation can still hide evidence-use drift. RCL therefore adds a complementary constraint: the previous checkpoint’s counterfactual reliance profile serves as a teacher signal, preserving not only what the model answers but also how it uses multimodal evidence.

2.0.0.3 Grounding, shortcut bias, and counterfactual evidence use.

A related line of VQA research shows that correct answers may arise from language priors rather than grounded visual evidence. VQA-CP exposes this issue by changing answer priors between training and testing [14], and Counterfactual VQA uses causal counterfactual reasoning to estimate and reduce language-bias effects [15]. These works motivate channel-level evidence diagnostics, but they are mainly designed for static debiasing or evaluation rather than continual adaptation. Our work brings this perspective into replay-free MLLM continual learning by turning counterfactual evidence suppression into a training-time reliance-preservation objective, while adding no inference-time overhead.

3 Preliminary Study↩︎

Before introducing our method, we ask whether continual multimodal adaptation forgets only final answers, or also the evidence path used to obtain them. This distinction matters because two checkpoints may produce the same answer while relying on different channels, such as visual regions, OCR tokens, chart elements, or language priors. Standard answer-level evaluation may therefore miss a form of hidden forgetting: the answer is preserved, but the underlying multimodal evidence use has drifted.

3.0.0.1 Diagnostic setup.

We conduct a replay-free pilot study on an eight-stage multimodal stream and evaluate two simple baselines: sequential LoRA tuning and output-level answer distillation. No component of our proposed method is used. After training, we perform post-hoc channel ablations to estimate a diagnostic evidence-use vector \(\boldsymbol{q}(f;x,y)\) for each checkpoint \(f\), where \(q_m\) is proportional to the loss increase caused by suppressing evidence channel \(m\). Full protocol details are provided in Appendix 8.1.

Let \(f_j\) be the checkpoint after learning stage \(j\), and \(f_k\) a later checkpoint after stage \(k>j\). For an old evaluation set \(\mathcal{E}_j\), we measure answer forgetting, evidence-sensitivity drift, and answer-preserved dominant-evidence flips as \[\label{eq:prelim95metrics} \begin{align} A_{j\rightarrow k} &= {\rm Acc}(f_j;\mathcal{E}_j) - {\rm Acc}(f_k;\mathcal{E}_j),\\ G_{j\rightarrow k} &= \mathbb{E}_{(x,y)\sim\mathcal{E}_j} \left[ \left\| \boldsymbol{q}(f_j;x,y)-\boldsymbol{q}(f_k;x,y) \right\|_1 \right],\\ H_{j\rightarrow k} &= \mathbb{E} \left[ \mathbf{1}\!\left[ \arg\max_m q_m(f_j;x,y) \neq \arg\max_m q_m(f_k;x,y) \right] \,\middle|\, f_j(x)=f_k(x)=y \right]. \end{align}\tag{1}\] Here \(A_{j\rightarrow k}\) captures visible answer degradation, whereas \(G_{j\rightarrow k}\) and \(H_{j\rightarrow k}\) diagnose whether the model still uses the same evidence when the answer is retained.

3.0.0.2 Findings.

Figure 1 shows that answer preservation does not imply evidence preservation. Many transitions incur less than \(2\%\) clean accuracy drop while exhibiting large evidence-sensitivity drift. Moreover, output-level distillation reduces the average clean accuracy drop from \(2.72\%\) to \(1.57\%\), but still leaves \(66.3\%\) of diagnostic slices in the hidden-drift region, compared with \(14.4\%\) for sequential LoRA. The effect is most pronounced on OCR-, chart-, and document-centric stages, where the model often keeps the correct answer but shifts away from grounded visual/textual evidence toward easier priors.

These observations motivate the central principle of our method: continual multimodal learning should preserve not only what the previous model answers, but also how it uses evidence. We next formalize this principle as reliance-constrained continual learning.

4 Method↩︎

We propose Reliance-Constrained Continual Learning (RCL), a replay-free framework for mitigating hidden forgetting in continual multimodal learning. RCL is built on a simple principle: a continual MLLM should preserve not only its answers, but also the evidence channels that support those answers. At each learning stage, RCL freezes the previous model as a behavioral reference, probes both the reference and current models through counterfactual channel interventions, estimates their modality-reliance profiles, and updates the current model by jointly optimizing task learning, prediction preservation, and reliance preservation. The framework updates only parameter-efficient adaptation parameters and introduces no additional computation at inference time.

4.1 Continual Multimodal Learning Setup↩︎

We consider a sequence of multimodal learning stages \(\{\mathcal{D}_1,\ldots,\mathcal{D}_K\}\), where each stage \(\mathcal{D}_k\) contains input-output pairs \((x,y)\). Each input \(x\) consists of a set of evidence channels \(\mathcal{M}_x\), such as visual tokens, question tokens, OCR tokens, chart elements, and other task-specific input sources. We write \(x=\{x^m:m\in\mathcal{M}_x\}\), where \(x^m\) denotes the component of \(x\) associated with channel \(m\).

Let \(f_\theta\) denote an MLLM with autoregressive answer distribution \(p_\theta(y\mid x)=\prod_{\ell=1}^{L}p_\theta(y_\ell\mid y_{<\ell},x)\). We use parameter-efficient adaptation and decompose the model parameters as \(\theta=\theta_0\cup\phi\), where \(\theta_0\) is the frozen backbone and \(\phi\) denotes trainable adapter parameters. At stage \(k\), the previous model is frozen as \(f_{\theta^-}\) with \(\theta^-=\theta_{k-1}\), and the current model \(f_\theta\) is optimized on \(\mathcal{D}_k\).

The standard task loss is \[\label{eq:task95loss} \mathcal{L}_{\rm task}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}_k} \left[ -\sum_{\ell=1}^{L} \log p_\theta(y_\ell\mid y_{<\ell},x) \right].\tag{2}\] This objective encourages correct answers on the current stage, but it does not constrain which evidence channels the model uses to obtain those answers.

4.2 Counterfactual Modality Reliance↩︎

RCL estimates modality reliance through counterfactual evidence suppression. For each channel \(m\in\mathcal{M}_x\), we define an intervention set \(\mathcal{I}_m\). Each intervention \(\tau_m\in\mathcal{I}_m\) suppresses access to channel \(m\) while keeping the remaining channels unchanged. For example, an image channel is suppressed by masking visual regions, an OCR channel by removing recognized text tokens, and a chart channel by masking chart elements. The complete intervention construction is given in Appendix 7.1.

For a model \(f_\theta\), we define the counterfactual sensitivity of channel \(m\) as \[\label{eq:counterfactual95score} \begin{align} \Delta_{\theta,m}(x,y) = \frac{1}{|\mathcal{I}_m|} \sum_{\tau_m\in\mathcal{I}_m} \bigg[ & \frac{1}{L} \sum_{\ell=1}^{L} D_{\rm KL} \left( p_\theta(\cdot\mid y_{<\ell},x) \,\middle\|\, p_\theta(\cdot\mid y_{<\ell},\tau_m(x)) \right) \\ &+ \beta \left( \ell_\theta(y\mid \tau_m(x)) - \ell_\theta(y\mid x) \right)_+ \bigg], \end{align}\tag{3}\] where \(\ell_\theta(y\mid x)=-\sum_{\ell=1}^{L}\log p_\theta(y_\ell\mid y_{<\ell},x)\), \((z)_+=\max(z,0)\), and \(\beta\geq0\) balances output-distribution shift and answer-loss increase. A larger \(\Delta_{\theta,m}(x,y)\) indicates that suppressing channel \(m\) causes a larger behavioral change, hence the model relies more strongly on that channel for the given sample.

We normalize channel sensitivities into a reliance vector: \[\label{eq:reliance95vector} r_{\theta,m}(x,y) = \frac{ \exp\left(\Delta_{\theta,m}(x,y)/\tau_r\right) }{ \sum_{n\in\mathcal{M}_x} \exp\left(\Delta_{\theta,n}(x,y)/\tau_r\right) }, \qquad \boldsymbol{r}_\theta(x,y) = \left[ r_{\theta,m}(x,y) \right]_{m\in\mathcal{M}_x}.\tag{4}\] Here \(\tau_r>0\) is a temperature parameter. The vector \(\boldsymbol{r}_\theta(x,y)\) lies in the simplex \(\Delta^{|\mathcal{M}_x|-1}\) and is always computed over the evidence channels available in the current sample. Teacher and student reliance vectors are therefore compared on the same support \(\mathcal{M}_x\).

4.3 Hidden Forgetting as Reliance Drift↩︎

Accuracy-based evaluation detects whether the final answer changes, but it does not detect whether the evidence path changes. We formalize this discrepancy through modality reliance drift. Given the frozen reference model \(f_{\theta^-}\) and the current model \(f_\theta\), the sample-level reliance drift is \[\label{eq:sample95mrd} {\rm MRD}(x,y;\theta^-,\theta) = D_{\rm JS} \left( \boldsymbol{r}_{\theta^-}(x,y) \,\middle\|\, \boldsymbol{r}_{\theta}(x,y) \right),\tag{5}\] where \(D_{\rm JS}\) denotes Jensen–Shannon divergence.

For a previous-stage evaluation set \(\mathcal{E}_j\), hidden forgetting occurs when answer accuracy is preserved while reliance drift is large: \[\label{eq:hidden95forgetting} {\rm HF}_{j\rightarrow k} = \mathbb{I} \left[ {\rm Acc}(f_{\theta_j};\mathcal{E}_j) - {\rm Acc}(f_{\theta_k};\mathcal{E}_j) \leq \delta_{\rm acc} \right] \cdot \mathbb{I} \left[ {\rm MRD}_{j\rightarrow k} \geq \delta_{\rm rel} \right].\tag{6}\] Here \({\rm MRD}_{j\rightarrow k}\) is the average of Eq. 5 over \(\mathcal{E}_j\), and \(\delta_{\rm acc},\delta_{\rm rel}\) are fixed thresholds. This definition separates visible forgetting, measured by answer degradation, from hidden forgetting, measured by evidence-use drift. Additional evaluation metrics are defined in Appendix 7.7.

4.4 Reliance-Constrained Continual Update↩︎

RCL constrains the current model to preserve the reference model’s reliance profile on the current-stage data. For each sample \((x,y)\), the frozen model \(f_{\theta^-}\) produces a reference reliance vector \(\boldsymbol{r}_{\theta^-}(x,y)\), while the trainable model produces \(\boldsymbol{r}_\theta(x,y)\). The reliance preservation loss is \[\label{eq:reliance95loss} \mathcal{L}_{\rm rel}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}_k} \left[ w_{\theta^-}(x,y) D_{\rm JS} \left( {\rm sg}\!\left[\boldsymbol{r}_{\theta^-}(x,y)\right] \,\middle\|\, \boldsymbol{r}_{\theta}(x,y) \right) \right],\tag{7}\] where \({\rm sg}[\cdot]\) denotes stop-gradient. The scalar \(w_{\theta^-}(x,y)\in[0,1]\) is a confidence weight computed from the reference model’s predictive entropy and intervention sensitivity; its exact form is provided in Appendix 7.2. This gate prevents unreliable reference predictions from dominating the update.

We also preserve the output behavior of the previous model through token-level prediction distillation: \[\label{eq:prediction95distillation} \mathcal{L}_{\rm pred}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}_k} \left[ \frac{\tau_{\rm kd}^2}{L} \sum_{\ell=1}^{L} D_{\rm KL} \left( p_{\theta^-}^{\tau_{\rm kd}}(\cdot\mid y_{<\ell},x) \,\middle\|\, p_{\theta}^{\tau_{\rm kd}}(\cdot\mid y_{<\ell},x) \right) \right],\tag{8}\] where \(p_\theta^{\tau_{\rm kd}}\) denotes the softmax distribution over logits divided by the temperature \(\tau_{\rm kd}\).

The full objective at stage \(k\) is \[\label{eq:full95objective} \min_{\phi} \quad \mathcal{L}_k(\phi) = \mathcal{L}_{\rm task} + \lambda_{\rm pred}\mathcal{L}_{\rm pred} + \lambda_{\rm rel}\mathcal{L}_{\rm rel} + \lambda_{\rm reg}\|\phi-\phi^-\|_2^2 ,\tag{9}\] where \(\phi^-\) denotes the adapter parameters from the previous stage. The prediction term preserves what the model answers, while the reliance term preserves how the model uses evidence. RCL does not impose a fixed preference for visual, textual, or OCR evidence. Instead, it preserves the instance-specific reliance profile induced by the previous model. Therefore, when the reference model genuinely relies on visual evidence, RCL discourages drift toward language shortcuts; when the reference model relies primarily on textual evidence, RCL preserves that allocation rather than forcing visual dependence.

4.5 Training and Inference↩︎

At each stage, RCL freezes \(f_{\theta^-}\), samples a minibatch from \(\mathcal{D}_k\), constructs counterfactual interventions for the available evidence channels, computes \(\boldsymbol{r}_{\theta^-}(x,y)\) and \(\boldsymbol{r}_\theta(x,y)\) on the same intervened inputs, and updates only the PEFT parameters \(\phi\) using Eq. 9 . RCL stores no raw multimodal examples from previous stages. It keeps the previous checkpoint only during the current stage, which is sufficient to provide prediction and reliance targets. After training stage \(k\), the updated model becomes the reference for stage \(k+1\).

During inference, RCL uses the updated MLLM \(f_{\theta_k}\) directly. Counterfactual interventions, frozen-reference forward passes, and reliance estimation are used only during training and evaluation, so the deployed model has the same inference cost as the underlying PEFT-adapted MLLM.

5 Experiments↩︎

5.1 Experimental Setup↩︎

3.0pt

Benchmarks and metrics. We evaluate RCL on CoIN, COAST, MCITlib, and an evidence-sensitive stream covering VQA, OCR/document, and chart/table reasoning [4], [5], [21]. We report official task scores, standard CL metrics, and reliance-aware diagnostics; details are in Appendix 8.2.

Comparison protocol. Comparison protocol. We use a replay-free, task-id-free protocol: methods store no previous multimodal examples, use no oracle task identity at inference, and follow the same task order and data budget. We compare against standard CL/PEFT baselines, recent multimodal continual instruction-tuning methods, and separately report replay- or memory-assisted references because they use stored or generated past-task signals. Detailed method grouping, baseline list, and fairness rules are provided in Appendix 8.3.

Backbones and implementation. We use LLaVA-1.5-7B as the default backbone and additionally evaluate LLaVA-1.5-13B and InternVL-Chat-7B to test scale and architecture robustness [22], [23]. Unless otherwise specified, the vision encoder and language backbone are frozen and only PEFT parameters are updated. All methods are trained with matched data, task order, trainable-parameter budget, and optimization budget whenever possible. For RCL, counterfactual interventions are used only during training and diagnostic evaluation, so the final model has the same inference cost as the underlying PEFT-adapted MLLM. Implementation details, hyperparameters, compute resources, and reproducibility settings are provided in Appendix 8.4.

5.2 Main Results↩︎

4.5pt

Table 1 shows that RCL delivers the strongest plasticity–retention trade-off across three backbones, improving final performance while reducing forgetting and keeping BWT closer to zero. Despite being replay-free, it outperforms replay- or task-recall-based references, highlighting evidence-use preservation as an effective anti-forgetting signal.

Table 2 shows that strong baselines can preserve answers while still suffering large MRD, DEF, and HFR, revealing substantial evidence-path drift. RCL consistently reduces reliance drift and dominant-channel flips while improving GCR, confirming that continual MLLM evaluation should assess both answers and their supporting evidence.

5.3 Ablation Study and Analysis↩︎

5.3.0.1 Single-factor ablation.

We ablate each major component of RCL on LLaVA-1.5-7B using the same task order, PEFT configuration, training budget, and evaluation protocol as the full model. As shown in Table 3, removing the reliance loss \(\mathcal{L}_{\rm rel}\) leads to the largest degradation in reliance-aware diagnostics, especially MRD, DEF, and HFR. This indicates that preserving output behavior or answer accuracy alone is insufficient to maintain stable evidence use during continual adaptation. Prediction distillation mainly improves conventional retention metrics, while the confidence gate helps filter unreliable teacher reliance targets and evidence-targeted interventions provide more faithful channel-level sensitivity signals. The weaker performance of random interventions, single-score reliance variants, and visual-only reliance further shows that the gains of RCL do not come from generic perturbation regularization. Instead, they arise from preserving instance-specific, multi-channel evidence reliance across continual learning stages.

3.4pt

5.3.0.2 Can Accuracy Preservation Hide Evidence-Use Forgetting?

Answer-level retention may conceal whether an old task is still supported by the same multimodal evidence. We analyze all old-to-later transitions in the evidence-sensitive stream and compare clean answer drop, modality reliance drift (MRD), and dominant evidence flip rate (DEF) on answer-preserved examples. Figure 3 shows that Answer-KD moves many points leftward by reducing the average answer drop from \(3.41\%\) to \(1.82\%\), but still leaves \(64.7\%\) of low-drop slices in the hidden-forgetting region with an average DEF of \(35.9\%\). By contrast, RCL reduces MRD from \(0.238\) to \(0.108\), lowers DEF from \(35.9\%\) to \(14.6\%\), and decreases hidden forgetting from \(64.7\%\) to \(10.8\%\). The largest reductions occur on OCR-, chart-, and document-centric slices, indicating that reliance preservation is most valuable when the correct answer depends on structured evidence rather than language priors.

5.3.0.3 Is Counterfactual Reliance Estimation Worth the Training Cost?

Because counterfactual reliance estimation introduces extra training-time forward passes, we test whether its hidden-forgetting gains justify the added cost. We compare sequential tuning, output distillation, random intervention regularization, sparse RCL, and RCL variants with different intervention budgets using relative GPU cost and HFR reduction. Figure 4 shows that output distillation adds little cost but reduces HFR by only \(13.9\%\), confirming that output preservation alone is insufficient. Random interventions incur similar cost to RCL but achieve only \(59.7\%\) HFR reduction and retain higher MRD, indicating that targeted evidence suppression is necessary. The default setting \(q_{\rm train}=1\) reduces HFR by \(79.0\%\) at \(1.56\times\) training cost, while increasing to \(q_{\rm train}=2\) and \(q_{\rm train}=4\) yields only marginal gains at substantially higher cost. Thus, one targeted intervention per available evidence channel already provides a strong reliance signal with no inference-time overhead.

6 Conclusion↩︎

This paper identifies hidden evidence-use forgetting in continual multimodal learning, where correct answers may be retained while grounding behavior silently drifts. We propose RCL, a replay-free framework that preserves evidence-reliance profiles through counterfactual channel interventions without adding inference-time cost. Experiments show that answer-level retention alone can mask substantial reliance drift, while RCL improves both performance retention and grounding stability. These results suggest that continual MLLMs should be evaluated by not only what they answer, but also how they use multimodal evidence. Future work may extend reliance-aware adaptation to longer streams and richer evidence structures.

7 Additional Method Details↩︎

7.1 Counterfactual Intervention Construction↩︎

RCL estimates reliance through evidence-suppressing interventions. Each intervention targets one evidence channel while keeping the remaining channels unchanged. We instantiate the intervention set \(\mathcal{I}_m\) according to the channel type.

7.1.0.1 Visual channel.

For image-based inputs, \(\mathcal{I}_{\rm img}\) contains region masking and visual-token removal. Region masking replaces selected image patches with a neutral value. Visual-token removal removes the corresponding visual embeddings before multimodal fusion. For datasets with object annotations, regions are sampled from annotated objects. For datasets without object annotations, regions are sampled from uniformly partitioned image patches.

7.1.0.2 Question-text channel.

For question or instruction text, \(\mathcal{I}_{\rm txt}\) contains content-word masking. Stop words and formatting tokens are preserved. This intervention suppresses semantic cues from the question while keeping the prompt format stable.

7.1.0.3 OCR channel.

For OCR-based inputs, \(\mathcal{I}_{\rm ocr}\) contains OCR-token removal and OCR-token masking. OCR-token removal deletes recognized text tokens from the input sequence. OCR-token masking replaces recognized text tokens with a neutral placeholder token.

7.1.0.4 Chart and table channels.

For chart and table reasoning, \(\mathcal{I}_{\rm chart}\) contains axis-label masking, legend masking, cell-value masking, and plotted-element masking. These interventions suppress numerical and structural evidence while preserving the global input format.

7.1.0.5 Video channel.

For video inputs, \(\mathcal{I}_{\rm vid}\) contains frame masking and sparse frame removal. Frame masking suppresses visual content in selected frames. Sparse frame removal drops selected frame embeddings before multimodal fusion.

For each minibatch, RCL samples a fixed number of interventions from \(\mathcal{I}_m\) for every available channel \(m\in\mathcal{M}_x\). The same intervened inputs are shared by the frozen reference model and the current model, ensuring that the reliance loss compares teacher and student behavior under identical counterfactual conditions.

7.2 Confidence Gate for Reliance Targets↩︎

The reliance loss in Eq. 7 uses a confidence weight \(w_{\theta^-}(x,y)\) to reduce the influence of unreliable reference targets. The gate combines predictive confidence and intervention sensitivity.

The reference predictive entropy is \[\label{eq:app95entropy} H_{\theta^-}(x,y) = \frac{1}{L} \sum_{\ell=1}^{L} H\!\left( p_{\theta^-}(\cdot\mid y_{<\ell},x) \right),\tag{10}\] where \(H(\cdot)\) denotes Shannon entropy. The total reference intervention sensitivity is \[\label{eq:app95total95sensitivity} A_{\theta^-}(x,y) = \sum_{m\in\mathcal{M}_x} \Delta_{\theta^-,m}(x,y).\tag{11}\] The confidence weight is defined as \[\label{eq:app95confidence95weight} w_{\theta^-}(x,y) = \sigma \left( \frac{\tau_H-H_{\theta^-}(x,y)}{\gamma_H} \right) \cdot \sigma \left( \frac{A_{\theta^-}(x,y)-\tau_A}{\gamma_A} \right),\tag{12}\] where \(\sigma(\cdot)\) is the sigmoid function, \(\tau_H\) and \(\tau_A\) are fixed thresholds, and \(\gamma_H,\gamma_A>0\) are temperature parameters. The first factor assigns larger weight to low-entropy reference predictions. The second factor assigns larger weight to samples where the reference model exhibits measurable dependence on at least one evidence channel.

7.3 Diagnostic Attribution Analysis↩︎

RCL does not use gradient-based attribution in the training objective. We use attribution only as an evaluation-time diagnostic to verify whether reliance preservation is consistent with local sensitivity in the model representation.

Let \(h_\theta^m(x)\) denote the hidden representation associated with channel \(m\). The attribution score for channel \(m\) is \[\label{eq:app95grad95attr} G_{\theta,m}(x,y) = \frac{ \left\| h_\theta^m(x) \odot \nabla_{h_\theta^m(x)} \ell_\theta(y\mid x) \right\|_1 }{ {\rm dim}\!\left(h_\theta^m(x)\right) }.\tag{13}\] Since \(G_{\theta,m}\) is never differentiated through during training, RCL avoids second-order optimization. We report attribution agreement by comparing the normalized attribution vector \(\boldsymbol{g}_\theta(x,y)\) with the counterfactual reliance vector \(\boldsymbol{r}_\theta(x,y)\) using rank correlation and Jensen–Shannon divergence.

7.4 Task-Preserving Perturbation Analysis↩︎

In addition to evidence-suppressing interventions, we use task-preserving perturbations for robustness analysis. These perturbations are not part of the main reliance loss. They evaluate whether a model is overly sensitive to changes that should not alter the answer.

For each channel \(m\), let \(\mathcal{P}_m\) be a set of task-preserving perturbations. Examples include mild image noise, small resolution changes, harmless crop-and-resize transformations, question paraphrases that preserve meaning, and OCR formatting changes that preserve recognized text. The invariance sensitivity is \[\label{eq:app95invariance95sensitivity} B_{\theta,m}(x,y) = \frac{1}{|\mathcal{P}_m|} \sum_{\pi_m\in\mathcal{P}_m} \frac{1}{L} \sum_{\ell=1}^{L} D_{\rm KL} \left( p_\theta(\cdot\mid y_{<\ell},x) \,\middle\|\, p_\theta(\cdot\mid y_{<\ell},\pi_m(x)) \right).\tag{14}\] A grounded and stable model should have high sensitivity to evidence-suppressing interventions and low sensitivity to task-preserving perturbations.

7.5 Efficient Training↩︎

The computational overhead of RCL comes from counterfactual forward passes through the frozen reference model and the current model. We control this overhead with three design choices.

First, for each minibatch, RCL samples a small fixed number of interventions per channel. Second, the same counterfactual inputs are used for both teacher and student models. Third, reliance estimation is applied only to the channels present in the current sample. Therefore, if a sample contains \(|\mathcal{M}_x|\) channels and \(q\) interventions are sampled per channel, the reliance module uses \(q|\mathcal{M}_x|\) counterfactual views for that sample.

The teacher model is evaluated under stop-gradient. The student model receives gradients only through the reliance vector \(\boldsymbol{r}_\theta(x,y)\) computed from counterfactual output shifts. No gradient-based attribution term is used in the training objective.

7.6 Training Procedure↩︎

At continual stage \(k\), RCL performs the following procedure.

1. Freeze the previous model \(f_{\theta^-}=f_{\theta_{k-1}}\).

2. Initialize the current trainable model with \(\theta=\theta^-\) and update only PEFT parameters \(\phi\).

3. Sample a minibatch \(\mathcal{B}_k\) from the current-stage dataset \(\mathcal{D}_k\).

4. For each \((x,y)\in\mathcal{B}_k\) and each channel \(m\in\mathcal{M}_x\), construct counterfactual inputs \(\tau_m(x)\).

5. Compute reference reliance \(\boldsymbol{r}_{\theta^-}(x,y)\) and current reliance \(\boldsymbol{r}_\theta(x,y)\) using Eq. 3 and Eq. 4 .

6. Compute \(\mathcal{L}_{\rm task}\), \(\mathcal{L}_{\rm pred}\), and \(\mathcal{L}_{\rm rel}\) using Eq. 2 , Eq. 8 , and Eq. 7 .

7. Update \(\phi\) by minimizing Eq. 9 .

8. After stage \(k\) finishes, set \(\theta_k=\theta\) and use \(f_{\theta_k}\) as the frozen reference for stage \(k+1\).

7.7 Evaluation Metrics↩︎

We evaluate continual multimodal learning from two perspectives: answer retention and reliance preservation.

7.7.0.1 Final average accuracy.

After the final stage \(K\), the final average accuracy is \[\label{eq:app95average95accuracy} {\rm AA}_K = \frac{1}{K} \sum_{j=1}^{K} {\rm Acc}(f_{\theta_K};\mathcal{E}_j),\tag{15}\] where \(\mathcal{E}_j\) is the evaluation set for stage \(j\).

7.7.0.2 Average forgetting.

The standard average forgetting score is \[\label{eq:app95average95forgetting} {\rm AF}_K = \frac{1}{K-1} \sum_{j=1}^{K-1} \left[ \max_{s\in\{j,\ldots,K\}} {\rm Acc}(f_{\theta_s};\mathcal{E}_j) - {\rm Acc}(f_{\theta_K};\mathcal{E}_j) \right].\tag{16}\]

7.7.0.3 Reliance drift.

The final reliance drift is \[\label{eq:app95final95mrd} {\rm MRD}_K = \frac{1}{K-1} \sum_{j=1}^{K-1} \frac{1}{|\mathcal{E}_j|} \sum_{(x,y)\in\mathcal{E}_j} D_{\rm JS} \left( \boldsymbol{r}_{\theta_j}(x,y) \,\middle\|\, \boldsymbol{r}_{\theta_K}(x,y) \right).\tag{17}\]

7.7.0.4 Channel-specific drift.

For channel \(m\), the channel-specific drift from stage \(j\) to the final model is \[\label{eq:app95channel95drift} \Delta_{j\rightarrow K}^{m} = \frac{1}{|\mathcal{E}_j^m|} \sum_{(x,y)\in\mathcal{E}_j^m} \left[ r_{\theta_K,m}(x,y) - r_{\theta_j,m}(x,y) \right],\tag{18}\] where \(\mathcal{E}_j^m=\{(x,y)\in\mathcal{E}_j:m\in\mathcal{M}_x\}\). A negative value for the visual channel indicates a drift away from visual evidence.

7.7.0.5 Hidden forgetting rate.

The hidden forgetting rate is \[\label{eq:app95hfr} {\rm HFR}_K = \frac{1}{K-1} \sum_{j=1}^{K-1} {\rm HF}_{j\rightarrow K},\tag{19}\] where \({\rm HF}_{j\rightarrow K}\) is defined in Eq. 6 . A reliable continual MLLM should achieve high final average accuracy, low average forgetting, low reliance drift, and low hidden forgetting rate.

8 Additional Experimental Details↩︎

8.1 Details of the Preliminary Study↩︎

This subsection provides the experimental details omitted from Section 3. The goal of the preliminary study is diagnostic: it exposes a failure mode of standard replay-free multimodal continual adaptation, rather than evaluating the proposed method.

8.1.0.1 Continual stream.

We construct an eight-stage multimodal learning stream: ScienceQA, GQA, VQAv2, OK-VQA, TextVQA, OCR-VQA, ChartQA, and DocVQA. These stages cover visual reasoning, compositional VQA, knowledge-intensive VQA, OCR-based question answering, chart understanding, and document understanding. After learning each stage \(j\), we save the resulting checkpoint \(f_j\). After learning a later stage \(k>j\), we evaluate the current checkpoint \(f_k\) on the previous-stage evaluation set \(\mathcal{E}_j\). No previous training examples are replayed during adaptation.

8.1.0.2 Baselines.

We use two simple baselines to avoid introducing any component of our method into the exploratory study. The first baseline, Seq-LoRA, updates only parameter-efficient adaptation parameters on the current-stage task loss: \[\label{eq:app95seq95lora} \mathcal{L}_{\rm Seq} = \mathbb{E}_{(x,y)\sim\mathcal{D}_k} \left[ -\log p_{\theta}(y\mid x) \right].\tag{20}\] The second baseline, Answer-KD, additionally preserves the previous checkpoint’s output distribution on the current-stage data: \[\label{eq:app95answer95kd} \mathcal{L}_{\rm KD} = \mathcal{L}_{\rm Seq} + \lambda_{\rm kd} \mathbb{E}_{(x,y)\sim\mathcal{D}_k} \left[ D_{\rm KL} \left( p_{\theta^-}^{\tau}(\cdot\mid x) \,\middle\|\, p_{\theta}^{\tau}(\cdot\mid x) \right) \right],\tag{21}\] where \(\theta^-\) denotes the previous checkpoint and \(\tau\) is the distillation temperature. This baseline preserves output behavior but imposes no constraint on which evidence channel supports the prediction.

8.1.0.3 Post-hoc channel ablations.

For each example \((x,y)\), we define a set of available evidence channels \(\mathcal{C}_x\). Depending on the task, this set may include visual regions, OCR tokens, document text, chart elements, and question-side language priors. For each channel \(m\in\mathcal{C}_x\), we construct an ablated input \(\tau_m(x)\) that suppresses channel \(m\) while keeping other channels unchanged. For example, visual ablation masks image tokens or visual regions; OCR/text ablation removes recognized text or document tokens; chart ablation masks chart elements; and question-prior ablation removes or neutralizes evidence-bearing lexical cues in the question. These ablations are used only after training for diagnosis. They are not used as training objectives, constraints, supervision signals, or data augmentation.

For a checkpoint \(f\), we define the diagnostic channel sensitivity as \[\label{eq:app95prelim95sensitivity} s_m(f;x,y) = \left[ \ell_f(y\mid \tau_m(x)) - \ell_f(y\mid x) \right]_+,\tag{22}\] where \(\ell_f(y\mid x)\) is the negative log-likelihood of the ground-truth answer and \([z]_+=\max(z,0)\). The normalized evidence-use vector is \[\label{eq:app95prelim95q} q_m(f;x,y) = \frac{ s_m(f;x,y) }{ \sum_{n\in\mathcal{C}_x}s_n(f;x,y)+\epsilon }, \qquad \boldsymbol{q}(f;x,y) = \left[ q_m(f;x,y) \right]_{m\in\mathcal{C}_x}.\tag{23}\] The small constant \(\epsilon\) prevents numerical instability when all channel sensitivities are close to zero. The vector \(\boldsymbol{q}(f;x,y)\) is only a post-hoc diagnostic proxy for evidence use.

8.1.0.4 Metrics.

For each old-to-new transition \((j,k)\) with \(k>j\), we report three quantities. The clean accuracy drop \(A_{j\rightarrow k}\) measures conventional answer forgetting. The evidence-sensitivity drift \(G_{j\rightarrow k}\) measures how much the normalized evidence-use vector changes between checkpoints. The dominant-evidence flip rate \(H_{j\rightarrow k}\) is computed only on examples for which both \(f_j\) and \(f_k\) predict the correct answer. Thus, \(H_{j\rightarrow k}\) isolates cases where the answer is preserved but the dominant evidence channel changes. In Figure 1, we mark a transition as falling into the hidden-drift region when \[\label{eq:app95hidden95drift95region} A_{j\rightarrow k} \leq 2\%, \qquad G_{j\rightarrow k} \geq 0.20.\tag{24}\] These thresholds are used only to visualize the qualitative phenomenon and are not used by our method.

8.1.0.5 Evaluation slices and aggregation.

To avoid hiding heterogeneous behavior under task-level averages, each previous-stage evaluation set is partitioned into several diagnostic slices, including easy, medium, hard, rare, compositional, and format-sensitive subsets. Each point in Figure 1 (a) corresponds to one tuple of old task, later task, random seed, evaluation slice, and training baseline. The x-axis is \(A_{j\rightarrow k}\), the y-axis is \(G_{j\rightarrow k}\), the marker size is \(H_{j\rightarrow k}\), the marker shape denotes the baseline, and the color denotes the old-task family. Figure 1 (b) focuses on Answer-KD. Each cell corresponds to evaluating an old task after a later stage is learned; color indicates the dominant-evidence flip rate, and bubble size indicates the clean accuracy drop.

8.1.0.6 Why this does not leak the proposed method.

The preliminary study does not optimize reliance preservation, intervention consistency, evidence-channel matching, or any loss introduced in the main method. The baselines are trained using only task supervision and, for Answer-KD, output-level distillation. Channel ablations are applied strictly after training to diagnose the learned checkpoints. Therefore, the study serves only as motivation: it reveals that output preservation can mask evidence-use drift, which motivates the reliance-preservation objective introduced in Section 4.

8.1.0.7 Relation-level alignment utility protocol.

For the observability volcano plot in Figure [fig:obs_volcano], each point corresponds to one modality–semantic relation \(r=(m,n,s)\). We compute the final observability score \(O_r=O_{m,n,s}\) after training and evaluate the relation-conditioned validation performance of three strategies: Task-only FedAvg, Uniform Alignment, and FedSO. The relation-level gains are defined as \[G^{\mathrm{UA}}_{r} = \operatorname{Score}^{\mathrm{UA}}_{r} - \operatorname{Score}^{\mathrm{Task}}_{r}, \qquad G^{\mathrm{FedSO}}_{r} = \operatorname{Score}^{\mathrm{FedSO}}_{r} - \operatorname{Score}^{\mathrm{Task}}_{r}. \label{eq:obs95volcano95gain}\tag{25}\] A positive value indicates beneficial transfer, while a negative value indicates negative transfer. All relation-level quantities are averaged over three random seeds before plotting. The horizontal axis shows \(O_r\), the vertical axis shows the corresponding gain, point size indicates relation support, point color encodes observability, and red markers highlight negative-transfer relations. For summary statistics, we define low-observability relations as \(O_r<0.35\) and high-observability relations as \(O_r>0.65\).

8.1.0.8 Low-observability negative-transfer protocol.

For Figure [fig:low_obs_violin], we group all modality–semantic relations according to their final observability scores: \[\text{Low-O}: O_r < 0.35, \qquad \text{Mid-O}: 0.35 \le O_r < 0.65, \qquad \text{High-O}: O_r \ge 0.65. \label{eq:obs95group95def}\tag{26}\] For each group and each method \(\mathcal{A}\), we report the relation-conditioned gain over Task-only FedAvg: \[G_r^{(\mathcal{A})} = \operatorname{Score}^{(\mathcal{A})}_r - \operatorname{Score}^{\mathrm{Task}}_r, \qquad \mathcal{A} \in \{\mathrm{Task},\mathrm{UA},\mathrm{FedSO}\}. \label{eq:violin95relation95gain}\tag{27}\] The Task-only baseline is therefore centered at zero and serves as the reference distribution. Each violin aggregates all modality–semantic relations from the corresponding observability group across benchmark datasets and seeds. We overlay the group mean and interquartile range on each violin. The negative-transfer rate is computed as \[\operatorname{NTR}^{(\mathcal{A})}_{\mathcal{G}} = \frac{1}{|\mathcal{G}|} \sum_{r\in\mathcal{G}} \mathbf{1}\!\left[ G_r^{(\mathcal{A})}<0 \right], \label{eq:negative95transfer95rate}\tag{28}\] where \(\mathcal{G}\) denotes one of the three observability groups.

8.1.0.9 Relation-geometry visualization protocol.

For Figure [fig:relation_geometry_umap], we visualize the geometry of learned modality–semantic prototypes. For each modality \(m\) and semantic unit \(s\), we extract the final global prototype \[\bar{\mu}_{m,s} = \frac{ \sum_{i} e_{i,m,s} n_{i,m,s}\rho_{i,m,s}\tilde{\mu}_{i,m,s} }{ \sum_{i} e_{i,m,s} n_{i,m,s}\rho_{i,m,s} +\varepsilon }, \label{eq:appendix95global95proto}\tag{29}\] where \(e_{i,m,s}\), \(n_{i,m,s}\), \(\rho_{i,m,s}\), and \(\tilde{\mu}_{i,m,s}\) follow the semantic sketch definitions in the main method section. We collect all prototypes \(\{\bar{\mu}_{m,s}\}_{m,s}\) and project them into two dimensions using UMAP with cosine distance. The same projection procedure and visualization set are used for Uniform Alignment and FedSO. Each point in the plot corresponds to one prototype \(\bar{\mu}_{m,s}\), point color indicates the semantic unit, and marker shape indicates the modality. We draw an edge between \(\bar{\mu}_{m,s}\) and \(\bar{\mu}_{n,s}\) when the corresponding relation is highly observable, i.e., \[O_{m,n,s}>\tau_h. \label{eq:appendix95high95obs95edge}\tag{30}\] To quantify the geometry, we report the average cross-modal prototype distance over high-observability relations: \[D_{\mathrm{high}} = \frac{ 1 }{ |\mathcal{R}_{\mathrm{high}}| } \sum_{(m,n,s)\in\mathcal{R}_{\mathrm{high}}} \left\| \bar{\mu}_{m,s} - \bar{\mu}_{n,s} \right\|_2, \qquad \mathcal{R}_{\mathrm{high}} = \{(m,n,s): O_{m,n,s}>\tau_h\}. \label{eq:high95obs95proto95distance}\tag{31}\] We also compute an inter-semantic margin: \[\operatorname{Margin} = \frac{1}{S} \sum_{s=1}^{S} \min_{s'\ne s} \left\| \bar{\mu}_{s} - \bar{\mu}_{s'} \right\|_2, \qquad \bar{\mu}_{s} = \frac{1}{|\mathcal{M}_s|} \sum_{m\in\mathcal{M}_s} \bar{\mu}_{m,s}, \label{eq:inter95semantic95margin}\tag{32}\] where \(\mathcal{M}_s\) is the set of modalities that provide a valid prototype for semantic unit \(s\). A smaller \(D_{\mathrm{high}}\) indicates tighter alignment among reliable relations, while a larger margin indicates better separation between semantic units.

8.2 Detailed Experimental Setup↩︎

8.2.1 Benchmark Suites↩︎

We evaluate RCL on four benchmark suites that together cover standard multimodal continual learning, recent multimodal continual instruction tuning, and the evidence-use drift targeted by our method.

8.2.1.1 CoIN.

CoIN [4] is used as a primary continual instruction tuning benchmark for MLLMs. It contains diverse multimodal tasks, including visual question answering, OCR reasoning, object recognition, image classification, and referring expression comprehension. Following the official protocol, we train the model sequentially over the task stream and evaluate all seen tasks after each stage. This benchmark is important because many recent MCIT methods report CoIN results, making it a suitable setting for direct SOTA comparison.

8.2.1.2 COAST.

We use the COAST benchmark from Continual LLaVA [5] to evaluate three complementary continual adaptation scenarios: domain-incremental learning, capability-incremental learning, and dataset-incremental learning. The domain-incremental setting evaluates robustness across visual domains such as natural images, documents, charts, medical-style images, and scientific visual content. The capability-incremental setting evaluates whether the model can sequentially acquire different multimodal abilities, including recognition, OCR, chart reasoning, visual conversation, and complex reasoning. The dataset-incremental setting evaluates whether the model can adapt to new datasets without catastrophically forgetting previously learned ones.

8.2.1.3 MCITlib.

We further evaluate on MCITlib [21], which provides leakage-aware and reproducible multimodal continual instruction tuning protocols. We use its UCIT, MLLM-DCL, and MLLM-ACL settings. UCIT evaluates unified continual instruction tuning across heterogeneous task families. MLLM-DCL emphasizes domain continual learning, where the task format may remain similar but the visual or semantic domain changes. MLLM-ACL emphasizes ability continual learning, where the model must sequentially acquire different reasoning and perception abilities. This suite is used to ensure that our conclusions are not specific to a single benchmark design.

8.2.1.4 Evidence-sensitive stream.

To directly evaluate hidden evidence-use forgetting, we construct an eight-stage stream: \[\text{ScienceQA} \rightarrow \text{GQA} \rightarrow \text{VQAv2} \rightarrow \text{OK-VQA} \rightarrow \text{TextVQA} \rightarrow \text{OCR-VQA} \rightarrow \text{ChartQA} \rightarrow \text{DocVQA}.\] This stream is designed to contain tasks with different dominant evidence sources. ScienceQA and GQA emphasize visual reasoning and compositional grounding; VQAv2 and OK-VQA include strong language-prior and external-knowledge effects; TextVQA and OCR-VQA require OCR grounding; ChartQA requires chart-element and numerical reasoning; DocVQA requires document text and layout understanding. This makes the stream suitable for testing whether a continual MLLM preserves the evidence path behind retained answers.

4pt

8.2.2 Dataset Preprocessing and Answer Normalization↩︎

For all benchmarks, we follow the official train/validation/test splits whenever available. If a benchmark provides multiple task orders, we report the average over the official orders. For datasets without an official continual split, we use the original training set for the corresponding continual stage and reserve the official validation set for evaluation. To prevent dataset size from dominating the sequential dynamics, we cap each stage at the same maximum number of training examples unless the official benchmark protocol specifies otherwise.

For closed-form VQA tasks, we apply standard VQA answer normalization, including lowercasing, punctuation removal, article removal, and number normalization. For OCR- and document-centric tasks, we additionally normalize whitespace, line breaks, and common OCR artifacts. For chart and table QA, we normalize numerical answers by allowing minor formatting differences such as commas, percentage signs, and decimal-place variants. For open-ended generative tasks, we use the official evaluator of the corresponding benchmark. When GPT-assisted scoring is required by the benchmark, we use it only for evaluation and never for training or hyperparameter selection.

8.2.3 Task-Level Metrics↩︎

We report the official metric for each task family:

• VQA-style tasks: exact-match or soft VQA accuracy, depending on the dataset evaluator.

• Classification and recognition: top-1 accuracy.

• Referring expression comprehension: \(\mathrm{Acc@0.5}\) based on box IoU.

• OCR and document QA: normalized exact match and ANLS when supported by the official evaluator.

• Chart and table QA: relaxed exact match for numerical answers and official ChartQA-style accuracy.

• Open-ended instruction following: official GPT-assisted or rule-based score, following the benchmark protocol.

When a benchmark reports a single aggregated score, we use the official aggregation. When a benchmark reports separate task-family scores, we report both the task-family score and the macro-average across task families.

8.2.4 Continual Learning Metrics↩︎

Let \(a_{i,j}\) denote the score on evaluation set \(\mathcal{E}_i\) after training stage \(j\). For a \(K\)-stage stream, we report immediate task acquisition, final retained performance, average forgetting, backward transfer, and mean accuracy over the learning trajectory: \[\label{eq:app95standard95cl95metrics} \begin{align} \mathrm{MFT} &= \frac{1}{K}\sum_{i=1}^{K} a_{i,i}, \\ \mathrm{MFN} &= \frac{1}{K}\sum_{i=1}^{K} a_{i,K}, \\ \mathrm{AF} &= \frac{1}{K-1}\sum_{i=1}^{K-1} \left( \max_{j\in\{i,\ldots,K\}}a_{i,j} - a_{i,K} \right), \\ \mathrm{BWT} &= \frac{1}{K-1}\sum_{i=1}^{K-1} \left(a_{i,K}-a_{i,i}\right), \\ \mathrm{MAA} &= \frac{1}{K} \sum_{j=1}^{K} \frac{1}{j} \sum_{i=1}^{j}a_{i,j}. \end{align}\tag{33}\] Here MFT measures how well the model learns each new stage when it is first trained; MFN measures final retained performance; AF measures the amount of forgetting; BWT measures whether later training improves or harms previous tasks; and MAA measures average performance throughout the whole continual trajectory.

8.2.5 Reliance-Aware Metrics↩︎

Standard continual learning metrics only evaluate whether the final answer is correct. Since our work studies hidden evidence-use forgetting, we additionally evaluate how the model’s reliance profile changes during continual adaptation.

For each example \((x,y)\) and checkpoint \(\theta_j\), we compute a normalized reliance vector \(\boldsymbol{r}_{\theta_j}(x,y)\) over available evidence channels. The evidence channels may include image tokens, question tokens, OCR tokens, document text, chart elements, table cells, layout regions, and language-prior cues. The channel set is sample-dependent and is denoted by \(\mathcal{M}_x\).

8.2.5.1 Modality reliance drift.

The reliance drift from stage \(i\) to the final checkpoint \(K\) is \[\label{eq:app95metric95mrd} \mathrm{MRD}_{i\rightarrow K} = \frac{1}{|\mathcal{E}_i|} \sum_{(x,y)\in \mathcal{E}_i} D_{\rm JS} \left( \boldsymbol{r}_{\theta_i}(x,y) \middle\| \boldsymbol{r}_{\theta_K}(x,y) \right).\tag{34}\] We report the stream-level score \[\label{eq:app95metric95mrd95avg} \mathrm{MRD}_{K} = \frac{1}{K-1} \sum_{i=1}^{K-1} \mathrm{MRD}_{i\rightarrow K}.\tag{35}\] Lower MRD indicates that the final model preserves the evidence-use pattern of earlier checkpoints.

8.2.5.2 Dominant evidence flip rate.

The dominant evidence flip rate isolates examples where the answer remains correct but the dominant evidence channel changes: \[\label{eq:app95metric95def} \mathrm{DEF}_{i\rightarrow K} = \mathbb{E}_{(x,y)\in\mathcal{E}_i} \left[ \mathbf{1} \left[ \arg\max_{m\in\mathcal{M}_x} r_{\theta_i,m}(x,y) \neq \arg\max_{m\in\mathcal{M}_x} r_{\theta_K,m}(x,y) \right] \middle| f_{\theta_i}(x)=f_{\theta_K}(x)=y \right].\tag{36}\] A high DEF means that answer preservation is partly misleading: the model still answers correctly but uses a different evidence source.

8.2.5.3 Hidden forgetting rate.

We define hidden forgetting as the event where answer degradation is small but reliance drift is large: \[\label{eq:app95metric95hfr} \mathrm{HFR}_{K} = \frac{1}{K-1} \sum_{i=1}^{K-1} \mathbf{1} \left[ a_{i,i}-a_{i,K}\leq \delta_{\rm acc} \right] \cdot \mathbf{1} \left[ \mathrm{MRD}_{i\rightarrow K}\geq \delta_{\rm rel} \right].\tag{37}\] Unless otherwise specified, we set \(\delta_{\rm acc}=2\%\) and \(\delta_{\rm rel}=0.20\) for all experiments. These thresholds are fixed before evaluation and are not tuned per dataset.

8.2.5.4 Channel-specific drift.

For a channel \(m\), we report \[\label{eq:app95metric95channel95drift} \Delta^{m}_{i\rightarrow K} = \frac{1}{|\mathcal{E}_i^m|} \sum_{(x,y)\in \mathcal{E}_i^m} \left[ r_{\theta_K,m}(x,y)-r_{\theta_i,m}(x,y) \right],\tag{38}\] where \(\mathcal{E}_i^m=\{(x,y)\in\mathcal{E}_i:m\in\mathcal{M}_x\}\). Negative visual, OCR, chart, or document drift indicates that the final model relies less on the corresponding grounded channel than the earlier checkpoint.

8.2.6 General Multimodal Ability Evaluation↩︎

After the final continual stage, we additionally evaluate the final checkpoint on general multimodal ability benchmarks, including POPE, MME, MMBench, and SEED-Bench, following MCITlib [21]. These benchmarks are not used for training or hyperparameter selection. They serve as post-hoc checks that reliance preservation does not merely overfit the continual stream while degrading hallucination robustness, perception, commonsense reasoning, or general visual instruction following.

8.3 Comparison Protocol↩︎

8.3.1 Method Groups↩︎

We group compared methods according to the information and memory they require.

8.3.1.1 Replay-free and task-id-free methods.

This is the primary comparison group. All methods in this group are trained sequentially without storing previous examples and are evaluated without oracle task identity. This group includes Seq-LoRA, EWC, LwF, L2P, DualPrompt, CODA-Prompt, O-LoRA, MoELoRA, Model Tailor, Continual LLaVA, CL-MoE, HiDe-LLaVA, BranchLoRA, ProgLoRA, ModalPrompt, SEFE, MLLM-CL, IGVP, and RCL. This table is the main table for fair comparison.

8.3.1.2 Replay- or memory-assisted methods.

Some methods store old examples, generate replay data, or preserve past-task questions. We report them separately because they use additional past-task signals unavailable to strictly replay-free methods. This group includes rehearsal, DER [24], GIFT [12], and Ask-and-Remember [25]. For rehearsal and DER, we use a \(1\%\) memory buffer unless otherwise specified by the official protocol. For synthetic replay methods, we follow the official generation budget and report the additional generated-sample count.

8.3.1.3 Oracle or task-aware methods.

If a method requires oracle task identifiers, a task-specific expert selector, or separate task-specific heads at inference time, we report it in an oracle table. Oracle results are useful upper references but are not mixed with the main replay-free task-id-free comparison. When the official implementation provides both oracle and non-oracle variants, we report both.

8.3.2 Fairness Rules↩︎

We apply the following fairness rules across all directly comparable methods:

• Same task order. All methods are trained on the same continual stream and evaluated after the same stages.

• Same data budget. Each method sees the same current-stage training examples. Replay-free methods do not access previous-stage examples.

• Same backbone. Direct comparisons are made under the same MLLM backbone.

• Matched trainable parameters. When a method allows rank, prompt length, or expert-count adjustment, we tune it so that the trainable parameter count is within \(5\%\) of RCL.

• Matched optimization budget. Methods are trained with the same number of epochs, batch size, and total update steps whenever their official protocol permits.

• No hidden task identity. The main table does not provide oracle task IDs at inference time.

• Separate memory accounting. Methods using stored or generated replay are reported with their memory or generation budget.

• Same evaluation interventions. Reliance-aware diagnostics use the same post-hoc counterfactual intervention set for every method.

8.3.3 Baseline Implementation Notes↩︎

8.3.3.1 Seq-LoRA.

Seq-LoRA updates LoRA adapters sequentially on each stage using only the current-stage task loss. It is the closest minimal baseline to RCL because it uses the same backbone and adapter placement but does not preserve predictions or reliance profiles.

8.3.3.2 Answer-KD / LwF.

Answer-KD preserves the previous checkpoint’s output distribution on current-stage samples using token-level KL divergence. It tests whether output preservation alone is sufficient to prevent hidden evidence-use forgetting.

8.3.3.3 Prompt-based continual learning.

L2P, DualPrompt, and CODA-Prompt are adapted to the multimodal instruction tuning setting by inserting learnable prompts into the language-side input or adapter-compatible prompt slots. When task identifiers are not available, prompt selection uses the official key-query matching mechanism.

8.3.3.4 Adapter- and LoRA-based MCIT methods.

O-LoRA, BranchLoRA, ProgLoRA, Model Tailor, HiDe-LLaVA, ModalPrompt, CL-MoE, SEFE, and MLLM-CL are implemented following their official designs whenever code is available. If official code is unavailable, we reproduce the method from the paper using the same backbone and training schedule used for other baselines.

8.3.3.5 Replay-based methods.

For rehearsal and DER, we maintain a fixed-size buffer selected uniformly from previous stages. For GIFT and Ask-and-Remember, we follow the original synthetic-data or question-only replay protocol and report the replay budget separately.

8.4 Implementation Details↩︎

8.4.1 Backbones↩︎

We use three MLLM backbones:

• LLaVA-1.5-7B [22]: the default backbone for main experiments because it is widely used in recent MCIT work.

• LLaVA-1.5-13B [22]: used to evaluate whether reliance preservation remains effective at a larger model scale.

• InternVL-Chat-7B [23]: used to test whether RCL generalizes beyond the LLaVA model family.

Unless otherwise specified, the visual encoder, language backbone, and multimodal projector are frozen. Only PEFT parameters are updated during continual adaptation.

8.4.2 PEFT Configuration↩︎

For RCL, we use LoRA adapters on attention projection layers. The default configuration is: \[r=16,\qquad \alpha=32,\qquad p_{\rm dropout}=0.05.\] For larger backbones, we keep the same LoRA rank unless otherwise specified. For baselines with tunable parameter budgets, such as prompt length, expert number, or adapter rank, we adjust their configuration to match the trainable parameter count of RCL within \(5\%\).

8.4.3 Optimization↩︎

We train all methods using AdamW with bfloat16 mixed precision. The default optimization configuration is: \[\text{global batch size}=32,\quad \text{warm-up ratio}=3\%,\quad \text{gradient clipping}=1.0,\quad \text{weight decay}=0.\] For LLaVA-style backbones, the default learning rate is \(2\times 10^{-4}\) for LoRA parameters. For InternVL-style backbones, we use \(1\times 10^{-4}\) for stability. For COAST and MCITlib settings with official hyperparameters, we additionally report the official-setting result when it differs from the matched-budget setting. Each continual stage is trained for two epochs unless the official benchmark protocol requires a fixed update budget.

8.4.4 RCL Hyperparameters↩︎

The default RCL objective is \[\mathcal{L} = \mathcal{L}_{\rm task} + \lambda_{\rm pred}\mathcal{L}_{\rm pred} + \lambda_{\rm rel}\mathcal{L}_{\rm rel} + \lambda_{\rm reg}\|\phi-\phi^-\|_2^2.\] We use the following default values: \[\tau_r=0.5,\quad \tau_{\rm kd}=2.0,\quad \beta=1.0,\quad \lambda_{\rm pred}=0.5,\quad \lambda_{\rm rel}=1.0,\quad \lambda_{\rm reg}=10^{-4}.\] We select these values using the validation split of the first stream only and keep them fixed for all later stages and all benchmark suites. This avoids using previous-task validation sets as a hidden replay signal.

8.4.5 Counterfactual Intervention Budget↩︎

For each sample, RCL identifies the available evidence channels \(\mathcal{M}_x\). During training, we sample one intervention per available channel: \[q_{\rm train}=1.\] During diagnostic evaluation, we use four interventions per available channel: \[q_{\rm eval}=4.\] The same intervened inputs are used for the previous checkpoint and current checkpoint, so teacher–student reliance vectors are compared under identical counterfactual views.

8.4.6 Evidence Channel Taxonomy↩︎

We instantiate evidence channels according to the input structure of each dataset:

• Image channel: image patches, visual tokens, or annotated regions.

• Question channel: instruction tokens and content-bearing question tokens.

• OCR channel: recognized text tokens and OCR bounding boxes.

• Document channel: document text spans, line blocks, and layout regions.

• Chart/table channel: axes, legends, plotted marks, table cells, and numerical entries.

• Language-prior channel: lexical cues in the question that can induce shortcut answers.

For datasets without explicit OCR, document, or chart annotations, the corresponding channels are omitted rather than approximated. Reliance vectors are always normalized over the channels available in the current sample.

8.4.7 Counterfactual Intervention Construction↩︎

We use evidence-suppressing interventions rather than task-preserving augmentations for reliance estimation. The goal is to measure whether suppressing a channel changes the model’s output distribution or answer likelihood.

8.4.7.1 Image interventions.

We mask image patches, remove selected visual tokens, or replace visual regions with neutral image statistics. If object annotations are available, regions are sampled from annotated objects; otherwise, patches are sampled from a uniform grid.

8.4.7.2 OCR interventions.

We remove OCR tokens, replace them with a neutral placeholder, or mask OCR spans while preserving the prompt template. When OCR bounding boxes are available, visual regions corresponding to OCR spans are also masked in an aligned intervention.

8.4.7.3 Document interventions.

We mask document lines, paragraph blocks, or layout regions. For DocVQA-style inputs, we preserve page order and prompt structure while suppressing the selected document evidence.

8.4.7.4 Chart and table interventions.

We mask axis labels, legends, plotted elements, cell values, and numerical entries. For tables, we distinguish header masking from body-cell masking because these two perturbations often affect different reasoning paths.

8.4.7.5 Question-prior interventions.

We mask content words in the question while preserving stop words, formatting tokens, and instruction templates. This intervention estimates whether the model relies on question-side language priors rather than grounded visual or textual evidence.

8.4.8 Compute Resources↩︎

All main experiments are run on \(8\) NVIDIA A100-80GB GPUs. For LLaVA-1.5-7B experiments, each continual stage uses approximately \(6\)–\(10\) GPU hours depending on the dataset size and intervention budget. For LLaVA-1.5-13B, each stage uses approximately \(12\)–\(18\) GPU hours. For InternVL-Chat-7B, each stage uses approximately \(8\)–\(12\) GPU hours. The full main benchmark suite requires approximately \(1{,}200\)–\(1{,}600\) A100 GPU hours, including three random seeds, ablations, and diagnostic evaluation. Reliance diagnostics are more expensive than standard answer evaluation because they require counterfactual forward passes, but they are used only during training and evaluation, not at deployment time.

8.4.9 Random Seeds and Reporting↩︎

Unless otherwise specified, we report the mean and standard deviation over three random seeds. The seeds affect model initialization of PEFT parameters, task sampling order when multiple official orders are available, data shuffling, and intervention sampling. For main tables, we report mean performance. For ablations and diagnostic plots, we include standard deviation or confidence intervals when space permits. All reliance-aware metrics are computed on the same evaluation examples across methods to avoid intervention-sampling artifacts.

8.4.10 Inference Cost↩︎

RCL uses counterfactual interventions and the frozen previous checkpoint only during training and diagnostic evaluation. After each continual stage, the deployed model is the updated PEFT-adapted MLLM. No reference model, intervention module, or reliance estimator is used at inference time. Therefore, RCL has the same inference-time architecture, memory footprint, and latency as the underlying LoRA-adapted backbone.

8.4.11 Hyperparameter Sensitivity↩︎

We evaluate sensitivity to three key hyperparameters: \[\lambda_{\rm rel}\in\{0.25,0.5,1.0,2.0\},\qquad \tau_r\in\{0.25,0.5,1.0\},\qquad q_{\rm train}\in\{1,2,4\}.\] We report both answer-level metrics and reliance-aware metrics. This analysis tests whether reliance preservation is robust to the strength of the reliance loss, the sharpness of the reliance distribution, and the number of counterfactual interventions used during training.

8.4.12 Ablation Settings↩︎

We evaluate the following ablations:

• w/o reliance loss: removes \(\mathcal{L}_{\rm rel}\) and keeps only task learning and prediction distillation.

• w/o prediction distillation: removes \(\mathcal{L}_{\rm pred}\) and keeps task learning and reliance preservation.

• w/o confidence gate: sets \(w_{\theta^-}(x,y)=1\) for all examples.

• visual-only reliance: preserves reliance only for visual channels.

• text/OCR-only reliance: preserves reliance only for text and OCR channels.

• random interventions: replaces evidence-targeted interventions with random token or patch masking.

• larger intervention budget: increases \(q_{\rm train}\) from \(1\) to \(2\) or \(4\).

The ablations are designed to test whether the performance gain comes from reliance preservation itself, from output distillation, from the confidence gate, or merely from generic perturbation regularization.

8.5 Main Experiment Details↩︎

8.5.0.1 Answer-drop–MRD bubble protocol.

This paragraph provides the full protocol for Figure 3. For every method \(\mathcal{A}\), random seed, old stage \(j\), later checkpoint \(k>j\), and diagnostic slice \(\mathcal{E}_{j,s}\), we compute the clean answer drop \[\label{eq:app95answer95drop} \Delta_{\mathrm{ans}}^{(\mathcal{A})}(j,k,s) = \mathrm{Acc}\!\left(f_{\theta_j}^{(\mathcal{A})};\mathcal{E}_{j,s}\right) - \mathrm{Acc}\!\left(f_{\theta_k}^{(\mathcal{A})};\mathcal{E}_{j,s}\right).\tag{39}\] We compute modality reliance drift using the reliance vectors defined in Section 4.2: \[\label{eq:app95mrd95transition} \mathrm{MRD}^{(\mathcal{A})}_{j\rightarrow k,s} = \frac{1}{|\mathcal{E}_{j,s}|} \sum_{(x,y)\in\mathcal{E}_{j,s}} D_{\rm JS} \left( \boldsymbol{r}_{\theta_j}^{(\mathcal{A})}(x,y) \middle\| \boldsymbol{r}_{\theta_k}^{(\mathcal{A})}(x,y) \right).\tag{40}\] The dominant evidence flip rate is computed only on examples whose answers are preserved: \[\label{eq:app95def95transition} \mathrm{DEF}^{(\mathcal{A})}_{j\rightarrow k,s} = \mathbb{E}_{(x,y)\in\mathcal{E}_{j,s}} \left[ \mathbf{1} \left[ \arg\max_{m\in\mathcal{M}_x} r_{\theta_j,m}^{(\mathcal{A})}(x,y) \neq \arg\max_{m\in\mathcal{M}_x} r_{\theta_k,m}^{(\mathcal{A})}(x,y) \right] \middle| f_{\theta_j}^{(\mathcal{A})}(x)=f_{\theta_k}^{(\mathcal{A})}(x)=y \right].\tag{41}\] Each bubble in Figure 3 corresponds to one tuple \[(\mathcal{A},\; \mathrm{seed},\; j,\; k,\; s).\] The horizontal coordinate is Eq. 39 , the vertical coordinate is Eq. 40 , and the bubble area is proportional to Eq. 41 . We mark a slice as hidden forgetting when \[\label{eq:app95hidden95region95threshold} \Delta_{\mathrm{ans}}^{(\mathcal{A})}(j,k,s)\leq 2\%, \qquad \mathrm{MRD}^{(\mathcal{A})}_{j\rightarrow k,s}\geq 0.20.\tag{42}\] These thresholds are fixed before evaluation and are used only to visualize the discrepancy between answer preservation and evidence-use preservation. All reported summary statistics are averaged over old stages, later checkpoints, diagnostic slices, and three random seeds.

8.5.0.2 Dominant-channel Sankey protocol.

This paragraph provides the construction details for Figure 5. For each method \(\mathcal{A}\) and old stage \(j\), we retain only answer-preserved examples: \[\label{eq:app95answer95preserved95set} \Omega^{(\mathcal{A})}_j = \left\{ (x,y)\in\mathcal{E}_j: f_{\theta_j}^{(\mathcal{A})}(x)=f_{\theta_K}^{(\mathcal{A})}(x)=y \right\},\tag{43}\] where \(K\) denotes the final continual stage. For each example, we identify the dominant evidence channel before and after continual adaptation: \[\label{eq:app95dominant95channel} c_j^{(\mathcal{A})}(x,y) = \arg\max_{m\in\mathcal{M}_x} r_{\theta_j,m}^{(\mathcal{A})}(x,y), \qquad c_K^{(\mathcal{A})}(x,y) = \arg\max_{m\in\mathcal{M}_x} r_{\theta_K,m}^{(\mathcal{A})}(x,y).\tag{44}\] The Sankey transition probability from old dominant channel \(u\) to final dominant channel \(v\) is \[\label{eq:app95sankey95transition} T_{u\rightarrow v}^{(\mathcal{A})} = \frac{ \sum_{j=1}^{K-1} \sum_{(x,y)\in\Omega^{(\mathcal{A})}_j} \mathbf{1}\!\left[c_j^{(\mathcal{A})}(x,y)=u\right] \mathbf{1}\!\left[c_K^{(\mathcal{A})}(x,y)=v\right] }{ \sum_{j=1}^{K-1} \sum_{(x,y)\in\Omega^{(\mathcal{A})}_j} \mathbf{1}\!\left[c_j^{(\mathcal{A})}(x,y)=u\right] +\epsilon }.\tag{45}\] Diagonal retention is computed as \[\label{eq:app95diagonal95retention} \mathrm{DiagRet}^{(\mathcal{A})} = \frac{ \sum_{j=1}^{K-1} \sum_{(x,y)\in\Omega^{(\mathcal{A})}_j} \mathbf{1}\!\left[ c_j^{(\mathcal{A})}(x,y)=c_K^{(\mathcal{A})}(x,y) \right] }{ \sum_{j=1}^{K-1} |\Omega^{(\mathcal{A})}_j| +\epsilon }.\tag{46}\] Shortcut-oriented flip rate into the question-prior channel \(q\) is \[\label{eq:app95question95prior95flip} \mathrm{QFlip}^{(\mathcal{A})} = \frac{ \sum_{j=1}^{K-1} \sum_{(x,y)\in\Omega^{(\mathcal{A})}_j} \mathbf{1}\!\left[ c_j^{(\mathcal{A})}(x,y)\neq q \right] \mathbf{1}\!\left[ c_K^{(\mathcal{A})}(x,y)=q \right] }{ \sum_{j=1}^{K-1} |\Omega^{(\mathcal{A})}_j| +\epsilon }.\tag{47}\] In Figure 5, ribbon width is proportional to Eq. 45 . Red-tinted ribbons correspond to transitions whose final channel is the question-prior channel. We additionally report channel-specific diagonal retention for OCR, chart, and document channels to quantify whether structured evidence remains the dominant support for answer-preserved examples.

8.5.0.3 Cost–performance Pareto protocol.

This paragraph provides the full protocol for Figure 4. We compare the following configurations: Seq-LoRA, Answer-KD, random intervention regularization, sparse RCL, and RCL with intervention budgets \(q_{\rm train}\in\{1,2,4\}\). For each configuration \(\mathcal{A}\), the relative training cost is \[\label{eq:app95relative95training95cost} C_{\mathrm{rel}}(\mathcal{A}) = \frac{ \mathrm{GPUHours}(\mathcal{A}) }{ \mathrm{GPUHours}(\mathrm{Seq-LoRA}) }.\tag{48}\] The hidden-forgetting reduction over Seq-LoRA is \[\label{eq:app95hfr95reduction} \mathrm{HFR-Red.}(\mathcal{A}) = \frac{ \mathrm{HFR}(\mathrm{Seq-LoRA}) - \mathrm{HFR}(\mathcal{A}) }{ \mathrm{HFR}(\mathrm{Seq-LoRA}) } \times 100.\tag{49}\] A method is Pareto-dominated if another method achieves both lower or equal relative cost and higher or equal HFR reduction, with at least one strict improvement: \[\label{eq:app95pareto95dominance} \mathcal{A} \prec \mathcal{B} \quad\Longleftrightarrow\quad C_{\mathrm{rel}}(\mathcal{B})\leq C_{\mathrm{rel}}(\mathcal{A}), \quad \mathrm{HFR-Red.}(\mathcal{B})\geq \mathrm{HFR-Red.}(\mathcal{A}),\tag{50}\] with one inequality strict. In Figure 4, the \(x\)-axis is Eq. 48 , the \(y\)-axis is Eq. 49 , bubble area is proportional to final average answer performance, and color encodes MRD. The default RCL configuration uses \(q_{\rm train}=1\), i.e., one targeted intervention per available evidence channel. Because counterfactual interventions are used only during training and diagnostic evaluation, all points in the figure have the same inference-time architecture and latency as their corresponding PEFT-adapted backbone.

9 Additional Experimental Results↩︎

9.1 Does RCL Prevent Dominant Evidence-Channel Flips?↩︎

To make hidden reliance drift interpretable, we inspect whether answer-preserved examples retain the same dominant evidence channel after continual adaptation. For each old-stage example answered correctly by both the old and final checkpoints, we compute the dominant channel before and after adaptation and aggregate the resulting transition matrix. Figure 5 shows that Answer-KD preserves many final answers but keeps only \(54.2\%\) of answer-preserved examples on the same dominant channel, while \(27.8\%\) flip into the question-prior channel. The shortcut-oriented transitions OCR\(\rightarrow\)question-prior, chart\(\rightarrow\)question-prior, and document\(\rightarrow\)question-prior together account for \(18.6\%\) of all answer-preserved examples. RCL increases diagonal retention to \(78.6\%\) and reduces question-prior flips to \(8.9\%\), with especially large gains for OCR, chart, and document evidence. These channel-level results explain why RCL lowers DEF and HFR: it preserves not only the old answer, but also the evidence path that supported it.

9.2 Task-Order Robustness↩︎

9.2.0.1 Protocol.

We first test whether the improvement of RCL depends on a favorable continual task order. In addition to the default stream used in the main paper, we evaluate the reversed stream and three random task orders. For each order, we report final average performance, average forgetting, and reliance-aware diagnostics. The random-order row reports the mean and standard deviation over three independently sampled task permutations.

4.5pt

9.2.0.2 Analysis.

Table 5 shows that the performance of RCL is stable across task orders. The average performance varies by less than \(0.6\) points between the default and reversed streams, and HFR remains below \(15\%\) in all settings. By contrast, Seq-LoRA and Answer-KD remain sensitive to order changes, especially in MRD and HFR. This suggests that reliance preservation is not exploiting a specific order of OCR-, chart-, or document-centric tasks; rather, it provides a general stabilization signal for old evidence-use behavior.

9.3 Reliance Metric Validation↩︎

9.3.0.1 Protocol.

MRD is computed from counterfactual evidence suppression. To verify that this diagnostic is not an artifact of the intervention design, we compare the channel ranking induced by counterfactual reliance with an independent gradient\(\times\)activation attribution signal. For each answer-preserved example, we compute the Spearman rank correlation between the two channel rankings. We also report the Jensen–Shannon divergence between the normalized counterfactual reliance vector and the normalized attribution vector. Higher rank correlation and lower JS divergence indicate better agreement between the two diagnostic views.

4.2pt

9.3.0.2 Analysis.

Table 6 supports the validity of counterfactual reliance as a diagnostic signal. Across all task families, counterfactual reliance correlates positively with gradient-based attribution, and the agreement is strongest under RCL. The largest improvements occur on OCR, chart, and document tasks, where evidence channels are structured and channel flips are more harmful. This result suggests that MRD does not merely measure arbitrary perturbation sensitivity; it tracks a channel-level evidence-use pattern that is also reflected in local attribution.

9.4 Task-Preserving Perturbation Test↩︎

9.4.0.1 Protocol.

A desirable grounded model should be sensitive to evidence removal but stable under harmless input variations. We therefore compare two perturbation families. The first family contains evidence-suppressing interventions, such as masking OCR tokens, chart elements, document spans, or visual regions. The second family contains task-preserving perturbations, such as mild image noise, question paraphrases, OCR whitespace changes, and chart style changes that do not alter the answer. For each method, we report output KL shift and accuracy drop under both perturbation types, as well as the separation ratio between evidence-suppressing and task-preserving sensitivity.

4.2pt

9.4.0.2 Analysis.

Table 7 shows that RCL does not simply make the model more sensitive to all input changes. Instead, it increases sensitivity to evidence-suppressing interventions while reducing sensitivity to task-preserving perturbations. Random interventions increase both ES-KL and TP-KL, which indicates generic perturbation sensitivity rather than grounded evidence use. In contrast, RCL achieves the highest separation ratio and the lowest task-preserving accuracy drop. This supports the intended behavior of the method: the model should react when task-relevant evidence is removed, but should not change substantially under irrelevant formatting or style changes.

9.5 Bias-Stratified Hidden Forgetting↩︎

9.5.0.1 Protocol.

We next test whether RCL is especially helpful when language shortcuts are strong. For each example, we estimate a question-prior score using a question-only model and divide the evaluation set into low-, medium-, and high-prior-bias groups. We then compare HFR, DEF, and GCR in each group. This stratification directly tests whether reliance preservation prevents the model from drifting toward shortcut priors on examples where such shortcuts are available.

4.2pt

9.5.0.2 Analysis.

Table 8 shows that hidden forgetting becomes more severe as question-prior bias increases. For Seq-LoRA and Answer-KD, high-prior-bias examples have substantially higher DEF and HFR, indicating that correct answers are often preserved through shortcut reliance. RCL reduces HFR in all groups and achieves the largest absolute reduction in the high-bias group, decreasing HFR from \(69.3\%\) under Answer-KD to \(16.5\%\). This confirms that reliance preservation is particularly useful when answer correctness alone is least reliable as a grounding indicator.

9.6 Evidence-Channel Retention by Task Family↩︎

9.6.0.1 Protocol.

We further analyze which evidence channels benefit most from RCL. For each old-task family, we compute the retention rate of the originally dominant grounded channel on answer-preserved examples. For example, OCR retention measures the fraction of answer-preserved examples whose dominant channel remains OCR after continual adaptation, conditioned on OCR being the old dominant channel. We also report the shortcut-flip rate into the question-prior channel.

3.8pt

9.6.0.2 Analysis.

Table 9 shows that the largest improvements occur on structured evidence channels. For Answer-KD, OCR, chart, and document retention remains below \(50\%\) in several task families, even though the answers are preserved. This means that output distillation often keeps the surface answer while allowing the evidence path to move toward question priors. RCL improves OCR retention to \(76.4\%\), chart retention to \(74.8\%\), and document retention to \(73.2\%\), while reducing flips into question-prior reliance to around \(9\)–\(13\%\). These results provide channel-level evidence that RCL preserves grounded reasoning rather than merely suppressing parameter drift.

9.7 Hyperparameter Sensitivity↩︎

9.7.0.1 Protocol.

We evaluate sensitivity to the reliance-loss weight \(\lambda_{\rm rel}\) and the reliance temperature \(\tau_r\). The first parameter controls the strength of reliance preservation, while the second controls the sharpness of the normalized reliance vector. All other hyperparameters are fixed to the default setting. We report both answer-level and reliance-aware metrics.

5.0pt

5.0pt

9.7.0.2 Analysis.

Tables 10 and 11 show that RCL is not overly sensitive to its main hyperparameters. Removing reliance preservation entirely produces the largest degradation in MRD, DEF, and HFR. Increasing \(\lambda_{\rm rel}\) from \(0.25\) to \(1.0\) consistently improves reliance preservation, while \(\lambda_{\rm rel}=2.0\) gives only a small additional reduction in MRD but slightly lowers average answer performance. This suggests a mild over-conservation effect when the reliance constraint is too strong. For \(\tau_r\), the default value \(0.5\) performs best, but both neighboring settings remain competitive. Overall, the default configuration is a stable and reasonable operating point rather than a fragile optimum.

9.8 Confidence-Gate Analysis↩︎

9.8.0.1 Protocol.

The confidence gate downweights unreliable teacher reliance targets. To analyze its effect, we bin samples by teacher predictive entropy and total intervention sensitivity. Low entropy indicates confident teacher predictions, while high intervention sensitivity indicates that the teacher has a measurable evidence-use pattern. For each bin, we compare full RCL with the variant that removes the gate.

4.2pt

9.8.0.2 Analysis.

Table 12 confirms that the confidence gate is mainly needed to avoid overfitting unreliable teacher targets. When the teacher is confident and evidence-sensitive, removing the gate causes only a small degradation. However, when teacher entropy is high or intervention sensitivity is low, the gap becomes much larger. In the high-entropy, low-sensitivity bin, the gate reduces HFR from \(31.5\%\) to \(20.9\%\) and improves GCR from \(77.8\%\) to \(83.5\%\). This indicates that the gate does not merely improve average performance; it specifically protects the model from preserving noisy or weakly grounded teacher reliance profiles.

9.9 Failure-Case Taxonomy↩︎

9.9.0.1 Protocol.

Finally, we inspect the remaining high-MRD examples under the default RCL model. We sample \(300\) answer-preserved examples with the highest MRD and manually categorize the dominant failure source. The goal is to understand the residual limitations of reliance preservation, not to introduce a new training signal.

4.5pt

9.9.0.2 Analysis.

Table 13 shows that the remaining failures of RCL are not dominated by a single systematic weakness. Only \(10.2\%\) of high-MRD cases are attributed to incorrect teacher reliance, suggesting that the confidence gate is largely effective. The largest categories involve ambiguous evidence, OCR noise, chart parsing ambiguity, and document layout mismatch. These are expected in multimodal reasoning settings where evidence channels are not always cleanly separable. Importantly, the taxonomy does not show a widespread collapse into question-prior shortcuts, which supports the main conclusion that RCL substantially mitigates hidden evidence-use forgetting.

10 Additional Discussion↩︎

10.1 Limitations↩︎

This work focuses on a specific but important failure mode in continual multimodal learning: the model may preserve the final answer while changing the evidence path used to obtain it. The proposed RCL framework assumes that each input can be decomposed into meaningful evidence channels, such as image tokens, question tokens, OCR tokens, chart elements, document spans, or table cells. This assumption is natural for the multimodal benchmarks considered in this paper, but future applications with more implicit or weakly structured inputs may require task-specific channel construction.

RCL also uses counterfactual channel interventions during training and diagnostic evaluation. These interventions introduce additional training-time computation, although they are not used at inference time and therefore do not change the deployed model architecture, memory footprint, or latency. In this paper, we control this cost by sampling a small fixed number of interventions per available channel and by sharing the same intervened inputs between the teacher and student models.

Our empirical evaluation covers standard continual multimodal instruction tuning benchmarks, an evidence-sensitive multimodal stream, and three representative MLLM backbones. These settings are designed to test the central claim that answer-level evaluation can miss reliance drift. Extending the same principle to longer streams, additional modalities such as video and audio, and more specialized deployment domains is a natural direction for future work.

10.2 Broader Impacts↩︎

The main positive impact of this work is to improve the reliability and interpretability of continual multimodal adaptation. In deployed multimodal assistants, preserving only the final answer may be insufficient: a model can appear accurate while gradually relying less on grounded visual, textual, OCR, chart, or document evidence. By explicitly measuring and constraining evidence-use drift, RCL encourages models to preserve the evidence path behind correct answers, which may benefit applications such as document understanding, chart reasoning, educational assistants, accessibility tools, and visual question answering systems.

At the same time, reliance preservation should not be interpreted as a complete safety guarantee. A model that preserves its previous reliance pattern may still inherit biases, spurious correlations, or factual errors from the reference checkpoint. Therefore, high-stakes applications should combine reliance-aware continual learning with task-specific evaluations for safety, fairness, privacy, and robustness. The proposed method is intended as a diagnostic and training-time reliability mechanism rather than as a standalone deployment safeguard.

The method does not require collecting new human-subject data, scraping new datasets, or releasing a new high-risk pretrained model. The main environmental cost comes from additional training-time counterfactual forward passes. We mitigate this cost by keeping the backbone frozen, updating only PEFT parameters, using a small intervention budget, and removing all reliance-estimation modules at inference time.

10.3 Existing Assets and Licenses↩︎

This paper uses existing public datasets, pretrained multimodal backbones, and baseline implementations. We cite the original papers for all datasets, models, and methods used in the experiments, and we follow their official licenses, research-use restrictions, and access terms. We do not redistribute raw datasets, pretrained model weights, or third-party code as part of this submission.

The main benchmark assets include CoIN, COAST, MCITlib, ScienceQA, GQA, VQAv2, OK-VQA, TextVQA, OCR-VQA, ChartQA, and DocVQA. The main pretrained backbones include LLaVA-1.5-7B, LLaVA-1.5-13B, and InternVL-Chat-7B. When reproducing the experiments, users should obtain each dataset and pretrained checkpoint from its official source and comply with the corresponding license or access agreement.

Baseline methods are either implemented from their published descriptions or adapted from official implementations when available. If an official implementation is used, we follow its license and attribution requirements. If an asset does not provide an explicit license, we use it only under its stated research-access terms and do not redistribute it.

10.4 LLM Usage↩︎

This paper studies continual adaptation of multimodal large language models and therefore uses MLLM backbones as experimental subjects. The backbones, including LLaVA-1.5 and InternVL-Chat, are specified in the experimental setup, and no closed-source LLM is used as an oracle evaluator, label generator, or hidden supervision source for the main method.

LLM-based tools may have been used for assistive purposes such as wording, grammar correction, LaTeX formatting, drafting code snippets, debugging, and organizing notes. Such usage did not determine the scientific claims, method design, experimental protocol, or reported results. All technical content, mathematical formulations, experimental choices, and conclusions were checked and approved by the authors.

10.5 Compute Resources↩︎

All main experiments are run on NVIDIA A100-80GB GPUs using bfloat16 mixed precision. Each training job uses up to 8 GPUs, with CPU workers used for data loading, preprocessing, and evaluation. The typical CPU configuration is 64–128 vCPUs with 512GB–1TB host memory per multi-GPU node. The full project uses approximately 4–6TB of temporary storage for datasets, checkpoints, logs, and counterfactual evaluation outputs.

For LLaVA-1.5-7B, each continual stage requires approximately 6–10 A100 GPU hours, depending on dataset size and the number of available evidence channels. For LLaVA-1.5-13B, each stage requires approximately 12–18 A100 GPU hours. For InternVL-Chat-7B, each stage requires approximately 8–12 A100 GPU hours. These estimates include task learning, prediction distillation, reliance estimation, and counterfactual intervention forward passes.

Reliance-aware diagnostic evaluation is more expensive than standard answer evaluation because it requires counterfactual forward passes over multiple evidence channels. In our default setting, training uses one sampled intervention per available channel, while diagnostic evaluation uses four interventions per available channel. This overhead is incurred only during training and evaluation; the final deployed model has the same inference-time architecture and latency as the underlying LoRA-adapted MLLM.

The full set of reported experiments, including main benchmark comparisons, three random seeds, ablations, hyperparameter sensitivity analyses, and reliance-aware diagnostic plots, requires approximately 1,200–1,600 A100 GPU hours. Preliminary diagnostic studies and configuration checks require an additional approximately 200–300 A100 GPU hours. No inference-time counterfactual intervention module, reference model, or reliance estimator is used after training.

References↩︎