1 Introduction↩︎

Large Vision-Language Models (VLMs) [1]–[4] have demonstrated strong performance across diverse multimodal tasks, including open-ended video question answering [5], [6] and spatial reasoning [7], [8]. These models couple pretrained vision encoders [9], [10] with Large Language Models (LLMs) [11], [12] and are trained on large-scale, web-curated image-video-text corpora containing broad but generic visual concepts [13]–[16]. However, when deployed in domains that exhibit a pronounced visual gap arising from differences in viewpoint, sensing modality, or task structure, their performance degrades substantially under distribution shift. This challenge is prevalent in video domain adaptation settings such as exocentric \(\rightarrow\) egocentric viewpoint, RGB \(\rightarrow\) depth modality, and visual understanding \(\rightarrow\) robotic control, where learning domain-specific visual representations is essential. Crucially, VLMs must adapt while preserving the broad knowledge acquired during pretraining: for example, a model that is adapted to egocentric video data should remain competent on exocentric video benchmarks.

A common strategy to address such distributional shift is partial fine-tuning of a pretrained VLM on domain-specific video instruction data. Freezing the vision encoder and updating only lightweight components (e.g., the vision–language connector) preserves pretrained knowledge but restricts visual specialization. In contrast, finetuning the vision encoder enables specialized visual understanding, albeit at the cost of catastrophic forgetting of pretrained knowledge [17]–[19]. This trade-off is particularly severe when the dominant shift is visual. Moreover, existing approaches rely primarily on the final-layer representations of a frozen vision encoder, discarding intermediate features that encode multi-level visual structure [20]. We argue that such abstracted representations are insufficient for visually prominent domain shifts and pose the following question: how can a VLM leverage domain-specific visual signals across the depth of a pretrained vision encoder while avoiding catastrophic forgetting?

To this end, we introduce Vision Contextualized Probing (VisCoP), a lightweight adaptation mechanism that enables pretrained VLMs to specialize to a target domain while preserving their general-purpose visual knowledge. Motivated by the progressive emergence of semantic representations across transformer depths [21]–[23], VisCoP performs two way probing: it first extracts domain relevant signals from intermediate layers of a frozen vision encoder, and subsequently conditions the language model embedding space to enhance domain specific visual reasoning. This is achieved through a compact set of learnable probe tokens that interact layer wise with intermediate visual representations, forming an alternative parameterized pathway for adaptation. By leveraging multi-level abstractions across the encoder depth, VisCoP learns a parallel representation pathway that extracts domain-specific signals from intermediate encoder features while preserving pretrained representations. This enables the model to learn domain-specific visual patterns while mitigating catastrophic forgetting. In contrast, existing approaches [24]–[28] typically operate on the final-layer representations of the vision encoder or introduce prompt tokens that condition downstream reasoning without explicitly learning new visual cues within the vision encoder. This design provides a simple yet effective approach to video domain adaptation under prominent visual gaps without modifying the pretrained visual encoder parameters, which is not enabled by existing approaches. Empirically, we find that the representations learned via the VisCoP adaptation pathway enable effective cross-view, cross-modal, and cross-task adaptation of VLMs, while retaining their broad capabilities learned during pretraining. Metaphorically, the name VisCoPreflects its role as a “traffic cop”, directing gradient flows away from the visual encoder and towards an alternative pathway for learning domain-specific visual features, avoiding the “crash" (catastrophic forgetting) that would otherwise occur if gradients flowed directly through the vision encoder.

To summarize, our key contributions:

1. We propose VisCoP (Vision Contextualized Probing), a novel video domain adaptation strategy for VLMs that learns domain-specific visual representations through layer-wise probing of a frozen vision encoder, enabling effective domain transfer and preventing catastrophic forgetting of multi-modal capabilities acquired during pretraining.

2. We establish a comprehensive evaluation setting for domain adaptation in VLMs, spanning three challenging target domains: cross-view (exocentric \(\to\) egocentric), cross-modality (RGB \(\to\) depth), and cross-task (action understanding \(\to\) robotic control), along with standardized metrics to evaluate performance.

3. Our experiments demonstrate that VLMs trained with VisCoP outperform alternative domain adaptation strategies across diverse target domains, while retaining more knowledge of the source domain illustrated in Figure 1.

2 Related Works↩︎

Domain adaptation in vision-language encoders. Domain adaptation of contrastively trained vision-language encoders, such as CLIP [9], [29], is typically achieved through prompt tuning or adapter-based approaches. Both strategies aim to learn domain-specific features while keeping the pretrained vision and text encoders frozen. To accomplish this, prompt tuning approaches [30]–[32] introduce learnable prompt vectors as additional input to the text encoder, steering the model toward target domain. Adapter-based approaches [17], [33] insert lightweight trainable modules directly into the encoder space, thus updating their pretrained representations. Head2Toe [34] similarly exploits intermediate representations of a frozen backbone, but passively selects static activations to train a linear classifier per target task, with no focus on source-domain retention. In contrast to these approaches, VisCoP addresses the setting of domain adaptation in generative VLMs, enabling them to learn domain-specific features without requiring updates to the pretrained encoder representations.

Domain adaptation in VLMs. Domain adaptation in VLMs has largely been achieved through data-centric strategies rather than through architectural changes [35]. Existing approaches typically leverage automated pipelines [36], [37] or closed-source VLMs [20], [38] to curate visual-instruction pairs from existing datasets in the target domain. Their adaptation strategy usually follows a multi-stage training scheme similar to LLaVA [39], where different VLM components are selectively trained at each stage. However, the choice of trainable components creates a trade-off between extracting domain-specific features and retaining pretrained knowledge. Training only lightweight connectors retains pretrained knowledge but limits domain-specific visual understanding, while training the vision encoder enables specialized visual understanding at the cost of catastrophic forgetting. VisCoP avoids this trade-off through the introduction of visual probes that extract domain-specific features from a frozen vision encoder, enabling adaptation without disrupting the pretrained visual representations.

Visual probing vs. visual compression and prompt tuning. Several approaches employ learnable tokens to bridge vision and language modalities [26], [27], [40] through architectures leveraging the Q-Former, Perceiver Resampler, or prompt tuning mechanisms. Q-Former [24] leverages learnable queries that cross-attend to representations from the final layer of the vision encoder, aggregating visual information into a reduced set of tokens for computational efficiency. Perceiver Resampler [25] operates similarly, aiming to compress the visual representations into a fixed number of learnable tokens. Prompt tuning methods [28], [30], [41] introduce learnable tokens to steer downstream reasoning, but operate solely on fixed encoder representations without enabling learning of new visual features. The visual probes proposed in VisCoP differ fundamentally, as they are designed to extract and learn domain-specific visual representations rather than to simply compress or condition on pretrained ones. This is enabled by their interaction with intermediate representations of the vision encoder, allowing the probes to extract domain-specific representations that are not propagated to the final representation of the pretrained vision encoder [9], [10].

3 Problem Formulation↩︎

Let \(\mathcal{S}\) denote the source domain, on which the vision-language model \(f_{\theta^0}\) has been pretrained, and let \(\mathcal{T}\) denote the target domain, the domain of interest for adaptation. The two domains differ in their underlying distributions (e.g., viewpoint, modality, or task), which causes \(f_{\theta^0}\) to perform poorly when directly applied to \(\mathcal{T}\).

Training supervision in these domains is provided as video-QA pairs \((v,q,a)\), where \(v\) is a video, \(q\) is an instruction or question, and \(a\) is the corresponding response. While \(f_{\theta^0}\) has been pretrained on samples \((v,q,a)\sim\mathcal{S}\), at adaptation time we only assume availability of target domain samples \((v,q,a)\sim\mathcal{T}\). The objective of domain adaptation is to update the pretrained parameters \(\theta^0\) to obtain \(\theta^\star\) that improves performance on domain \(\mathcal{T}\), while retaining performance on domain \(\mathcal{S}\). Formally, \[R_{\mathcal{T}}(\theta^\star) < R_{\mathcal{T}}(\theta^0) \quad \text{and} \quad R_{\mathcal{S}}(\theta^\star) \approx R_{\mathcal{S}}(\theta^0)\] where \(R_{\mathcal{D}}\) denotes the VLM’s expected autoregressive next-token prediction loss under domain \(\mathcal{D}\). In summary, our problem statement considers adaptation of a pretrained VLM to a novel domain using only video-QA pairs from that domain. The objective is to improve target-domain performance while minimizing catastrophic forgetting of source-domain capabilities. In the next section, we introduce our proposed method, which enables balanced domain adaptation under these constraints.

4 Method: Video Domain-adaptive VLM↩︎

Given a video input \(\mathbf{V} = \{\boldsymbol{I}_t\}_{t=1}^{T}\) consisting of \(T\) frames, the goal of the VLM is to generate the response corresponding to the input instruction in an autoregressive manner.

4.1 Preliminary↩︎

Existing VLMs for video representation learning [2], [3] consist of three standard components: (i) a vision encoder that maps visual inputs into a sequence of spatio-temporal tokens, (ii) a vision-language connector that projects the visual tokens to the embedding space of a language model, and (iii) an LLM that processes the projected visual tokens jointly with language tokens to enable multi-modal reasoning. For the input video \(\mathbf{V}\), each frame \(\boldsymbol{I}_t\) is processed independently by the vision encoder through a stack of \(L\) transformer layers. The visual tokens after the \(\ell\)-th layer are denoted as \[\mathbf{X}_t^{\ell} \in \mathbb{R}^{N \times d_v}, \quad \ell = 1, \dots, L\] where \(N\) is the number of spatial patch tokens per frame and \(d_v\) is the embedding dimension of the vision encoder. Concatenating these tokens over time yields \(\mathbf{X}^{\ell} \in \mathbb{R}^{(TN) \times d_v}\) which represents the sequence of spatio-temporal visual tokens at the \(\ell\)-th layer of the vision encoder. The final layer outputs \(\mathbf{X}^{L}\) are then projected to the language embedding space via a vision-language connector \(\mathcal{C}\) to obtain the visual embeddings used as input to the LLM \[\mathbf{E} = \mathcal{C}(\mathbf{X}^{L}) \in \mathbb{R}^{(T\tilde{N}) \times d_{\text{lm}}}\] where \(\tilde{N}\) is the number of visual tokens input to the LLM after spatial downsampling [3]. and \(d_{\text{lm}}\) is the embedding dimension of the LLM.

The VLM is then trained to optimize a standard autoregressive next token prediction loss. Specifically, given the visual embeddings \(\mathbf{E}\) and the tokenized QA pair \((\mathbf{Q}, \mathbf{A})\), we optimize the likelihood of predicting \(\mathbf{A}\) conditioned on the visual embeddings and the question \[P(\mathbf{A} \mid \mathbf{E}, \mathbf{Q}) = \prod_{j=1}^{\textrm{Len}} P_{\boldsymbol{\theta}}(\mathbf{a}_j \mid \mathbf{E}, \mathbf{Q}, \mathbf{A}_{<j})\] where \(\boldsymbol{\theta}\) are the trainable parameters of the VLM, \(\textrm{Len}\) indicates the token length of \(\mathbf{A}\), and \(\mathbf{A}_{<j}\) represents the subsequence of answer tokens preceding position \(j\).

For domain-adaptive post training of VLMs, finetuning the vision encoder of a pretrained VLM for a target domain \(\mathcal{T}\) often leads to overfitting on \(\mathcal{T}\) and catastrophic forgetting of the source domain [17]–[19]. To mitigate this trade-off, a domain-adaptive pathway is required that adapts the VLM to \(\mathcal{T}\) while retaining performance on \(\mathcal{S}\).

4.2 VisCoP: Vision Contextualized Probing↩︎

To capture the relevant visual context that would otherwise be lost by freezing the vision encoder, we propose Vision Contextualized Probing (VisCoP), a mechanism that augments the vision encoder with a compact set of learnable tokens, called visual probes, and an interaction module that acts as a semantic interface between the probes and intermediate visual representations, as illustrated in Figure [fig:method95diagram]. Unlike prior token-based approaches that reinterpret fixed encoder features  [24], [25], [28], VisCoP introduces a dedicated adaptation pathway that learns new visual representations from intermediate encoder layers while preserving the pretrained encoder parameters. In this section, we introduce how domain-adaptive VLMs are trained with VisCoP.

VisCoPaugments the frozen vision encoder of a VLM with a compact set of \(M\) learnable visual probes \(\mathbf{P}\in\mathbb{R}^{M\times d_v}\). The probes are trained to extract domain-specific spatio-temporal cues from intermediate representations of the vision encoder. To enable this extraction, a learnable interaction module \(\Phi^\ell\) inserted at each layer of the vision encoder conditions the probes on the hierarchical representations of the vision encoder at layer \(\ell\): \[\mathbf{P}^{\ell+1} \;=\; \Phi^{\ell}(\mathbf{P}^{\ell},\mathbf{X}^{\ell}).\] Concretely, \(\Phi^\ell\) is implemented as a vision-probe cross-attention between the visual embeddings and the probes at layer \(\ell\). Let \((\boldsymbol{W_q}, \boldsymbol{W_k}, \boldsymbol{W_v})\) be the projection matrices in \(\Phi^\ell\), then the probe update is \[\mathbf{P}^{\ell} \;=\; \mathrm{softmax}\!\left(\frac{\mathbf{P}^{\ell}\boldsymbol{W_q}^{\ell} (\mathbf{X}^{\ell}\boldsymbol{W_k}^{\ell})^\top}{\sqrt{d_v}}\right) (\mathbf{X}^{\ell} \boldsymbol{W_v}^{\ell}),\] Each \(\Phi^\ell\) is parameterized independently, enabling layer-specific aggregation of low- to high-level visual semantics. In contrast to the vision encoder of existing VLMs, which only learns spatial relationships through intra-frame self-attention, the visual probes attend to all frames in the video, enabling them to learn complex spatio-temporal relationships specific to video understanding. In some settings, such as robotic control, vision-probe cross-attentions are restricted to spatial tokens only.

After the final layer, the updated probes \(\mathbf{P}^{L}\) are projected to the language embedding space via a dedicated connector \(\mathcal{C}_{\text{probe}}\), \[\mathbf{Z}=\mathcal{C}_{\text{probe}}(\mathbf{P}^{L})\in\mathbb{R}^{M\times d_{\text{lm}}},\] and the VLM is trained with the standard autoregressive objective additionally conditioned on \(\mathbf{Z}\): \[P(\mathbf{A}\mid \mathbf{E},\mathbf{Q},\mathbf{Z}) = \prod_{j=1}^{\mathrm{Len}} P_{\boldsymbol{\theta}}\!\left(\mathbf{a}_j \mid \mathbf{E},\mathbf{Q},\mathbf{Z},\mathbf{A}_{<j}\right).\] Thus, the probes act as low-dimensional control knobs that bias learning toward domain-relevant structure and away from spurious artifacts. This is reinforced by applying updates through the probe connector, and through LoRA [42] updates in the LLM embedding space, which confine parameter changes to a low-rank, probe-defined visual subspace that preserves generalizable behavior while enabling targeted specialization.

5 Experiments↩︎

We evaluate VisCoPfor effective domain adaptation and minimal forgetting. Section 5.1 details the setup (architecture, training, metrics); Section [sec:experiments95targetsource95domains] reports results on egocentric, depth, and robotic-control targets; Section 5.3 presents ablations and representation analyses of the probes and interaction modules.

5.1 Experimental Setting↩︎

VLM Architecture. We consider a VLM architecture consisting of a SigLIP [10] vision encoder, Qwen 2.5 [11] LLM, and a 2-layer MLP vision-language connector, with all modules initialized from the pretrained weights of VideoLLaMA3 [3]. The embedding dimension of the vision encoder is \(d_v = 1152\), and the embedding dimension of the LLM is \(d_{\text{lm}} = 3584\). We refer to this pretrained model as the base VLM, and to models adapted to a target domain as expert VLMs. To adapt the base VLM to a target domain, we perform finetuning on the target domain with a learning rate of \(1\times10^{-5}\) for the LLM and vision-language connector, and a learning rate of \(2\times10^{-6}\) for the vision encoder (when trainable). The model is finetuned on 4 NVIDIA H200 GPUs for 3 epochs when adapting to video domains, or 2 epochs when adapting to robotic control domains.

VisCoP Details. By default, VisCoP  operates at every layer of the vision encoder and employs \(M=16\) visual probes unless otherwise stated. The visual probes are initialized from the normal distribution \(\mathcal{N}(0,0.02)\). Each interaction module \(\Phi^{\ell}\) is implemented as a multi-head cross-attention [21], and its weights are initialized from the self-attention weights of the vision encoder at layer \(\ell\). During domain adaptation, we freeze the vision encoder and update only the visual probes, interaction modules, vision–language connectors, and the LLM’s LoRA parameters. For adaptation to video understanding domains, we update the LLM using LoRA (\(r=16\)), while the entire LLM is updated when adapting to the robotic control domain. We keep the vision–language connector trainable throughout, as is standard in VLM training [4], [39].

Adaptation Metrics. We evaluate the domain adaptation of VLMs across two dimensions: (i) their “improvement" on the target domain \(\mathcal{T}\), and (ii) their”retention" on the source domain \(\mathcal{S}\). Improvement on the target domain is measured as the performance difference between the expert and base VLMs on target domain benchmarks; retention is the corresponding difference on source domain benchmarks. If \(\text{Acc}_\mathcal{D}\) denotes the average accuracy over all benchmarks within the domain \(\mathcal{D}\), then the metrics are computed by: \[\begin{align} \Delta_{\text{target}} &= \text{Acc}_{\text{target}}^{\text{expert}} - \text{Acc}_{\text{target}}^{\text{base}} \\ \Delta_{\text{source}} &= \text{Acc}_{\text{source}}^{\text{expert}} - \text{Acc}_{\text{source}}^{\text{base}} \end{align}\]

5.2 Source and Target Domains↩︎

The source domain \(\mathcal{S}\) is fixed throughout this paper: exocentric RGB videos of human actions reflecting the samples used to train generic VLMs for video representation learning. Our target domains \(\mathcal{T}\) deliberately shift the input distribution, and consist of (1) egocentric video understanding, (2) depth-modality video understanding, and (3) robotic control. All data (videos and instructions) in our chosen target domain benchmarks were not used in the pretraining of the base VLM [3]. We evaluate VisCoP’s adaptation to each target while measuring retention of source domain competencies: (i) when adapting to egocentric video, exocentric understanding should be preserved; (ii) when adapting to depth video, RGB understanding should be preserved; and (iii) when adapting to robotic control, human-action understanding should be preserved.

Training datasets. For ego and depth video understanding domains, we adapt using EgoExo4D [43], a large-scale multi-view dataset containing time-synchronized egocentric and exocentric videos of skilled human activities. We utilize a total of 24,688 videos from the keystep recognition subset to generate 74,064 video instruction pairs. These instructions are recaptioned from the instruction pairs provided in [44]. For the egocentric target domain, we adapt on 45,888 egocentric video-instruction pairs. For the depth target domain, we convert all exocentric RGB videos to depth using DepthAnythingV2 [45] while keeping the language instructions unchanged, yielding 28,176 depth instruction pairs.

We perform adaptation to the robotic control domain in both simulated and real-world robot environments. In the simulated environment, we leverage the training set of VIMA-Bench [46]. VIMA-Bench contains 17 object manipulation tasks with an action space comprising two 2D coordinates (for pick and place positions) and two quaternions (for rotation). Since the training set of VIMA-Bench lacks natural language instructions by default, we leverage the instruction pairs generated in LLaRA [47], resulting in 13,922 instruction pairs across 7,995 action trajectories. In the real-world environment, we collect a dataset using a 6-DoF xArm 7 robot arm deployed in a tabletop manipulation setting. This dataset, which we refer to as xArm-Det, contains 1,007 instruction pairs depicting novel objects and spatial configurations not present in simulation. During adaptation, we train jointly on VIMA-Bench and xArm-Det, resulting in a total of 14,929 instruction pairs. The large-scale simulated data enables the model to learn manipulation skills, while xArm-Det exposes the model to our novel robot environment. Illustrations of our real-world robot environment and examples from VIMA-Bench are provided in the Appendix.

5pt

5.2.1 Egocentric Video Understanding↩︎

Target and source benchmarks. For evaluation on the target domain, we evaluate on the Ego-in-Exo Perception [44] and EgoSchema [48] benchmarks. Ego-in-Exo Perception is derived from EgoExo4D and comprises 3,881 video question-answer (video-QA) pairs spanning four categories: action understanding (Action Und.), task-relevant region understanding (Task Regions), human-object interactions (HOI), and hand identification (Hand Ident.). Because it is derived from EgoExo4D, Ego-in-Exo Perception can be evaluated from either the egocentric or the exocentric viewpoint. For the ego target domain experiments, we report results using the egocentric videos, denoted as Ego-in-Exo Perception (Ego RGB). EgoSchema consists of 5,031 egocentric video-QA pairs derived from Ego4D [49].

For evaluation on the source domain, we select benchmarks that measure exocentric video understanding capability. Specifically, we evaluate on the NeXTQA [50], VideoMME [51], and ADL-X [37] benchmarks. NeXTQA and VideoMME are general-purpose video-QA benchmarks built from web-scraped videos (e.g., from YouTube), with 8,564 QA pairs in NeXTQA and 2,700 QA pairs in VideoMME. ADL-X is a video-QA benchmark built from videos of activities of daily living, it contains a total of 10,561 multiple-choice questions (ADL-X MCQ) and 1,862 video description questions (ADL-X Desc) derived from various activities of daily living datasets [52]–[55].

Results. Table [tbl:tab:ego2exo95sota] reports results of adaptation to the egocentric viewpoint. Training only the vision-language connector or the connector together with LLM LoRA adapters does not lead to effective adaptation to the target domain (\(\Delta_{\text{target}} < 1\)). Updating all three modules (connector, vision encoder, and LLM) improves performance on the target domain by \(\Delta_{\text{target}} = \textcolor{codegreen}{+0.57}\), but the large number of trainable parameters results in forgetting on the source benchmarks (\(\Delta_{\text{source}} = \textcolor{negativeorange}{-0.50}\)). In contrast, updating the connector and vision encoder alone slightly improves performance on the target domain and does not lead to forgetting on the source domain. These results highlight that the core difficulty of domain adaptation in existing VLMs arises from the necessity of updating the vision encoder to learn domain-specific visual representations, which inevitably leads to forgetting of pretrained knowledge. Our proposed VisCoP achieves the strongest adaptation performance, with the highest improvement on the target domain (\(\Delta_{\text{target}}=\textcolor{codegreen}{+3.5}\)) while simultaneously maintaining retention on the source benchmarks (\(\Delta_{\text{source}}=\textcolor{codegreen}{+1.8}\)). Interestingly, VisCoPnot only avoids catastrophic forgetting but also improves performance on some source benchmarks (e.g., ADL-X). We attribute this positive transfer to a multi-axis domain shift: although source and target differ in viewpoint (exocentric vs. egocentric), their action distributions overlap. ADL-X, while exocentric, encapsulates activities of daily living that closely aligns with the EgoExo4D action distribution, enabling beneficial cross-domain generalization.

5pt

5.2.2 Depth Video Understanding↩︎

Target and source benchmarks. For evaluation on the target domain, we evaluate on the Ego-in-Exo Perception [44] benchmark. In the depth-adaptation setting, we train on depth maps of exocentric videos extracted with DepthAnythingV2 [45] and evaluate on exocentric depth videos following [44], denoted Ego-in-Exo Perception (Exo Depth). For the source domain, we use RGB benchmarks of exocentric understanding: Ego-in-Exo Perception (Exo RGB), NeXTQA, VideoMME, and ADL-X.

Results. We present the results for adaptation to the depth modality in Table [tbl:tab:depth2rgb95sota]. In contrast to the results on egocentric viewpoint adaptation, we find that all training strategies achieve improvements on the target domain, reflecting the disparity of the visual embedding space between the depth and RGB modalities. We find that this disparity leads to different behavior across training strategies. Jointly updating the vision encoder and the vision-language connector preserves source performance for egocentric adaptation but causes severe catastrophic forgetting under depth adaptation (\(\Delta_{\text{source}}=\textcolor{negativeorange}{-3.87}\)). This arises from the substantial encoder updates required to bridge RGB and depth, which overwrite RGB representations. In contrast, VisCoP preserves RGB features and source performance while achieving the largest target domain gains (\(\Delta_{\text{target}}=\textcolor{codegreen}{+19.27}\)).

5pt

5.2.3 Robot Control↩︎

Target and source benchmarks. For evaluation on the target domain, we consider both simulated and real-world robotic environments. In simulation, we use the evaluation set of VIMA-Bench [46], which organizes tasks into three levels of difficulty: L1 (Object Placement), where all objects have been seen during training; L2 (Novel Combination), where objects seen during training appear in new pairings or contexts; and L3 (Novel Objects), where objects entirely unseen during training are introduced. Together, these levels measure generalization from familiar training conditions to progressively more challenging distributions. In the real-world setting, we evaluate on three tabletop manipulation tasks: T1) Place the {object} on the plate, T2) Pick up and rotate {object} by {angle}; and T3) Move all {color} objects onto the plate. Examples of each task and a list of objects used is provided in the Appendix. On these robotic control benchmarks, the reported accuracy corresponds to the success rate across all robot manipulation tasks. For source domain evaluation of VLMs trained on both real and simulated robotic environments, we use the human-activity video benchmarks Ego-in-Exo (Exo RGB), NeXTQA, VideoMME, and ADL-X.

8pt

Results. The results of adaptation to the robotic control domain are presented in Table [tbl:tab:robot2human95simulation]. The base VLM demonstrates weak performance on all robot control tasks, as its pretraining distribution lacks action trajectories (i.e., instruction data mapping from visual observations to robot actions). This lack of pretraining results in 0% accuracy across all levels of VIMA-Bench, and is consistent with prior works [47]. This highlights the extreme domain gap both in the visual space (robot observations vs. human videos) and in the language space (control actions vs. linguistic outputs) between the source and target domains. Similarly to the depth adaptation setting, we find that training the vision encoder improves performance on the target domain, but results in the worst source domain retention (\(\Delta_{\text{source}}=\textcolor{negativeorange}{-8.87}\)) of all robot control experts. In contrast, our proposed VisCoP achieves the best performance on the target domain (\(\Delta_{\text{target}}=\textcolor{codegreen}{+67.82}\)) while retaining the most source domain knowledge (\(\Delta_{\text{source}}=\textcolor{negativeorange}{-4.58}\)) compared to other experts, demonstrating the effectiveness of our method even when the gap between the source and target domains is very large. Also note that VisCoP operates on per-timestep images in these experiments; thus the visual probes consume the same visual tokens as the vision encoder, suggesting they extract domain-specific representations more effectively than the base encoder.

We further evaluate adaptation in the real-world setting using the xArm-Det dataset in Table [tbl:tab:robot2human95realworld]. We consider a transfer setting, where the experts are trained only on VIMA-Bench and directly evaluated on xArm-Det, and the setting where the experts are jointly trained on both VIMA-Bench and xArm-Det. In both cases, our proposed VisCoP outperforms the vision encoder trained experts on target domain adaptation as well as source domain retention.

5.3 Model Diagnosis and Analysis↩︎

In this section, we motivate the design of VisCoP through a diagnostic study, and perform an analysis on the visual representations it learns. We investigate the number of visual probes, as well as the placement of interaction modules within the vision encoder. We then analyze the domain-specific representations learned by VisCoP through t-SNE and attention visualizations.

Alternatives to learnable queries.Table 3 compares VisCoPagainst alternative adaptation strategies. Visual Probes Only (VP) trains only visual probes with their vision-language connector (\(\mathcal{C}_{probe}\)) without any interaction modules. Partial Encoder Training (Last-4) makes the final four layers of the vision encoder trainable. QFormer-Style Compression uses visual probes with interaction modules only at the vision encoder’s final layer, mimicking Q-Former’s compression approach [24]. Notably, several of these alternatives underperform even the frozen base VLM on target tasks. We attribute this to their reliance on the pretrained visual pathway: they can recombine existing representations but struggle to introduce novel cues (e.g., egocentric or depth structure) that are absent from those representations. Full fine-tuning is more competitive on the target precisely because it can overwrite this pathway, at the cost of source-domain performance. VisCoPoccupies the middle ground, introducing a parallel probing pathway that decouples target-domain learning from the frozen pretrained pathway, and thereby learns new cues without overwriting the source representation. Model Tailor [56] performs post-hoc domain adaptation by fusing parameter updates from a fine-tuned VLM back into the base VLM, modifying only the LLM parameters and leaving the vision encoder untouched. Training with QFormer-Style compression or only training with visual probes (VP) underperforms compared to VisCoP, indicating the importance of probe interactions at intermediate layers of the vision encoder to learn domain-specific features across multiple levels of abstraction. Similarly, training only the last four layers of the vision encoder, or training it with LoRA, also underperforms, highlighting that even partial parameter updates fail to capture domain-specific signals as effectively as VisCoP’s visual probes. Model Tailor also falls short in this setting, suggesting that approaches which do not leverage intermediate vision encoder representations struggle to learn domain-specific visual features.

1pt

Effect of interaction module scope. A key design choice in VisCoP is the scope of visual features that the probes interact through the interaction modules. In Table 2, restricting probes to spatial-only or temporal-only interactions yields weaker adaptation performance, while full spatio-temporal interaction achieves substantially larger gains (\(\Delta_{\text{target}}{=}\textcolor{codegreen}{+3.53}\)). This result indicates that domain-relevant signals emerge from the joint spatio-temporal structure of video representations, and that enabling probes to integrate information across both dimensions is critical for effective video domain adaptation.

6pt

Computation overhead of VisCoP. VisCoP  introduces only modest computational overhead relative to the base VLM, introducing only 2% more parameters than the base VLM. In Table ¿tbl:tab:computation95overhead?, we compute the average VRAM usage and latency during inference on the Ego-in-Exo Perception benchmark, as well as the total parameter count. We find that VisCoP increases VRAM usage by 3.4GB and adds just 0.013s of inference latency in visual feature extraction (Before LLM). Interestingly, the inference latency of our model is lower than that of the base VLM. We attribute this to the base VLM producing longer, less focused responses, which increases total decoding time.

Ablations on probes and interaction modules. We study the effect of the number of visual probes and the placement of interaction modules (Figure [fig:ablation95numprobes], Figure [fig:ablation95layerpos]). Probes consistently improve performance over the base VLM, with the best trade-off at 16 probes (\(\Delta_{\text{target}}=\textcolor{codegreen}{+3.53}\), \(\Delta_{\text{source}}=\textcolor{codegreen}{+2.12}\)); larger probe counts offer no further gains and can reduce performance due to redundancy. For interaction modules, applying them at every encoder layer yields the strongest adaptation, while sparse placement (e.g., every 6 or 9 layers) provides weaker or inconsistent gains. These results highlight the importance of using a small number of probes with dense access to intermediate features.

Visualizing attention in domain-adapted VLMs In Figure 4 (a), we analyze attention maps of various VLM adaptation strategies to assess how different components capture domain-specific visual features. For both the frozen and trainable vision encoders, we visualize attention using attention rollout [57], for VisCoP we visualize the attentions of the visual probes, averaged across all probes. The frozen vision encoder fails to focus consistently on relevant regions under the experimented domains, reflecting its limited ability to capture domain-specific features. The trained vision encoder yields sharper attention on the relevant regions, indicating its ability to learn domain-specific features, albeit at the cost of catastrophic forgetting of the source domain as shown in Section 5.2. In contrast, the visual probes of VisCoP have a sharp focus on the task-relevant regions, despite the vision encoder being frozen. This indicates that the probes alone are able to extract the domain-specific visual features necessary for adaptation. In Figure 4 (b), we visualize the attention of generated language tokens to visual embeddings. We find that VisCoP correctly responds to the query, with more focus given to tokens corresponding to relevant objects.

Learning domain-specific representations. Figure 4 (c) compares t-SNE embeddings of source and target domains across different VLMs. Circles represent individual samples, and ellipses denote 95% confidence regions of fitted 2D Gaussians. For the egocentric and depth target domains, each source-target pair corresponds to time-synchronized videos of the same action. For the robot target domain, pairs correspond to pick-and-place actions performed by humans. Ideally, the embeddings of paired samples should lie closer together in the embedding space, reflecting alignment across the source and target domains. We quantify this using two metrics: the Bhattacharyya distance (BD) computed between the Gaussians fitted to each domain, and the per-sample distance (PSD), defined as the mean Euclidean distance between paired embeddings across domains. We observe that the visual probes of VisCoPlearn stronger alignment between the source and target domains. Quantitatively, VisCoP yields the smallest Bhattacharyya distance and per-sample distance between paired source and target embeddings across all three target domains, indicating tighter cross-domain alignment than both the base VLM and the frozen-encoder baseline. The effect is most pronounced under the largest shifts (depth and robot control), where the frozen encoder’s paired embeddings remain widely separated while the probes draw them into closer correspondence.

Generalization beyond visual domains. The target domains considered thus far all shift the visual input distribution. To test whether VisCoP generalizes to non-visual shifts, we adapt to audio understanding, where audio is represented as a spectrogram, using video understanding as the source domain. Following OneLLM [58], we repurpose the vision encoder into a universal encoder (UE) that processes both audio and video, and instantiate VisCoP on top of it. We adapt and evaluate on MusicAVQA Li2022musicavqa?, treating its audio-centric questions as the target domain and its video-centric questions as the source. As shown in Table 3, VisCoP and the UE + LLM (LoRA) baseline are comparable on the target domain (\(\Delta_{\text{target}}=\textcolor{codegreen}{+11.9}\) vs. \(\textcolor{codegreen}{+11.5}\)), but VisCoP nearly eliminates source-domain forgetting (\(\Delta_{\text{source}}=\textcolor{negativeorange}{-0.7}\) vs. \(\textcolor{negativeorange}{-5.3}\)). This demonstrates VisCoP is not specific to visual shifts: it extracts domain-relevant cues while preventing forgetting even when the target modality is non-visual.

6 Conclusion↩︎

We introduced VisCoP, a mechanism that extracts domain-specific visual features through probing of a frozen vision encoder to enable effective domain adaptation in VLMs and prevent catastrophic forgetting. VLMs equipped with VisCoPachieve superior target domain performance, while maintaining strong source domain capabilities across cross-view, cross-modal, and cross-task adaptation scenarios. We will release all code, models, and evaluation protocols to facilitate future research.

Acknowledgments↩︎

This work was supported in part by the National Science Foundation (IIS-2245652) and the University of North Carolina at Charlotte. Computational resources were provided by the NSF National AI Research Resource Pilot (NAIRR-240338) and the NCShare initiative. We would like to thank Hieu Le and other members of the Charlotte Vision Lab for their valuable insights and discussions.

7 Appendix↩︎

7.1 Details of Simulated Robot Control Experiments↩︎

For our robot control simulation experiments, we use the VIMA-8K instruction set generated from the VIMA dataset, following [47]. Figure 5 illustrates representative examples of training tasks - simple visual manipulation (top row) and rotation (middle row).

For evaluation, we adopt the three levels of generalization defined in VIMA-Bench [46]: - L1 (Placement Generalization): tasks where the object placements differ from those seen in the training set. - L2 (Combination Generalization): tasks requiring new combinations of objects not paired during training. - L3 (Novel Object Generalization): tasks involving completely unseen objects that were not present in the training data.

7.2 Details of Real-World Robotics Experiments↩︎

We provide additional details of the experiments conducted in our novel robot environment, including the setup, data collection, and evaluation protocol.

7.2.1 Real-Robot Setup↩︎

Our setup consists of an xArm7 robotic arm with a gripper, tabletop, and an Intel RealSense D455 third person camera mounted in front of the arm to collect observations as seen in Figure 6. The action space of the end effector is two 2D cartesian coordinates representing the pick and place poses, and two quaternions for rotations similar to  [46]. We evaluated the effectiveness of our method mainly on three robot manipulative tasks:

T1 : Place the {object} on the plate. T2 : Pickup and Rotate the {object} by {degree} degrees. T3 : Move all the {colour} objects into the plate.

We uniformly sample {object} from a set of 10 toys : green apple, carrot, eggplant, banana, corn, grape, green pepper, tomato, strawberry, cucumber, clementine, and lemon. For T2, the target rotation angle is randomly selected from {30°, 45°, 60°, 90°, 180°}. For T3, the variable colour is chosen from four categories: {red, orange, yellow, purple}

7.2.2 Real-Robot data collection↩︎

We collected 1,007 images with resolution 640 x 640 of a real-robot setup with multiple objects scattered on the table. A one-shot object detection using Owlv2 [59] is applied to extract bounding boxes for each object. Based on these images and their corresponding bounding box annotations, we generate task instructions following the xArm-Det style similar to [47].

7.2.3 Evaluation protocol↩︎

All three tasks are evaluated under two settings: zero-shot and joint training. The observation space is illustrated in Figure 7. In zero-shot setting, we use the models trained on VIMA-8K where as in the joint training setting, we finetune VLM jointly on both VIMA-8K and collected xArm-Det data. For each task, we conduct 20 trials with objects placed at random initial positions on the table. Each episode is limited to a maximum of 4 steps. We report the average success rate across all trials as performance metric and below are the success criteria for each task that we follow :

T1 : A trial is considered successful if at least 50% of the object lies inside the plate.

T2 : A trail is successful by visually verifying whether the object has been rotated to the specified target angle.

T3 : A trial is successful only if all objects of the specified color are moved into the plate; otherwise, it is a failure.

7.3 Expanded Experimental Results↩︎

7.4 Is VisCoPsimply regularizing drift from the source model?↩︎

A natural hypothesis is that VisCoP’s source retention stems from limiting drift away from the base VLM, which dedicated retention objectives could achieve directly. We test this on the depth target domain with two retention-focused baselines: (i) source-data replay, adding 5%, 10%, and 25% source-domain samples to the adaptation data, and (ii) KL regularization, adding a KL divergence between the output distributions of the adapted and base VLMs. As reported in Table 4, replay improves source retention over target-only training but remains below the base VLM even at 25% source data. KL regularization is a stronger baseline, improving both target accuracy and source retention. Yet VisCoP outperforms both on each axis, indicating that retention regularization alone does not account for its gains: the visual probes provide a more effective adaptation mechanism by exposing target-domain cues from intermediate representations without updating the pretrained encoder.

7.4.1 Varying number of visual probes↩︎

In Figure 8, we examine the effect of varying the number of visual probes across all three target domains. We verify that 16 probes provides a fair overall tradeoff across all target domains, retaining the most source-domain performance across all domains while remaining near the top in egocentric and depth adaptation, and performing substantially better than larger probe counts in the robotics setting. We attribute this to the fact that robotics tasks require the model to integrate visual cues with precise action semantics—when the number of probes becomes large, the additional probe signals tend to dominate the representation space, causing the LLM to overcommit to action-execution patterns and generate robotic command-like outputs even when inappropriate. In contrast, a smaller probe set provides focused domain-specific visual information without overwhelming the pretrained multimodal alignment, resulting in stronger performance and better retention.

7.4.2 ADL-X Benchmark↩︎

In this section, we present expanded results on the ADL-X benchmark across three target domains: ego-video understanding Table  5, depth-video understanding Table  6, and robot control Table 7. In addition, we provide comprehensive source-domain results for the real-world domain expert Table  ¿tbl:tab:robot2human95realworld95domain95expand?, as well as detailed ablation studies Table  ¿tbl:tab:ablations95extend?.

For the ADL-X description benchmark, we restrict evaluation to the Charades Description  [37].

5pt

5pt

5pt

8pt

5pt

7.5 Qualitative results↩︎

In this section, we provide qualitative comparisons of three models—Base VLM, trained vision encoder (VL-C+VE), and VisCoP  across the three domain experts ego-video understanding, depth-video understanding, and robot control. Figures 9, 10, and 11 show representative examples from each expert. Each figure shows representative samples from both the target domain and the source domain.

We demonstrate that VL-C+VE successfully adapts the Base VLM to the target domain, enabling correct predictions. However, this adaptation comes at the expense of source-domain performance, where VL-C+VE frequently makes mistakes. In contrast, VisCoP  achieves the best of both: it adapts effectively to the target domain while simultaneously retaining strong performance on the source domain, thereby avoiding catastrophic forgetting.

We also provide qualitative comparisons of video descriptions on the source domain (ADL-X) using the ego-video understanding expert and the depth-video understanding expert. As shown in Figure 12 and Figure 13, our method generates descriptions that are both more accurate and more detail-oriented compared to the trained vision encoder (VL-C+VE). While VL-C+VE can adapt to the target domain, on the source domain it often introduces hallucinated details. In contrast, VisCoP  preserves correctness, capturing the scene, actions and object interactions without hallucination.

References↩︎