1 Introduction↩︎

Spatial understanding serves as the foundation for real-world reasoning and interaction, acting as a pivotal technology for critical applications such as robotic navigation and embodied intelligence. The rapid advancement of Multimodal Large Language Models (MLLMs)[1]–[6] has brought substantial breakthroughs to this domain. By bridging the general reasoning capabilities of Large Language Models (LLMs) with visual or point-cloud encoders [7], [8], MLLMs facilitate sophisticated spatial perception and reasoning within complex environments.

Recently, numerous studies have explored incorporating knowledge from pre-trained foundation models (e.g., VGGT [9], WAN [10]) into MLLMs [11]–[14]. Typically, this paradigm involves leveraging foundational features as auxiliary inputs or aligning the latent representations of MLLMs with foundation models through distillation or feature injection. Such a paradigm has proven effective in augmenting MLLMs with domain-specific expert knowledge to boost spatial awareness. Despite their success, existing efforts are predominantly restricted to exploiting prior knowledge from a single expert, while overlooking the potential of fusing diverse knowledge from multiple foundation models. Intuitively, different foundation models capture distinct priors shaped by their varied pre-training data and objectives, making it highly desirable to aggregate their respective advantages in MLLMs. Therefore, how to synergistically harmonize these complementary strengths from distinct models remains a significant, yet unresolved challenge.

In this paper, we seek to explore two questions that form our core motivations: (i) Do different foundation models exhibit distinct specializations across various spatial understanding tasks when serving as priors for MLLMs? (ii) How can we synergistically integrate the expert knowledge from multiple foundation models into MLLMs? Regarding the first question, we conduct comprehensive empirical studies by integrating the priors of different foundation models [9], [10], [15]–[17] into MLLMs across a wide range of spatial understanding tasks [18]–[23]. Our results reveal that these models contribute differently to specific tasks, indicating a strong complementarity among diverse visual priors and underscoring the necessity of multi-model integration. Based on these findings, a straightforward solution to the second question would be a naive ensemble of multiple priors. However, such an approach suffers from several critical drawbacks: First, the computational overhead becomes prohibitive, as extracting priors from each foundation model requires an independent forward pass during inference. Second, since different visual priors inherently exhibit distinct strengths and specializations, a naive ensemble not only fails to prioritize the most relevant prior knowledge for a given task, but also risks confusing the MLLM due to the distribution disparities among different foundation models.

To address these challenges, we propose ViPS (illustrated in Figure 1), a novel multi-model prior framework designed to fully unleash the potential of multiple Visual Priors in MLLMs for Spatial understanding. ViPS harmoniously integrates priors from multiple models into the MLLM by introducing an Efficient Prior Proxy and a Dynamic Prior Fusion mechanism. Specifically, the Efficient Prior Proxy is designed to generate multiple foundational priors without requiring multiple forward passes of foundation models. Instead of directly deploying all foundation models independently, we employ one base model alongside several lightweight proxies to estimate the priors of the foundation models and align the proxy outputs with the ground-truth priors during training. This design is motivated by the insight that different foundation models often share common low- and mid-level visual and geometric features. Therefore, a robust base model providing sufficient representations is enough, from which other distinct priors can be effectively distilled using lightweight proxies. Furthermore, the Dynamic Prior Fusion is proposed to achieve harmonious and context-aware prior selection from proxies. To this end, we first generate dynamic weights based on the input task, and then use these weights to aggregate the priors from different foundation models into a fused multi-expert prior, which is subsequently injected into the MLLM. Additionally, a set of zero-initialized convolutional layers is applied before aggregation to ensure that diverse priors are harmoniously fused. Extensive experiments demonstrate that ViPS achieves state-of-the-art performance across diverse spatial reasoning and spatial understanding benchmarks.

To summarize, we make the following contributions: First, we conduct an empirical study on integrating diverse foundation models into MLLMs, revealing that different visual priors exhibit distinct specializations across various tasks. Second, we propose ViPS, a novel multi-model prior framework for spatial understanding, which features an Efficient Prior Proxy to generate multi-model priors with minimal overhead, and a Dynamic Prior Fusion mechanism for harmonious and context-aware prior integration. Third, extensive experiments demonstrate that ViPS achieves new state-of-the-art performance across multiple spatial reasoning and spatial understanding benchmarks.

2 Method↩︎

2.1 Empirical Study on Diverse Model Priors↩︎

2.1.0.1 Preliminary.

Our work focuses on spatial understanding based on MLLMs. Formally, given a visual input comprising a sequence of frames (e.g., a video clip or multi-view images of a scene) \(\mathcal{V} = \{v_1, v_2, \dots, v_T\} \in \mathbb{R}^{T \times H \times W \times 3}\), alongside a corresponding textual task instruction \(\mathcal{X}_{text} = \{x_1, x_2, \dots, x_N\}\), the objective of the MLLM is to generate a coherent and accurate textual response \(\mathcal{Y} = \{y_1, y_2, \dots, y_M\}\) that adequately answers the query by reasoning over the holistic spatial structures of the scene. In standard video-based MLLM architectures (e.g., LLaVA [2], Qwen2-VL [1]), the visual sequence \(\mathcal{V}\) is first processed by a generic vision encoder \(\Phi_{vis}\), producing visual embeddings \(F_{vis} = \Phi_{vis}(\mathcal{V})\). A modality projector \(\mathcal{P}\) then aligns these embeddings into the text semantic space. The LLM, denoted as \(f_{LLM}\), takes the concatenated tokens of the projected visual features and the text instruction to auto-regressively predict the response: \[P(\mathcal{Y} \mid \mathcal{V}, \mathcal{X}_{text}) = \prod_{i=1}^{M} P \left( y_i \mid \mathcal{P}(F_{vis}), \mathcal{X}_{text}, y_{<i}; \theta \right),\] where \(\theta\) encapsulates the learnable parameters of the system. To bridge the gap in 3D spatial awareness, recent approaches explicitly introduce external foundation model priors. Given a pre-trained visual expert \(\mathcal{E}\) (e.g., encoders from foundation models like VGGT), it extracts domain-specific prior features \(F_{prior} = \mathcal{E}(\mathcal{V})\). The LLM leverages this supplementary knowledge to enhance generation, yielding \(P(\mathcal{Y} \mid F_{vis}, F_{prior}, \mathcal{X}_{text})\). While integrating a single prior model provides distinct utility, relying exclusively on an individual \(\mathcal{E}\) inevitably limits the representation capacity, forming the motivation for our extensive empirical investigations.

2.1.0.2 Key Finding and Motivation.

As highlighted in the introduction, existing efforts are restricted to exploiting the priors of a single model, overlooking the potential of fusing diverse knowledge from multiple foundation models. Fundamentally, different foundation models encapsulate distinct representations and spatial semantics, thereby endowing them with disparate advantages when serving as priors for MLLMs. To demonstrate this, we select a diverse set of foundation models for evaluation, including VGGT [9], DepthAnything3 [15], TraceAnything [16], Wan2.1 [10], and RADIO [17]. First, we conduct a prior analysis on these diverse models. As illustrated in Figure 2 (Left), the visualization demonstrates that priors extracted from distinct foundation models cluster into disparate regions within the latent space. Furthermore, the activation heatmaps in Figure 2 (Right) reveal that different models focus on varying structural and semantic cues within the identical scene. These results validate that different foundation models indeed encapsulate distinct representations and spatial cues due to their diverse training paradigms. Subsequently, we explore their relative advantages across a wide range of spatial understanding tasks when serving as priors for MLLMs (See 6.3 for more details). The results are shown in Figure 3. From a column-wise perspective, the best-performing model varies across different sub-tasks. From a row-wise perspective, each specific model achieves its optimal performance on disparate sub-tasks. These results demonstrate that no single foundation model achieves universal dominance; instead, different models exhibit distinct strengths and specializations. The highly complementary nature of these diverse visual priors directly motivates our core objective: to efficiently extract and adaptively harmonize distinct prior knowledge from multiple foundation models for comprehensive spatial reasoning.

2.2 Proposed Framework: ViPS↩︎

Building upon the above insights, we introduce ViPS (Visual Priors for Spatial understanding), a novel framework designed to fully unleash the potential of leveraging diverse prior models in MLLMs. The overall framework is illustrated in Figure 4. ViPS first processes the input visual sequence \(\mathcal{V}\) through a standard vision encoder \(\Phi_{vis}\) [24], producing the base visual embeddings \(F_{vis}\). Instead of relying on a single expert or naively ensembling multiple heavy models, ViPS seamlessly integrates comprehensive prior knowledge through two pivotal mechanisms: Efficient Prior Proxy and Dynamic Prior Fusion.

Specifically, the Efficient Prior Proxy utilizes a base foundation model alongside lightweight MLPs to efficiently estimate the prior features of various external models, generating a set of diverse prior representations \(\{F^1_{prior}, F^2_{prior}, \dots, F^K_{prior}\}\). Subsequently, these representations are fed into the Dynamic Prior Fusion part, which first applies independent, zero-initialized convolutional layers [25] to each prior branch and then employs a context-aware weighting mechanism, guided by the final token of the input instruction \(\mathcal{X}_{text}\), to dynamically compute fusion weights. Finally, the harmoniously fused multi-expert prior \(\hat{F}_{prior}\) is incorporated into the MLLM, empowering the LLM to leverage the most relevant visual knowledge tailored to the specific spatial reasoning task. We detail the formulations of these two components below.

2.2.0.1 Efficient Prior Proxy.

Our framework aims to effectively integrate highly complementary priors from various foundation models for comprehensive spatial understanding. Directly extracting visual features from a set of \(K\) distinct foundation models, denoted as \(\{\mathcal{E}_1, \mathcal{E}_2, \dots, \mathcal{E}_K\}\), conventionally requires \(K\) independent forward passes, i.e., computing \(\mathcal{E}_k(\mathcal{V})\) for \(k \in \{1, \dots, K\}\). This paradigm introduces a severe computational bottleneck, as the inference latency and memory footprint scale linearly with the number of integrated priors. To decouple the computational overhead from the integration of multiple models while effectively preserving the rich diversity of visual prior knowledge, we propose the efficient prior proxy mechanism. Specifically, we use a single robust vision encoder as the base model \(\mathcal{E}_{base}\) to extract a unified foundational representation: \[F_{base} = \mathcal{E}_{base}(\mathcal{V}) \in \mathbb{R}^{S \times D_{base}},\] where \(S\) and \(D_{base}\) denote the sequence length and channel dimension of the visual tokens, respectively. In our experiments, the base model is directly initialized with the encoder of one of the foundation models (we present experiments using encoders from different foundation models as the base model in the appendix).

To approximate the distinct knowledge of the \(K\) targeted foundation models, we instantiate a set of \(K\) lightweight proxy networks. Each proxy \(\phi_k(\cdot)\) is implemented as a simple Multi-Layer Perceptron (MLP) designed to extract specific prior semantics (e.g., depth cues, geometric boundaries, or detailed semantics) from the shared foundational feature \(F_{base}\). The representation for the \(k\)-th prior is uniformly formulated as: \[F^k_{prior} = \phi_k(F_{base}) \in \mathbb{R}^{S \times D_{prior}},\] where \(D_{prior}^{k}\) is the feature dimension for the \(k\)-th prior feature.

To guarantee the fidelity of these estimated prior representations, we explicitly supervise the output of prior proxies during the training phase. Specifically, the features extracted by the \(K\) distinct foundation models are utilized as the ground-truth targets, denoted as \(F^k_{gt} = \mathcal{E}_k(\mathcal{V})\). We apply an \(L_2\) loss to enforce strict alignment between the proxy outputs \(F^k_{prior}\) and their corresponding ground-truth priors \(F^k_{gt}\): \[\mathcal{L}_{alignment} = \sum_{k=1}^K \big\| F^k_{prior} - F^k_{gt} \big\|_2^2.\]

The viability of this proxy-based estimation stems from the intuition that different foundation models inherently share substantial low- and mid-level visual semantics; thus, a robust base representation contains sufficient foundational knowledge from which distinct, high-level prior semantics can be efficiently derived via shallow proxies. Given that the computational cost of a shallow MLP \(\phi_k\) is negligible compared to a full foundation model, our framework can scale to an arbitrary number of diverse priors without linearly increasing inference overhead, while ensuring that the estimated visual priors remain accurate through the alignment loss.

2.2.0.2 Dynamic Prior Fusion.

With the diverse prior representations \(\{F^1_{prior}, \dots, F^K_{prior}\}\) efficiently established, the remaining challenge is seamlessly integrating them into the MLLM. Directly adding these distinct and heterogeneous priors fails to prioritize the most effective priors for the specific input task, and can overwhelm the language model, consequently degrading overall performance. To achieve harmonious and context-aware integration, we propose the Dynamic Prior Fusion mechanism.

Accordingly, we design a dynamic weighting mechanism to selectively focus on the precise knowledge required by the given task. Specifically, we extract the representation of the final token of the text query, denoted as \(x_{last}^{text}\), which naturally encapsulates the aggregated semantics of the entire instruction due to the causal mechanism of the LLM. This context vector is passed through an MLP to compute a set of \(K\)-dimensional logits \(z \in \mathbb{R}^K\): \[z = \text{MLP}_{weight}(x_{last}^{text}).\] A Softmax activation is subsequently applied to \(z\) to obtain the normalized fusion weights \(w \in [0, 1]^K\). These weights will be utilized to dynamically combine the diverse priors. However, our experiments show that directly doing so fails to yield progressive performance improvements. We attribute this phenomenon to the disparate prior distributions originating from different source models, which tend to confuse the MLLM during the early stages (see Section 2.1 and Section 3.4). To overcome this issue and enable progressive prior injection, we apply an independent zero-initialized convolutional layer, denoted as \(\text{ZeroConv}_k(\cdot)\), to each proxy branch prior to the weighted fusion. This strategy ensures that the outputs of the convolution are initially zero, preserving the original MLLM representations and preventing the prior perturbations from disrupting early training, while gradually scaling up as the MLLM progressively learns from the diverse prior knowledge. The processed prior \(\tilde{F}^k_{prior}\) is computed as: \[\tilde{F}^k_{prior} = \text{ZeroConv}_k(F^k_{prior}).\]

Finally, the context-aware weights are utilized to linearly combine the aligned priors, yielding the overall harmonized multi-expert prior \(\hat{F}_{prior}\): \[\label{eq:prior95fusion} \hat{F}_{prior} = \sum_{k=1}^K w_k \cdot \tilde{F}^k_{prior}.\tag{1}\] This fused representation \(\hat{F}_{prior}\), which dynamically captures the most pertinent visual prior requested by the query, is directly added element-wise to the corresponding projected visual embeddings \(\mathcal{P}(F_{vis})\) (i.e., the image tokens) of the MLLM. In this way, Dynamic Prior Fusion seamlessly integrates diverse priors into the MLLM, adaptively tailoring knowledge selection to the input task while leveraging zero-initialization to ensure progressive learning without disrupting early training.

It is worth noting that, for simplicity of presentation, the above formulation describes the prior injection process at a single layer. In practice, we uniformly apply this Dynamic Prior Fusion mechanism across five layers of the MLLM. This multi-layer injection ensures that the diverse prior knowledge is deeply integrated, progressively guiding the spatial reasoning of the model.

3 Experiments↩︎

3.1 Dataset and Evaluation Metric↩︎

We evaluate our method across a total of six datasets: VSI-Bench [18] for spatial reasoning tasks, and five ScanNet-series benchmarks for 3D spatial understanding (ScanRefer [19], Multi3DRefer [20], Scan2Cap [21], ScanQA [22], and SQA3D [23]). Specifically, VSI-Bench comprises eight fine-grained spatial reasoning tasks: object counting, absolute distance estimation, object size estimation, room size estimation, relative distance, relative direction, route planning, and object appearance order. Following the standard protocol, we report the accuracy (%) across all its sub-tasks. The five ScanNet-series benchmarks, derived from the ScanNet corpus, span three primary tasks: (i) 3D visual grounding, where ScanRefer targets free-form object localization and Multi3DRefer extends this to multi-target and zero-target ambiguity; (ii) dense captioning, using Scan2Cap to generate natural language descriptions for localized objects; and (iii) embodied question answering, employing ScanQA for geometry-grounded open-ended questions and SQA3D for complex situated reasoning from a specific agent perspective. Following standard protocols, we report Acc@0.25/0.5 for ScanRefer, F1@0.25/0.5 for Multi3DRefer, CIDEr@0.5 for Scan2Cap, CIDEr and Exact Match (EM) for ScanQA, and Exact Match for SQA3D. More details can be found in the Appendix.

3.2 Implementation Detail↩︎

We use both Qwen2-VL-7B [1] and Qwen3-VL-8B [6] as base MLLMs on VSI-Bench; for all other experiments, we adopt Qwen2-VL-7B as the default model. To comprehensively capture spatial knowledge, we integrate \(K=5\) distinct foundation models to extract complementary visual priors. Specifically, we employ VGGT [9] for general visual-geometric feature extraction, which also serves as the base foundation model in the Efficient Prior Proxy. We further incorporate DepthAnything3 [15] for precise monocular depth cues, TraceAnything [16] for object-centric motion and relationship tracking, Wan2.1 [10] for rich spatio-temporal dynamics, and RADIO [17] for robust generic visual representations. These diverse priors collectively provide a holistic understanding of 3D scenes. We uniformly sample 32 frames from each video as the input to the vision encoder. During training, we first apply the alignment loss to train the Efficient Prior Proxy, and then freeze the proxies while fine-tuning the MLLM. We freeze the vision encoder and apply Low-Rank Adaptation (LoRA) [26] to the LLM backbone during MLLM fine-tuning. We optimize the model using the Adam optimizer [27] with a batch size of 16 and a maximum learning rate of \(1 \times 10^{-5}\). The model is trained for 1 epoch on each dataset. More details can be found in the Appendix.

3.3 Comparison with State-of-the-Art Methods↩︎

We evaluate ViPS against leading MLLMs and specialized 3D spatial understanding models. Table 1 and Table 2 summarize our results on VSI-Bench and the ScanNet-series benchmarks respectively.

Spatial Reasoning on VSI-Bench. As shown in Table 1, ViPS achieves a leading average score of \(63.8\%\), surpassing the previous best spatial-enhanced model VLM-3R [28] (\(57.2\%\)). While trailing VG-LLM-8B [14] in Appearance Order, ViPS dominates most other categories, proving the advantage of dynamically injecting multiple priors over relying solely on a single foundation model like VLM-3R or 3DRS [11].

3D spatial understanding on ScanNet-series. Table 2 shows our results across grounding, captioning, and QA tasks. Compared to generalist 3D-LLMs like Vega-3D [12] and 3DRS [11], ViPS delivers highly competitive performance. It yields top scores on visual grounding (ScanRefer, \(64.6\%\) Acc@0.25) and QA (ScanQA, \(107.9\) CIDEr), outperforming Vega-3D. Although ViPS slightly trails 3DRS on Scan2Cap, these results confirm that our unified proxy injection effectively harnesses multiple foundation priors for complex 3D spatial understanding tasks.

3.4 Ablation Study↩︎

Effectiveness of Individual Visual Priors. Table 3 evaluates the impact of individual foundation models compared to a baseline MLLM without prior injection. While each single prior yields distinct performance gains, our full ViPS framework integrating all five priors achieves the highest scores across all metrics. This confirms both the effectiveness of individual priors and the strong complementarity of harmonizing heterogeneous visual knowledge for 3D spatial understanding.

3.5pt

Effectiveness of Dynamic Prior Injection. Table 4 compares the full ViPS model against two injection variants. First, replacing zero-initialized convolutions with standard random initialization (w/o Zero-init) causes a severe performance drop. This confirms our motivation that zero-init is essential to prevent the diverse, unaligned prior distributions from confusing the MLLM during early training, preserving the original latent space and enabling progressive prior injection. Second, substituting the dynamic proxy fusion with a straightforward feature sum (Vanilla Addition) degrades performance. This demonstrates that our dynamic mechanism, which adapts to the input query, achieves more effective prior injection than static feature aggregation.

4.5pt

Effectiveness of Efficient Prior Proxy. To demonstrate the efficiency and accuracy of our Efficient Prior Proxy, we compare our method against an upper-bound setting (w/ GT Priors) where ground-truth features from all five foundation models are used to replace priors from the Efficient Prior Proxy. As shown in Table 5, our method employs multiple lightweight proxies to replace full foundation models, significantly reducing parameter overhead and inference latency (\(1\times\) vs. \(\sim5\times\)) with only a marginal drop in overall performance. We also report the estimation error (0.252 in cosine distance) between the estimated priors and the ground-truth priors during inference. These experimental results consistently demonstrate the effectiveness of our proposed approach. Furthermore, we conduct an ablation study by removing the alignment loss (\(\mathcal{L}_{alignment}\)) while retaining all other prior proxy structures (in this setting, the whole framework, including the MLLM and Efficient Prior Proxy, is trained end-to-end). Surprisingly, although the performance is inferior to the setting with the alignment loss, it still achieves considerable improvements compared to the baseline. We attribute this to the fact that even without the alignment loss, different prior proxies can still serve as projectors onto distinct sub-feature spaces, enabling the MLLM to learn diverse knowledge from the base model, akin to the multi-head mechanism in attention networks.

1.2pt

3.5 Analysis of Dynamic Prior Weights↩︎

To further understand the behavior of our Dynamic Prior Fusion mechanism, we visualize the learned fusion weights (\(w_k\) in Equation 1 ) across different spatial understanding tasks. As illustrated in Figure 5, the weight distribution adapts dynamically depending on different downstream tasks (e.g., ScanQA, ScanRefer, Multi3DRefer, SQA3D, and Scan2Cap), as well as across distinct question types within the same dataset (e.g., VSI-Bench). This observation aligns with our motivation that different foundation models exhibit distinct specializations, which shows that our framework successfully achieves context-aware prior selection by dynamically adjusting the fusion weights based on the specific scenario rather than statically relying on a single expert.

4 Related Work↩︎

4.1 Spatial Understanding with Large Language Models↩︎

Spatial understanding, a foundational pillar for real-world interaction and reasoning, has witnessed a paradigm shift with the advent of LLMs [46]–[49]. Early attempts, such as PointLLM [7], PointBind [50], GPT4Point [8], MiniGPT-3D [51], and Chat-3D [52], focused on aligning 3D point-cloud encoders directly with the LLM embedding space. To facilitate more effective cross-modal feature fusion, subsequent frameworks like Grounded-3D-LLM [40], LL3DA [53], 3D-LLaVA [44], and Inst3D-LLM [43] introduced advanced representation learning schemes. However, the inherent scarcity and noise of 3D point-cloud data often limit the scalability of these methods.

Recent research has gravitated towards video-based inputs. Prominent works such as 3D-LLM [37], Scene-LLM [39], Video-3D LLM [45], GPT4Scene [54] and SpatialStack [55] establish dense correlations between 2D features and 3D scenes by building upon powerful pre-trained MLLMs [1], [2]. Specifically, Scene-LLM [39] captures fine-grained 3D knowledge through efficient 3D visual representation learning, while Video-3D LLM [45] introduces position-aware encodings for video sequences. Similarly, LLaVA-3D [42] achieves robust perception by learning a set of 3D voxels. While our work also falls within the video-input MLLM paradigm, we diverge from these approaches by exploring how to synergistically integrate prior knowledge from diverse foundation models to further elevate 3D spatial understanding.

4.2 Integration of Foundation Model Prior in MLLMs↩︎

The rapid evolution of foundation models—such as VGGT [9], DepthAnything3 [15], and WAN [10]—has inspired a new line of research that injects diverse foundational priors into MLLMs to bolster their perceptual capabilities. Recent methods including VG-LLM [14], 3DRS [11], MiLO [56], VLM-3R [28], Vega-3D [12], ROSS3D [57], and GeoThinker [13] have demonstrated the efficacy of this paradigm. Specifically, VG-LLM employs a 3D visual geometry encoder to extract geometric priors from video sequences, while 3DRS aligns the latent space of MLLMs with VGGT via knowledge distillation. GeoThinker introduces an active perception mechanism for MLLMs to retrieve necessary geometric features, and Vega-3D injects world knowledge from video generation models, predicated on the hypothesis that these models inherently capture the underlying dynamics of scene transitions. In summary, existing efforts have predominantly focused on: (i) more efficient mechanisms for utilizing prior knowledge, and (ii) evaluating the performance of different individual model priors. In contrast, we propose ViPS, an efficient multi-model prior framework. By adaptively harmonizing various priors from disparate sources, our method significantly improves MLLM performance in spatial understanding tasks.

5 Conclusion↩︎

In this paper, we have introduced ViPS, a multi-model prior framework that unifies heterogeneous foundation-model priors within a single MLLM for spatial understanding. Our empirical study first reveals that no single foundation model dominates across tasks: different models supply distinct and complementary spatial knowledge. Building on this insight, ViPS couples an Efficient Prior Proxy, which distills the knowledge of multiple foundation models into lightweight branches sharing a common backbone and thereby avoids the linear cost of independent forward passes, with a Dynamic Prior Fusion mechanism that adaptively re-weights the resulting priors according to the input query and injects them through zero-initialized convolutions for stable training. On VSI-Bench and five ScanNet-series benchmarks, ViPS delivers state-of-the-art results in spatial reasoning, 3D visual grounding, dense captioning, and embodied question answering, while retaining single-encoder-level inference cost. These results suggest that context-aware harmonization of complementary priors is a promising path toward spatially grounded multimodal reasoning.

Acknowledgements↩︎

This work is supported by Hong Kong Research Grants Council – General Research Fund (Grant No.), Hong Kong Innovation and Technology Commission – Innovation and Technology Fund (Grant No.ITS/488/24FP), and HKU Seed Fund for PI Research.

6 Detailed Training Dataset Description↩︎

6.1 ScanNet-series Dataset↩︎

For the 3D spatial understanding tasks, our training protocol aligns with the configurations established by previous works such as Video-3D LLM and 3DRS, which aggregate data from five distinct benchmarks, yielding a combined training set of approximately 223K sample pairs. Specifically, the composition includes:

• ScanRefer and Scan2Cap: Each dataset contributes 36,665 QA instances.

• Multi3DRefer: This multi-target grounding dataset provides 43,838 data entries.

• ScanQA: We utilize 26,515 question-answering pairs from this dataset.

• SQA3D: Serving as the largest subset in our collection, SQA3D contributes 79,445 samples.

With the exception of SQA3D, which is sourced from 518 unique 3D environments, the remaining four datasets are constructed upon 562 unique scans from the ScanNet corpus.

6.2 VSI-Bench Dataset↩︎

For the spatial reasoning evaluation, we follow the dataset composition presented in VLM-3R to train our model on VSI-Bench. Our training subset comprises a total of 207,658 instruction-tuning QA pairs. The dataset is structurally diverse, encompassing various scene typologies and specific reasoning tasks. The detailed distribution is as follows:

• ScanNet++ QA: The majority of the data originates from the high-fidelity ScanNet++ corpus, accounting for 135,119 QA pairs.

• ScanNet QA: An additional 51,630 QA pairs are derived from ScanNet.

• Absolute Distance Estimation: 16,805 samples are dedicated to object absolute distance estimation tasks.

• Route Planning: The dataset also includes 4,104 instances specifically designed to evaluate and enhance the model’s route planning and navigational reasoning capabilities.

6.3 Details of Section 2.1↩︎

To ensure computational efficiency during our exploration, here we train the models on a 10% subset of the original training datasets used in 3DRS [11] and VLM-3R [28]. We conduct our empirical assessment on the VSI-Bench [18] and ScanNet-series benchmarks including ScanRefer [19], Multi3DRefer [20], Scan2Cap [21], ScanQA [22], and SQA3D [23]. We employ Qwen2-VL [1] as our base MLLM. For each evaluated variant, we extract features from each foundation model via its respective encoder and project them into the same hidden dimension as the MLLM using an MLP. Then inject them into the image tokens of the MLLM via addition. The evaluation metrics are consistent with those in the main text.

7 Training Detail↩︎

A comprehensive summary of all detailed hyperparameters and configurations is provided in Table 6 and Table 7. To align the external visual priors with the internal representations, we downsample all features extracted by the prior models to match the resolution of the MLLM’s image tokens.

The trainable parameters in our ViPS framework include the LLM backbone (updated via LoRA), the multimodal projector (mm_projector), the zero-initialized convolutions and MLPs responsible for generating dynamic weights within the Dynamic Prior Fusion, and all parameters within the Efficient Prior Proxy. Under our hardware configuration, the training process takes approximately 24 hours for the ScanNet-series datasets and 30 hours for the VSI-Bench dataset.

8 Impact of Different Base Model↩︎

In the main paper, we default to using VGGT as the base foundation model within our Efficient Prior Proxy. To further investigate the robustness and flexibility of our ViPS framework, we conduct additional experiments by substituting VGGT with other foundation models as the base model, while keeping the rest of the framework unchanged.

As shown in Table 8, using different base models yields comparable and consistently strong performance across the ScanNet-series benchmarks. Although VGGT yields the best overall performance, we observe that TraceAnything achieves better results on certain metrics (e.g., 62.02 F1@0.25 on Multi3DRefer and 85.53 C@0.5 on Scan2Cap). Furthermore, all other foundation models maintain highly comparable performance when used as the base model. These results demonstrate that our Efficient Prior Proxy and Dynamic Prior Fusion mechanisms are robust and not strictly dependent on a specific base model choice.

9 Visualization Result↩︎

To further demonstrate the effectiveness of our proposed ViPS framework, we provide visualization results in Figure 6. In this comparison, the baseline model is obtained by fine-tuning the Qwen2-VL architecture on the same training dataset used for our model. As illustrated, by synergistically harmonizing diverse visual priors, our ViPS framework achieves a more precise understanding of complex spatial relationships and generates more accurate responses compared to the baseline.

10 Broader Impact↩︎

Positive Impacts. The capabilities introduced by our ViPS framework significantly advance the spatial reasoning and 3D spatial understanding of MLLMs. By effectively harmonizing diverse visual priors, our approach paves the way for more intelligent embodied agents, such as domestic robots and autonomous navigation systems. Furthermore, our Efficient Prior Proxy minimizes inference overhead, facilitating the deployment of sophisticated 3D perception systems on resource-constrained edge devices. This efficiency can democratize access to advanced AI, fostering innovations in assistive applications for visually impaired individuals to navigate complex environments safely.

Potential Negative Impacts and Mitigation. Endowing AI systems with highly accurate 3D spatial understanding capabilities inherently presents privacy and security risks. If deployed irresponsibly, such technologies could be exploited for unauthorized surveillance or the invasive reconstruction of private indoor spaces. To mitigate these ethical concerns, future real-world applications must incorporate robust anonymization pipelines—such as the automatic obfuscation of human faces and sensitive documents—prior to processing scene data. Additionally, strict access controls, transparent user consent mechanisms, and compliance with data protection regulations are essential to safeguard individual privacy.

11 Limitation and Future Work↩︎

While ViPS demonstrates state-of-the-art performance in 3D spatial understanding, several limitations remain. First, due to computational resource constraints, our experiments are currently restricted to a 7B-parameter base MLLM and training datasets on the scale of a few hundred thousand samples. Scaling up both the model size and data volume could further enhance performance. Second, extracting spatial awareness from external visual priors is ultimately a sub-optimal workaround. A more fundamental future direction is to internalize this geometric knowledge directly during the MLLM pre-training phase, which could be achieved by introducing more diverse spatial datasets, fine-grained 3D sub-tasks, and tailored loss functions. Finally, our evaluations are currently confined to standard offline benchmarks. Deploying and validating spatial-aware MLLMs in real-world robotics and physical embodied AI scenarios remains an exciting avenue for future exploration.

References↩︎

---

1. Corresponding author.↩︎