1 Introduction↩︎

Vision-Language Navigation (VLN)[1], [2] is a challenging and crucial task in embodied AI, which requires agents to accept visual observation inputs and follow language instructions to navigate in an unseen environment. Recent advances in Multimodal Large Language Model (MLLM)[3]–[5] and Vision-Language-Action (VLA)[6]–[8] model have prompted VLN to receive more attention. In particular, advancing explicit spatial reasoning to accurately align language instructions with continuous 3D geometry and environmental topology has emerged as a critical prerequisite for achieving robust, long-horizon navigation in complex settings.

Existing VLN methods are generally divided into two main paradigms: the end-to-end paradigm[6], [7], [9]–[15] and the zero-shot paradigm[16]–[24]. The end-to-end VLN paradigm typically employs a unified vision-language policy, fine-tuned on expert navigation trajectories to directly predict sequential actions. While effective in constrained or seen environments, it struggles with long-horizon adaptation and lacks the dynamic, step-wise reasoning required for complex multi-step decision-making. Conversely, zero-shot pipelines leverage pre-trained MLLM to enable training-free generalization across unseen scenes. However, their planning reliability is inherently constrained by the limited spatial grounding and geometric reasoning capabilities of off-the-shelf foundation models. Consequently, existing methodologies face complementary limitations that hinder robust navigation in complex open-world scenarios. Although recent studies[25], [26] have proposed dual-system approaches to compensate for the limitations of individual-systems, these methods merely provide a coarse integration of the two fundamental paradigms, without fundamentally enhancing the capabilities of the underlying systems. Specifically, compared to semantic capabilities, spatial capabilities are the most abstract and easily overlooked in VLN tasks, yet spatial awareness can directly improve the agent’s robustness and success rate in long-horizon navigation.

First, to address the limitations of the single system in VLN, we introduce a dual-system with fast–slow cooperation to jointly complete navigation tasks. Next, to enhance the spatial awareness of the underlying systems in spatial navigation, we equip System 1 with Global and Local Spatial Enhancement Strategies, enabling the VLM model to comprehend the environmental structure from coarse to fine. In System 2, we adopt a three-stage “Mapping–Rendering–Reasoning” pipeline, which provides the MLLM with reasoning cues from both global and dynamic perspectives.

Our contributions are as follows:

1. Dual-System VLN Framework: We propose SEDualVLN, a dual-system architecture that combines a fast VLM-based action generator with a slow MLLM-based planner, coordinated by a scheduler, to jointly perform complex navigation tasks.

2. Spatially-Enhanced VLM (System 1): We introduce Global and Local Spatial Enhancement strategies for the VLM model, enabling coarse-to-fine comprehension of the environment and improving spatial awareness for robust long-horizon navigation.

3. Mapping–Rendering–Reasoning MLLM (System 2): We design a three-stage pipeline that provides the MLLM with both global and dynamic visual cues, enhancing spatial reasoning and waypoint planning in unseen environments.

4. Comprehensive Evaluation: We demonstrate that SEDualVLN achieves state-of-the-art performance on VLN-CE benchmarks and conduct ablation studies to validate the effectiveness of each system and module in improving navigation success and robustness.

2 Related Work↩︎

2.1 Vision-Language Navigation (VLN)↩︎

Vision-Language Navigation (VLN) requires an embodied agent to follow natural language instructions to navigate in unseen photorealistic 3D environments[27]–[30]. Early studies mainly focused on discrete settings (e.g.,R2R[27]), emphasizing cross-modal alignment via attention mechanisms and topological memory for long-term planning [31], [32]. Recent advances[7], [12], [25] have shifted toward continuous environments (e.g.,VLN-CE[29], [30]), exploring pretraining, data augmentation, and online adaptation techniques to improve generalizability.

End-to-end VLN approaches integrate visual perception, language understanding, and action prediction in a unified framework by fine-tuning Video-LLMs[3], [4] or Vision-Language-Action (VLA) models on large-scale navigation data. These methods[6], [7], [9], [12] leverage video stream encoding, spatio-temporal context modeling, and techniques such as token pruning or DAgger[33] to generate low-level actions directly from RGB observations[9], [12], [14]. While achieving strong benchmark performance, they often struggle with long-horizon navigation due to accumulating context length and redundant historical token processing, and exhibit limited dynamic reasoning in complex scenes.

Zero-shot VLN approaches employ powerful multimodal LLMs (e.g., GPT-4o[34]) as planners with carefully designed prompts and Chain-of-Thought reasoning, often in a training-free manner. Dual-process frameworks further combine lightweight expert models for routine navigation with LLMs for anomalous cases. These methods[17], [19], [26] excel at commonsense reasoning and generalization but lack fine-grained spatial and geometric understanding, leading to suboptimal decisions in precise spatial correlation and continuous video streams.

2.2 Spatial Reasoning in VLN↩︎

Spatial reasoning is fundamental to VLN, requiring precise alignment between linguistic instructions, environmental topology, and continuous 3D geometry. Recent video-based VLMs and VLA models [9], [12], [13] enhance spatial retention through temporal context modeling and memory decoupling. However, their spatial representations remain largely implicit, making them susceptible to metric drift and poor interpretability in complex layouts. To mitigate this, explicit spatial reasoning has emerged as a critical direction. Topological planners [35] construct evolving connectivity graphs with dual-scale transformers for global action planning, while geometry-aware methods [36] integrate depth/normal maps with slot attention to learn geometry-enhanced visual representations. Dedicated spatial grounding frameworks like Spatial-VLN [17] and SpatialNav [20] explicitly model geometric constraints and spatial relations through structured perception and scene graphs. Concurrently, LLM-driven approaches [25], [26] decompose navigation into deliberative spatial planning via chain-of-thought or constraint-aware filtering. Despite their logical strength, these methods predominantly operate over abstract textual cues or fragmented 2D proxies, lacking direct access to continuous, physically consistent 3D geometry—frequently inducing spatial hallucination and degrading long-horizon robustness.

To bridge this gap, we propose a spatially-enhanced dual-system architecture. System 1 explicitly models spatial semantics and connectivity to establish a robust topological prior for efficient path grounding. System 2 leverages 3D mapping to render observed trajectories, providing high-fidelity, multi-view-consistent 3D context for deliberative planning. By unifying explicit topological reasoning with physically grounded 3D observation, our framework achieves more accurate, interpretable, and robust spatial reasoning compared to prior implicit or abstraction-dependent methods.

3 Method↩︎

We formulate continuous VLN task as a sequential decision-making problem under monocular RGB perception. At each step \(t\), the agent processes a language instruction \(I\) alongside a trajectory of ego-centric observations \(O_t = \{x_0, \dots, x_t\}\), where \(x_\tau\) is the visual input at time \(\tau\). To navigate successfully, the agent must learn a policy \(\pi(O_t, I)\) to dictate the next atomic action \(a_{t+1} \in \mathcal{A}\) based on its visual history and the given instruction. The discrete action space is restricted to \(\mathcal{A} = \{\text{Move Forward}, \text{Turn Left}, \text{Turn Right}, \text{Stop}\}\). An episode is deemed a success if the agent issues a \(Stop\) command within a proximity threshold of 3m from the target place, without exceeding a maximum budget of 500 steps.

As illustrated in Figure 1, SEDualVLN is a dual-system VLN framework. System 1(Sec. 3.1), is a VLM model that generates actions based on video stream input. System 2 (Sec. 3.2), consists of a general MLLM model and mapping module that determines optimal navigation waypoints using 3D mapping overhead view and frontier-path Rendering visualizations. Systems 1 and 2 operate at different frequencies to achieve a fast-slow coordination: while System 2 requires more time for waypoint-level reasoning, System 1 executes multiple low-level actions during this period. Once System 2 completes its planning, the agent proceeds to execute the planned waypoints, thereby mitigating delays from lengthy reasoning while still incorporating global spatial information.

3.1 System1: Spatially-Enhanced VLM Model↩︎

The overall framework of System 1 is illustrated in Figure 2. Our model is built upon the StreamVLN[12] backbone, adopting its multi-turn dialogue mechanism, KV cache, and spatial token pruning. We introduce two major modifications, enhancing its spatial awareness for long-range tasks. First, We enhance VLM by integrating a global spatial enhancement strategy that guides the distribution of visual token embeddings so that they implicitly encode spatial information, without relying on depth maps from sensors or 3D models. Second, we introduce Local Spatial Enhancement Strategy, enabling the VLM to focus more effectively on the connectivity of nearby passages and thereby reducing the risk of taking incorrect branches.

3.1.0.1 Global Spatial Enhancement Strategy

Compared to approaches that incorporate spatial information via depth maps or explicit 3D reconstruction, we adopt an implicit strategy to enhance the spatial awareness of visual tokens. Prior work [38] has shown that supervising relatively deep but not the deepest layers is most effective for improving action performance, likely because the deeper layers lose more low-level visual features, making it more difficult for them to receive supervision targeted at spatial representations. Therefore, we apply supervision at intermediate attention fusion layers of LLaVA-Video [4] (the specific layers are selected based on ablation studies).

For each frame along the agent’s navigation sequence, visual token embeddings are projected through a two-layer MLP with batch normalization to align with the spatial representation dimension. These per-frame embeddings are then supervised against spatial features generated by a pretrained 3D foundation model, VGGT [37], with positional encodings added to preserve spatial ordering. The supervision is implemented by minimizing the cosine distance between the projected visual tokens and the corresponding 3D spatial representations, effectively guiding the model to integrate geometric and structural cues from the environment. Formally, let \(\mathbf{V}_t\) denote the projected visual token at frame \(t\), \(\mathbf{S}_t\) the spatial representation from the 3D model, and \(\mathbf{p}_t\) the positional embedding; the combined training objective is defined as \[\mathcal{L}_{\text{SE}} = \mathcal{L}_{\text{action}} + \alpha \cdot \frac{1}{N} \sum_{t=1}^{N} \Big[ 1 - \cos\big(\mathbf{V}_t, \mathbf{S}_t + \mathbf{p}_t\big) \Big],\] where \(N\) is the number of frames and \(\alpha\) balances the action prediction loss and the spatial supervision loss.

3.1.0.2 Local Spatial Enhancement Strategy.

In VLN tasks, it is common for an agent to move from one region to another, where passages play a critical role as the connecting structures between regions. Most failures of VLM occur when the agent enters an incorrect passage, making it unable to return to the correct trajectory. Extracting global visual tokens from the current RGB frame using a 3D model is a common approach; however, such visual tokens often struggle to capture passage existence due to interference from numerous irrelevant tokens. To address this issue, we propose a Channel Connectivity Extraction Module, designed to enable the VLM to explicitly focus on passage connectivity. This module operates as follows: (1) the presence of passages (e.g., corridors, doors) is detected using an open-vocabulary Grounding DINO[39] approach; (2) the detected passages are then segmented via SAM[40] to obtain binary masks; (3) all passage masks are set to 1 while non-passage regions are set to 0, completing the image encoding. Finally, MLP is employed to project the resulting representations to the same dimensionality as the global visual tokens.

3.2 System2: Mapping–Rendering–Reasoning MLLM↩︎

The overall workflow of System 2 is illustrated in Figure 3, comprising three main Stages: Mapping, Rendering, and Reasoning.

3.2.0.1 Mapping.

We construct a lightweight 3D map of the environment and a 2D frontier map in real-time. The 2D frontier map guides the agent to explore new regions, following the construction rules detailed in VLFM [41]. Multi-view 3D mapping is computationally expensive and requires many agent steps. To alleviate this, we adopt a feed-forward approach based on LingBot-Map [42], which incrementally builds the 3D map online.

3.2.0.2 Rendering.

Let \(\mathcal{F} = \{\mathbf{f}_1, \dots, \mathbf{f}_n\}\) denote the set of frontier points in 2D coordinates. Each frontier point \(\mathbf{f}_i\) is projected into the 3D map to obtain its corresponding 3D location \(\mathbf{F}_i\). For each \(\mathbf{F}_i\), we compute a collision-free path from the agent’s current position \(\mathbf{x}_0\) to \(\mathbf{F}_i\) using the A* algorithm: \(\mathcal{P}_i = \text{A*}(\mathbf{x}_0, \mathbf{F}_i)\).

To generate a dense set of Rendering views along the path, we perform interpolation between consecutive path nodes. Specifically, if the Euclidean distance between two consecutive points \(\mathbf{p}_j\) and \(\mathbf{p}_{j+1}\) exceeds a threshold \(d\), we insert intermediate points such that the spacing satisfies \(\|\mathbf{p}_{j+1} - \mathbf{p}_j\| < d\). Each interpolated point corresponds to a virtual camera pose, and the RGB image at that pose is Rendering as \(\mathbf{I}_k\). To reduce redundant information and MLLM token usage, we apply a cosine-similarity based pruning: \[s(\mathbf{I}_k, \mathbf{I}_{k+1}) = \frac{\langle \phi(\mathbf{I}_k), \phi(\mathbf{I}_{k+1}) \rangle}{\|\phi(\mathbf{I}_k)\|\|\phi(\mathbf{I}_{k+1})\|} < \tau \implies \text{keep } \mathbf{I}_{k+1},\] where \(\phi(\cdot)\) denotes a CLIP[43] encoder embedding and \(\tau\) is a similarity threshold. After sampling and redundancy removal, we obtain \(m\) RGB frames \(\{\mathbf{I}_1, \dots, \mathbf{I}_m\}\) for each path.

3.2.0.3 Reasoning.

The reasoning stage operates in two phases. In the first phase, self-perception environment enhancement, the GPT-4o[34] is provided with the 3D top-down map and is required to describe the agent’s current location and nearby environment. In the second phase, simulated agent motion, the GPT-4o evaluates each frontier path by comparing the Rendering image sequences \(\{\mathbf{I}_1, \dots, \mathbf{I}_m\}\) with the natural language instruction. It then selects the frontier point whose path best aligns with the instruction. This two-stage process allows the agent to integrate spatial context and visual guidance when making navigation decisions (For the detailed procedure, please refer to the Appendix 7).

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Simulation Benchmark Setup.

We evaluate our method on two general VLN-CE benchmarks, R2R-CE [29] and RxR-CE [30]. Both datasets are based on Matterport3D reconstructions, with navigation episodes exceeding 10 meters. The agent is allowed to navigate freely using low-level actions, such as moving forward 0.25 meters or turning left 15 degrees, rather than relying on fixed navigation graph nodes. All experiments are evaluated using standard metrics, including Navigation Error (NE), Success Rate (SR), Oracle Success Rate (OS), and Success weighted by Path Length (SPL), normalized Dynamic Time Warping (nDTW), following prior works.

4.1.0.2 Base Models and Implementation Details.

We build System 1 based on the LLaVA-Video [4] 7B model, which employs Qwen2-7B[44] as the language backbone. We perform several additional modifications: first, during training, we reconstruct the loss function by incorporating a spatial alignment loss; second, we use Local Spatial Enhancement Strategy and extract additional visual tokens from the RGB images. Initially, we fine-tune the model for one epoch using only the oracle VLN trajectories. Subsequently, we use this model to collect DAgger trajectories and continue training for another epoch using both the VLN data and general multimodal data. Each training step processes 128 video clips. The entire training procedure takes approximately 2000 A100 GPU hours. System 1 runs using a single RTX 4090 GPU. System 2 requires no training; it utilizes the pre-trained 3D model LingBot-Map[42] for 3D mapping, and its deployment also occupies one RTX 4090 GPU.

4.2 Experiment Result↩︎

4.2.0.1 Quantitative analysis.

As shown in Table1, we compare our SEDualVLN with previous state-of-the-art methods on the two splits of VLN-CE. These methods can be categorized into four types: (1) multi-sensor-based approaches (CMA[45], BERT[45], ETPNav[46], Seq2Seq[29]), (2) VLM methods using first-person RGB-D images (LAW[47]), (3) VLM methods using only first-person RGB images (NaVILA[7], StreamVLN[12]), and (4) dual-system architecture methods (DualVLN[25]). Experimental results show that, without relying on additional data sources, our method achieves a SR improvement of 3% and 2.5% over DualVLN on the two splits, respectively, reaching a new state-of-the-art level.

4.2.0.2 Qualitative analysis.

As shown in Figure 4, we compare with the state-of-the-art method StreamVLN. The results demonstrate that our model significantly reduces the probability of taking wrong turns at junctions, indicating enhanced spatial awareness. As shown in Figure 5, we visualize navigation and mapping visualization of our SEDualVLN (The reasoning process of System 2 can be found in the Appendix 7). For more experimental visualization results, please refer to the supplementary video material.

4.3 Ablation Study↩︎

To demonstrate that each system and module of SEDualVLN is indispensable, we conduct extensive ablation studies, including: (1) evaluating the effect of Spatial Enhancement Strategy at different attention layers of the VLM on the navigation performance; (2) assessing the effect of System 1’s Channel Connectivity Extraction Module; (3) comparing the navigation results when System 1 and System 2 are run independently versus collaboratively; and (4) analyzing the impact of different general-purpose MLLM.

4.3.0.1 Effect of Global Spatial Enhancement Strategy at Different Attention Layers of the VLM.

We analyze whether the Spatial Enhancement Strategy effectively improves the agent’s global spatial awareness, and we further study its effect when supervised at different layers of the VLM model. The experiments are conducted on two datasets for both training and evaluation. As shown in Table 2, the results indicate that supervising the Global Spatial Enhancement Strategy at layer 24 yields the best performance. Specifically, compared to the model without supervision, improving the SR by 6.0% and SPL by 3.8% on the R2R Val-Unseen split, and the SR by 14.4% and SPL by 11.3% on the RxR Val-Unseen split. These results indicate that supervising the Spatial Enhancement Strategy at mid-to-deep layers is most effective in enhancing the agent’s spatial.

4.3.0.2 Effect of Local Spatial Enhancement Strategy.

As shown in Table 3. Incorporating Local Spatial Enhancement Strategy (LSES) consistently improves navigation metrics across both R2R and RxR Val-Unseen sets. On R2R, NE is reduced by 0.20, SR improves by 3.5%, SPL increases by 1.0% , and OS rises by 3.4%. For RxR, NE decreases by 0.11, SR improves by 4.1%, and nDTW increases significantly by 6.3%. The SPL increases by 1.4% Overall, these results suggest that agents benefit from LSES during navigation, and LSES significantly improves success rates at decision points such as intersections.

4.3.0.3 Impact of System 1 and System 2 Collaboration on VLN.

To further demonstrate the effectiveness of the dual-system approach, we introduce an additional metric, Average Time (AT, in seconds), which measures the mean time taken by the agent to complete a navigation eposide. Comparing (a) and (d), we observe that the dual-system achieves significantly higher success rates than the pure VLM system, with SR improvements of 4.5% and 5.4%, albeit at the cost of slightly increased time. Comparing (b) and (d), the dual-system outperforms the general MLLM single-system, with SR gains of 7.5% and 7.3%, while the average completion time is reduced by a factor of six. Additionally, we examine the metrics under different switching frequency ratios, as shown in (c), (d), and (e), and find that a frequency ratio of 20:1 allows our SEDualVLN to best balance SR and time efficiency.

4.3.0.4 Effect of different general-purpose MLLM on VLN.

As shown in Table5, using Gemini results in two benchmarks SR of 66.9% and 63.4%. The Qwen2.5-VL model performs slightly better, achieving an SR of 67.1% and 63.7%. However, GPT-4o demonstrates the highest efficacy. This indicates that GPT-4o provides superior spatial reasoning and zero-shot generalization capabilities.

5 Limitation↩︎

Our main limitation lies in deployment on physical devices. This is because the latest real-time 3D mapping features still face constraints in terms of real-world real-time performance, which leads to distorted 3D mapping results and consequently impacts the entire system.

6 Conclusion↩︎

We presented SEDualVLN, a spatially-enhanced dual-system Vision-Language Navigation framework that successfully integrates the strengths of end-to-end VLM and zero-shot MLLM pipeline. To overcome the challenges of long-horizon navigation and limited spatial grounding, SEDualVLN employs a fast-slow coordinated approach. Both systems are explicitly designed to prioritize and enhance spatial awareness. Our results on the VLN-CE benchmarks establish a new state-of-the-art. By cultivating a stronger sense of direction and dynamic reasoning capabilities, SEDualVLN paves the way for more robust and capable navigation agents in complex, unseen environments.

7 Technical appendices and supplementary material↩︎

In this section, we first build upon the previously mentioned qualitative experiment Section4.2.0.2 to further illustrate how the two-stage reasoning of the MLLM in System 2 is performed. We then provide the prompts used in both System 1 and System 2. For complete video visualizations and additional experimental segments, please refer to the supplementary video material.

As shown in Figure 6 and Figure 7, the reasoning of the MLLM is divided into two stages. In the first stage, we guide the MLLM to analyze the environmental structure, object relationships, and possible orientations based on the top-down map. In the second stage, we provide the MLLM with additional cues through Rendering views, enabling it to complete the selection of waypoints.

Below are prompt templates for VLM and MLLM.

References↩︎