Autonomous Vehicle Navigation; Integrated Planning and Learning; AI-Based Methods.

1 Introduction↩︎

End-to-end (E2E) autonomous driving has fundamentally shifted the paradigm by directly mapping raw multi-modal sensor inputs to continuous vehicle control actions or waypoints [1]. By unifying perception, trajectory prediction, and motion planning within a single differentiable framework, E2E models learn shared latent representations that effectively mitigate the cascading errors inherent in traditional modular pipelines. This paradigm has evolved from early behavioral cloning approaches to unified multi-task frameworks such as UniAD [2] and VAD [3], and has been further advanced by generative priors and Mixture-of-Experts routing, as exemplified by DiffusionDrive [4] and ARTEMIS [5].

Despite these advances, conventional E2E models remain largely data-driven black boxes. Their limited semantic interpretability and lack of high-level reasoning hinder generalization in long-tail scenarios, weaken multi-agent interaction modeling, and restrict the integration of human instructions.

Vision–Language Models (VLMs) address these limitations by introducing semantically grounded reasoning. Trained on large-scale image–text pairs, VLMs align visual inputs with language in a shared representation space, enabling natural language understanding and Chain-of-Thought (CoT) reasoning conditioned on the scene. This cross-modal abstraction improves generalization in complex and long-tail scenarios [6], [7]. Recent works, such as DriveLM [8] and DriveGPT4 [9], leverage CoT prompting to jointly generate driving decisions and interpretable rationales, achieving strong zero-shot performance. Follow-up work like ReasonPlan [10] further enhances transparency by producing step-by-step decision processes.

However, extending VLMs to low-level control exposes a fundamental semantic–physical gap [11]. VLM outputs are inherently discrete and autoregressive (e.g., “turn left” or “slow down”), whereas vehicle control requires continuous, high-frequency signals governed by trajectory planning. Moreover, tokenization and deep abstraction progressively degrade fine-grained spatial and geometric information critical for precise motion generation. As a result, VLMs typically operate as open-loop “advisors”, relying on downstream controllers to translate symbolic outputs into executable actions. This multi-stage design introduces latency and further loss of spatial fidelity, limiting control reliability.

To bridge the coherence gap between language and action, Vision-Language-Action (VLA) models unify perception, reasoning, and control within a single policy. Originating in robotics (e.g., RT-2 [12] and VLA-Adapter [13]), VLAs encode visual observations, ego states, and language instructions into a shared latent space, where cross-modal attention infers intent and action heads generate continuous control signals [14]–[16]. Recent advances demonstrate the ability to ground free-form instructions into trajectories [17], produce interpretable reasoning via CoT [18], and incorporate expressive planners such as diffusion-based action generators [19].

Nevertheless, existing VLA architectures introduce a critical structural limitation. Most approaches rely on supervised fine-tuning (SFT) to enforce action generation or align control outputs with highly abstract features from deep network layers. While effective for coarse behavior prediction, these strategies overlook the progressive loss of fine-grained spatial information. This issue is exacerbated by the prevalent single-stream design, which directly maps high-level semantic representations to low-level control, entangling reasoning with execution. Consequently, cross-modal alignment is performed on heavily compressed embeddings, neglecting the rich spatial cues preserved in earlier visual features and inducing a systematic semantic-over-geometry bias. Although such models can mimic expert behavior, their lack of explicit spatial and geometric priors limits spatial awareness, often resulting in unsafe or geometrically inconsistent trajectories.

As illustrated in Fig. 1, current paradigms face structural limitations. To address these challenges, we introduce \(\mathrm{S}^2\)-VLA, which decouples semantic and spatial processing streams in Vision-Language-Action models. Unlike conventional VLAs that entangle reasoning and control, \(\mathrm{S}^2\)-VLA explicitly decouples semantic reasoning from spatial perception, enabling safe, boundary-aware, and physically compliant trajectory generation.

The main contributions of this work are as follows.

(1) We propose \(\mathrm{S}^2\)-VLA to overcome the semantic–physical gap inherent in current models by fundamentally decoupling the semantic and spatial streams. By harvesting spatially dense features from shallow visual layers and applying auxiliary perception supervision, the spatial stream explicitly grounds the model in rich geometric priors for accurate spatial reasoning.

(2) We design a Dual-Stream Planning Adapter that reconciles high-level semantic intent with low-level control. Through hierarchical cross-attention, it effectively fuses semantic guidance with strict spatial constraints, ensuring both logical consistency and geometric constraints.

(3) Extensive evaluations on the NAVSIM closed-loop benchmark demonstrate that \(\mathrm{S}^2\)-VLA achieves a Predictive Driving Model Score (PDMS) of 87.1, establishing a new state-of-the-art for VLA models under a purely supervised fine-tuning (SFT) setting. By mitigating the spatial representation collapse inherent to traditional VLMs, our framework significantly outperforms existing baselines, notably achieving the highest No Collision (NC) rate of 98.4.

2 Method↩︎

The overall architecture of the proposed (\(\mathrm{S}^2\)-VLA) model is illustrated in Fig. 2. To resolve the semantic-geometric entanglement inherent in conventional VLA models, \(\mathrm{S}^2\)-VLA processes multimodal inputs, including a high-level navigation command \(C_\text{nav}\), ego-vehicle motion history \(T_\text{hist}\), a front-view image \(I\), and learnable visual and action queries. Following initial tokenization and vision encoding, the architecture explicitly decouples representation learning into two streams.

The Semantic Stream leverages InternVL3-2B [20] as its vision-language backbone. By jointly encoding text and visual tokens, it extracts high-level, multi-scale semantic features that encapsulate complex reasoning and driving intent.

In parallel, the Spatial Stream bypasses the autoregressive language bottleneck by directly extracting uncompressed visual features from the Visual Encoder. Furthermore, this stream preserves fine-grained spatial details and enforces explicit spatial and geometric priors via auxiliary map and agent prediction tasks.

Finally, the Dual-Stream Planning Adapter aligns these decoupled representations via cascaded attention mechanisms, seamlessly fusing abstract semantic intent with precise spatial constraints to generate dynamically feasible trajectories.

2.1 Multi-Scale Semantic Stream↩︎

Current VLM-based driving systems primarily rely on the final-layer representations of their backbone networks. Due to heavy network compression, these highly abstract features suffer from semantic discretization and a distinct semantic-over-action bias. Consequently, they lack the fine-grained multimodal details necessary to ground high-level reasoning into precise, executable actions for trajectory planning. To overcome this limitation, we propose a Multi-Scale Semantic Stream designed to selectively extract and aggregate hierarchical features.

Multimodal Input and Query Injection. All inputs are tokenized into a unified sequence of text and visual tokens. At the input of the semantic stream, we inject \(N_{\text{act}} = 64\) learnable action queries \(Q_{\text{act}}\) to explicitly drive the network to aggregate execution-relevant features into action-centric representations. The choice of \(N_{\text{act}} = 64\) directly follows the configuration validated in VLA-Adapter [13]. This number provides the optimal balance between representational capacity and computational efficiency.

Backbone and Sparse Sampling. We adopt InternVL3-2B [20] as the backbone, which combines the InternViT visual encoder with the Qwen2.5 language model to jointly process multimodal tokens and action queries. While full-layer feature fusion is computationally prohibitive, Transformer layers yield inherently complementary representations: shallow layers retain fine-grained details, while deeper layers capture high-level semantics [13], [21], [22].

To balance inference efficiency with representation richness, we adopt a sparse multi-scale sampling strategy. Specifically, we extract hidden states exclusively from a carefully selected subset of layers, \(L = \{3, 8, 13, 18, 23, 24\}\), forming the multi-scale feature set \(\big\{ (V_\text{sem}^{(l)}, V_\text{act}^{(l)}) \mid l \in L \big\}\), where \(V_\text{sem}^{(l)} \in \mathbb{R}^{(N_\text{text}+N_\text{img}) \times d_\text{vlm}}\) denotes semantic features at layer \(l\), and \(V_\text{act}^{(l)} \in \mathbb{R}^{N_\text{act} \times d_\text{vlm}}\) represents the corresponding action-centric features. \(N_\text{text}\), \(N_\text{img}\), and \(N_\text{act}\) denote the sequence lengths of the text tokens, image tokens, and action queries, respectively, while \(d_\text{vlm}\) is the hidden dimension of the backbone.

Output and State Integration. The semantic stream outputs dense semantic features \(V_{\text{sem}}\) and action-centric features \(V_{\text{act}}\). To explicitly incorporate kinematic priors, the ego-vehicle’s historical states are encoded via a Multi-Layer Perceptron (MLP) into an ego-state embedding \(E_{\text{ego}} \in \mathbb{R}^{1 \times d_{\text{vlm}}}\). At each layer \(l\), this embedding is concatenated along the sequence dimension with the action-centric features \(V_{\text{act}}^{(l)}\) to form a unified state memory \(E_{\text{mem}}\). Together, this memory \(E_{\text{mem}}\) and the dense semantic features \(V_{\text{sem}}\) provide a structured, intent-aware representation that serves as the crucial semantic foundation for the downstream Dual-Stream Planning Adapter.

2.2 Task-Driven Spatial Stream↩︎

The autoregressive pipeline in VLMs inherently compresses visual representations, severely weakening explicit spatial awareness. To address this limitation, we design a dedicated Spatial Stream that directly accesses visual features from the Vision Transformer (ViT) and injects explicit spatial and geometric priors through task-driven supervision.

Visual Input and Feature Extraction. We adopt a dynamic resolution strategy [23], partitioning each high-resolution front-view image into \(N_{patch}=9\) parallel-processed patches, consisting of eight local crops and one global thumbnail. To actively capture spatial information, we introduce \(N_\text{vis}=64\) learnable visual queries \(Q_\text{vis}\) per patch. These queries are concatenated with the standard class token \(H_{\text{CLS}}\) and patch tokens \(H_{\text{patch}}\), formulating the input sequence to the Vision Transformer (ViT) as \([Q_{\text{vis}}; H_{\text{CLS}}; H_{\text{patch}}]\).

During the forward pass, \(Q_{vis}\) aggregates fine-grained geometric and texture details via self-attention. After encoding, the visual queries are decoupled and reorganized into a spatially dense visual feature \(V_{\text{vis}} \in \mathbb{R}^{B \times (N_{\text{patch}} \times N_{\text{vis}}) \times C}\), successfully preserving fine-grained geometric information. To establish global spatial coherence, we add tile-level positional embeddings \(P_{tile}\) and apply a Transformer encoder, which is then explicitly grounded in spatial and geometric priors via two auxiliary perception heads, illustrated in Fig. 3.

Map Head. To explicitly establish a spatial geometric prior of the environment, the 1D token sequence \(V\) is deterministically reshaped into a 2D spatial grid \(V_{2D} \in \mathbb{R}^{d \times 24 \times 24}\). Subsequently, a progressive sequence of upsampling and convolutional layers decodes this intermediate grid to output a high-resolution BEV multi-class semantic map \(\hat{M}_{\text{sem}} \in \mathbb{R}^{C \times 128 \times 256}\). Because our architecture relies exclusively on a monocular front-view camera, the Map Head focuses on reconstructing a local, visible BEV semantic space immediately ahead of the ego-vehicle(specifically spanning \(X \in [0, 32]\) m and \(Y \in [-32, 32]\) m). To mitigate the severe class imbalance between background pixels and safety-critical elements, this branch is supervised via a weighted cross-entropy loss: \[\mathcal{L}_{\text{map}} = -\sum_{c} \omega_c M_{gt}^{(c)} \log \hat{M}_{\text{sem}}^{(c)},\] where \(M_{gt}^{(c)}\) is the ground-truth binary mask for class \(c\), \(\hat{M}_{\text{sem}}^{(c)}\) denotes the predicted semantic probability, and \(\omega_c\) represents class-specific weights applied to emphasize critical topological features such as drivable areas and lane boundaries. Ultimately, this localized representation provides explicit physical references regarding road topology, serving as a crucial foundation for compliant trajectory planning.

Agent Head. To capture dynamic objects, we follow the DETR paradigm [24] using a Transformer decoder equipped with \(N_{\text{agent}}=30\) learnable queries. A MLP head decodes these queries into continuous agent predictions: \[\hat{Y}_{\text{agent}} = \{\hat{b}_i, \hat{p}_i\}_{i=1}^{N_{\text{agent}}},\] where \(\hat{b}_i\) parameterizes the predicted oriented 2D bounding box (position, size, and heading) and \(\hat{p}_i\) indicates the predicted classification confidence. We employ the Hungarian algorithm to establish an optimal bipartite matching \(\sigma\) between the predicted queries and the ground-truth obstacles. For the matched positive samples, we align the predictions with the ground-truth bounding box state \(b_{gt}\) and the target class label \(p_{gt}\). The total agent loss is a weighted sum of an \(L_1\) spatial regression loss and a Binary Cross-Entropy (BCE) classification loss: \[\mathcal{L}_{\text{agent}} = \lambda_{\text{reg}}\mathcal{L}_{L1}(\hat{b}_{\sigma}, b_{gt}) + \lambda_{\text{cls}}\mathcal{L}_{BCE}(\hat{p}_{\sigma}, p_{gt}),\] where \(\lambda_{\text{reg}}\) and \(\lambda_{\text{cls}}\) are hyperparameter coefficients used to balance the two loss terms.

Through this multi-task supervision, the spatial stream is strictly regularized to encode metric layouts and spatial boundaries rather than arbitrary visual textures. Finally, the intermediate representation \(V\) is linearly projected to form \(V_{\text{spatial}}\), a dense, geometrically grounded feature. This representation serves as a rigorous spatial constraint for the downstream Dual-Stream Planning Adapter, guaranteeing that the generated trajectories are physically feasible and strictly aligned with environmental constraints.

2.3 Dual-Stream Planning Adapter↩︎

To bridge high-level semantic intent with low-level physical constraints, we introduce the Dual-Stream Planning Adapter, a cascaded multi-stage cross-attention decoder. This module progressively fuses representations from the semantic and spatial streams to produce trajectories that are both semantically aligned and physically feasible.

We initialize a set of learnable planning tokens \(P^{(0)} \in \mathbb{R}^{M \times d}\), where \(M=8\) represents the future trajectory waypoints and \(d\) is the hidden dimension. The Adapter consists of \(N\) stacked blocks. Each block refines the planning tokens through two sequential stages:

2.3.1 Semantic and State Alignment↩︎

Inspired by hierarchical bridging strategies [13], the cross-modal attention mechanism in this stage establishes coarse-grained navigation intent by leveraging the commonsense reasoning capabilities of the VLM. Specifically, the planning tokens are aligned with multi-scale semantic features \(V_{\text{sem}}\) and ego-state memory \(E_{\text{mem}}\) to capture both contextual intent and dynamic constraints. Given \(P^{(l-1)}\) as queries, we perform parallel cross-attention and self-attention operations: \[\begin{align} P_{\text{raw}} &= \mathrm{MHCA}_1(P^{(l-1)}, V_{\text{sem}}, V_{\text{sem}}), \\ P_{\text{ego}} &= \mathrm{MHCA}_2(P^{(l-1)}, E_{\text{mem}}, E_{\text{mem}}), \\ P_{\text{sa}} &= \mathrm{MHSA}(P^{(l-1)}). \end{align}\]

A learnable gating parameter \(g \in \mathbb{R}^{d}\) modulates the contribution of the semantic features. The fused intermediate representation is computed via linear projection and concatenation: \[P_{\text{fuse}} = P^{(l-1)} + \mathrm{Linear}\big( [ \tanh(g) \odot P_{\text{raw}} ; P_{\text{ego}} ; P_{\text{sa}} ] \big).\]

2.3.2 Visual Spatial Refinement↩︎

While \(P_{\text{fuse}}\) successfully captures high-level semantic intent, directly decoding it often yields physically infeasible trajectories due to the inherent lack of explicit spatial boundaries in language models. To rectify this, the second stage introduces a rigorous spatial refinement mechanism. By utilizing the dense geometric features \(V_{\text{spatial}}\) extracted from the Spatial Stream, we inject explicit spatial and geometric priors into the network. This forces the planning tokens to strictly adhere to real-world motion constraints and obstacle boundaries: \[\begin{align} P_{\text{vis}} &= P_{\text{fuse}} + \mathrm{MHCA}_3(P_{\text{fuse}}, V_{\text{spatial}}, V_{\text{spatial}}), \\ P^{(l)} &= P_{\text{vis}} + \mathrm{FFN}(P_{\text{vis}}). \end{align}\]

2.3.3 Trajectory Decoding and Multi-Task Objective↩︎

After \(N\) cascaded blocks, the final planning tokens \(P^{(N)}\) are processed by a lightweight MLP decoder to predict the future ego-vehicle trajectory \(\hat{Y} \in \mathbb{R}^{M \times 3}\). The trajectory prediction is supervised against the ground-truth trajectory \(Y_{\text{gt}}\) using an \(L_1\) regression loss. To ensure physical feasibility and rider comfort, this objective is augmented with kinematic smoothness constraints that explicitly penalize excessive acceleration (\(a\)) and jerk (\(j\)): \[\mathcal{L}_{\text{plan}} = \mathcal{L}_{L1}(\hat{Y}, Y_{\text{gt}}) + \lambda_{\text{smooth}} \sum_{x \in \{a, j\}} \mathcal{L}_{\text{SmoothL1}}(x, 0).\]

The entire \(\mathrm{S}^2\)-VLA framework is trained end-to-end using a joint multi-task objective function that balances planning, dynamic agent perception, and static map grounding: \[\mathcal{L}_{\text{total}} = \lambda_{\text{plan}}\mathcal{L}_{\text{plan}} + \lambda_{\text{agent}}\mathcal{L}_{\text{agent}} + \lambda_{\text{map}}\mathcal{L}_{\text{map}}.\]

3 Experiments↩︎

3.1 Dataset and Metrics↩︎

We conduct end-to-end planning training and evaluation on the NAVSIM benchmark [25], which emphasizes closed-loop interactions between the agent and dynamic environments.

NAVSIM introduces the Predictive Driver Model Score (PDMS) as a comprehensive metric for evaluating closed-loop performance, showing strong correlation with standard driving metrics. PDMS is computed based on five criteria: No Collision (NC), Drivable Area Compliance (DAC), Time to Collision (TTC), Comfort (Comf), and Ego Progress (EP).

3.2 Implementation Details↩︎

We employ a three-stage training procedure. First, we perform Supervised Fine-Tuning (SFT) on the pre-trained VLM using the ReCogDrive VQA dataset [15] for 3 epochs, significantly enhancing its instruction-following and spatial reasoning capabilities in complex traffic scenarios. Second, with the VLM backbone frozen, we apply LoRA [26] and train the intent extraction and auxiliary perception modules from scratch for 4 epochs, temporarily decoupling the visual spatial features from the Dual-Stream Planning Adapter. In the final stage, we freeze the perception and intent components, dedicating 4 epochs of optimization exclusively to the visual refinement module to align generated trajectories with precise spatial constraints. The entire framework is optimized for end-to-end trajectory planning on the NAVSIM benchmark [25]. All experiments are conducted using the AdamW optimizer on 4 NVIDIA A100 GPUs. Detailed hyperparameters are summarized in Table 1.

4 Results↩︎

4.1 Quantitative Results↩︎

We evaluate \(\mathrm{S}^2\)-VLA against state-of-the-art E2E, VLMs and VLA methods on the NAVSIM closed-loop planning benchmark. To ensure a rigorous and fair comparison, all models reported in Table ¿tbl:tab:navsim? are evaluated strictly under their respective supervised training paradigms (i.e., behavioral cloning for E2E models and supervised fine-tuning for VLMs/VLAs). We intentionally exclude post-training strategies from the baselines, such as closed-loop reinforcement learning via Group Relative Policy Optimization (GRPO), to ensure a fair comparison of the network designs.

Comparison with E2E Models. As shown in Table ¿tbl:tab:navsim?, \(\mathrm{S}^2\)-VLA achieves a PDMS of 87.1 using only a single front-view camera. It substantially outperforms vision-only counterparts such as UniAD (83.4) and PARA-Drive (84.0), and even surpasses established LiDAR-fused methods like ARTEMIS (87.0) and DRAMA (86.9). This indicates that our dedicated spatial stream effectively infers robust 3D structural constraints directly from 2D imagery, significantly mitigating the inherent depth ambiguity of monocular vision.

While DiffusionDrive achieves a marginally higher PDMS of 88.1, maintaining a slight lead over our method primarily in Drivable Area Compliance, it is important to note the architectural differences. DiffusionDrive relies on an iterative, computationally heavy diffusion decoder and explicit LiDAR point clouds, which naturally provide absolute physical boundaries to circumvent 2D scale ambiguity. The fact that \(\mathrm{S}^2\)-VLA achieves highly competitive performance without relying on expensive 3D sensors or iterative decoding underscores the sheer representational efficiency of our decoupled semantic-spatial architecture.

Comparison with VLM/VLA Models. Compared to a naive VLM baseline, which utilizes the same InternVL3-2B backbone but relies on autoregressive text-head decoding for trajectory prediction, \(\mathrm{S}^2\)-VLA improves PDMS by 3.0 points. This substantial gain directly exposes the semantic-geometric misalignment problem inherent in standard VLMs: when trajectories are generated solely via discrete language tokens, continuous spatial information is inevitably quantized and degraded, leading to geometrically imprecise plans. Our dual-stream design resolves this by routing geometric reasoning through a dedicated spatial pathway, successfully preserving the structural fidelity that language abstraction otherwise discards. Furthermore, \(\mathrm{S}^2\)-VLA outperforms recent methods like ReCogDrive (86.5) and ImagiDrive (86.4), which attempt to enhance VLA planning through enriched textual reasoning chains or imagination-augmented training. Notably, our model achieves a higher PDMS despite utilizing a significantly smaller backbone.

A breakdown of the sub-metrics further highlights the nature of these improvements. \(\mathrm{S}^2\)-VLA achieves the highest No Collision and Ego Progress scores among all evaluated VLM/VLA methods, reflecting superior collision avoidance and forward progress. These safety-critical gains stem directly from explicit spatial grounding: the injected geometric priors enable highly accurate perception of obstacle proximity and road curvature, which are critical physical boundaries where discrete token representations fundamentally fail.

4.2 Qualitative Analysis↩︎

We present qualitative trajectory comparisons between \(\mathrm{S}^2\)-VLA and three representative baselines on the NAVSIM navtest benchmark in Fig. 4.

Despite having access to explicit LiDAR data, the E2E Transfuser[27] baseline exhibits unsafe behaviors, such as curb encroachment and boundary violations in complex road topologies (Fig. 4 (a)). This indicates that raw multi-modal sensor fusion, without sufficient high-level semantic understanding, struggles in scenarios that demand a synthesis of geometric precision and contextual reasoning.

InternVL3-2B [20], the VLM baseline exhibits systematic geometric deviations during high-curvature turns and at intersections (red circles in Fig. 4 (b)), frequently failing to maintain lane centering. This failure pattern highlights the inherent limitation of representing continuous spatial geometry via discrete language tokens: the quantization of spatial coordinates into vocabulary entries introduces irrecoverable rounding errors. These errors inevitably accumulate along the trajectory horizon, producing progressively larger deviations from the intended path.

ReCogDrive[15] attempts to improve planning by injecting VLM-derived cognitive tokens into a diffusion-based planner. However, its long-range predictions still diverge from the ground truth, particularly in high-curvature turns and intersections lacking clear lane markings (Fig. 4 (c)). This perfectly illustrates our core motivation: the deep feature compression inherent to standard VLM backbones inherently destroys the fine-grained spatial details and structural priors required for accurate, boundary-aware trajectory planning.

In contrast, \(\mathrm{S}^2\)-VLA consistently generates trajectories that closely align with expert demonstrations across diverse and complex scenarios. The planned paths remain smooth, stable, and strictly adherent to physical road structures, even over long-range horizons featuring occlusions or degraded lane markings. This robustness directly validates our core architectural hypothesis: by explicitly decoupling high-level semantic reasoning from low-level geometric perception, and seamlessly fusing them via the Dual-Stream Planning Adapter, the model successfully marries contextual scene understanding with precise spatial grounding. Ultimately, \(\mathrm{S}^2\)-VLA achieves a synergistic balance of capabilities that neither a pure VLM nor a pure geometric planner can provide alone.

4.3 Ablation Study↩︎

Table ¿tbl:tab:ablation95study? presents a systematic ablation study that isolates the contribution of each component, highlighting their contribution in semantic reasoning, geometric precision, and spatial grounding, and thereby validating the design of \(\mathrm{S}^2\)-VLA.

Introducing the planning adapter with multi-level semantic features improves PDMS by +1.5 (84.1 \(\rightarrow\) 85.6). The gains are most pronounced in Ego Progress (EP, +1.1) and Time to Collision (TTC, +1.2), indicating enhanced long-horizon planning and hazard anticipation. By capturing both high-level intent and contextual scene understanding, the hierarchical semantic features enable better coordination between immediate actions and long-term objectives. However, the relatively modest gain in Drivable Area Compliance (DAC, +0.9) confirms that semantic reasoning alone is insufficient for precise geometric alignment.

To address this, adding the ViT-based spatial stream further increases the PDMS to 86.2 (+0.6), with the most significant improvement observed in DAC (94.0 \(\rightarrow\) 94.5). This highlights the complementary role of the spatial stream: by preserving fine-grained geometric details via spatially dense visual features, it explicitly enforces structural alignment. Consequently, the model achieves superior lane adherence and boundary awareness, effectively mitigating the geometric limitations of purely semantic representations. Achieving this empirical gain as a residual refinement atop a converged semantic baseline also validates our cascaded architectural ordering. The design mirrors the natural factorization of trajectory planning, which establishes intent before verifying geometric feasibility. A parallel alternative would instead force planning tokens to compete across heterogeneous keys, diluting each stream’s specialization and recreating the very entanglement \(\mathrm{S}^2\)-VLA is designed to prevent.

Finally, incorporating auxiliary perception tasks yields a further +0.9 PDMS gain, reaching a final score of 87.1. Notably, the largest improvements occur in safety-critical metrics, including No Collision (NC, +0.3) and EP (+1.0). These gains are driven by explicit multi-task supervision comprising both BEV map generation and agent prediction. This dual objective ensures that the latent representations successfully capture both static geometric boundaries and dynamic obstacle trajectories, providing a holistic physical prior for the downstream planning adapter.

5 Conclusion↩︎

This letter introduced \(\mathrm{S}^2\)-VLA, a Semantic-Spatial Dual-Stream Vision-Language-Action framework designed for end-to-end autonomous driving. To bridge the fundamental gap between high-level semantic reasoning and low-level physical execution in standard VLMs, we developed a dedicated multi-modal planning adapter. This adapter seamlessly fuses multi-scale semantic features from the language backbone with uncompressed, fine-grained geometric features extracted directly from the visual encoder. By further integrating BEV agent state prediction and semantic mapping as auxiliary supervision, we explicitly inject spatial and geometric priors into the network. Evaluations on the NAVSIM benchmark demonstrate that \(\mathrm{S}^2\)-VLA successfully mitigates the spatial perception deficiencies of traditional VLMs, achieving highly competitive closed-loop performance under pure supervised fine-tuning. Ultimately, this research confirms that preserving continuous spatial features, bypassing the quantization bottleneck of discrete language heads, is critical for generating safe and dynamically feasible driving trajectories.

Despite its effectiveness, our framework has two limitations, each suggesting a natural direction for future work. First, the dense extraction and dual-stream fusion of spatial features incur non-trivial computational overhead; we aim to develop sparser feature-representation mechanisms that replace global attention fusion and inject richer geometric priors at lower cost. Second, because our dual-stream architecture is orthogonal to the training paradigm, augmenting it with closed-loop reinforcement learning post-training offers a natural and compatible next step.

References↩︎