1 Introduction↩︎

Autonomous Driving (AD) has long been a challenging application [1]. The previous dominant paradigm relied on a modular design, decomposing the driving process into perception [2]–[4], prediction [5]–[7], and planning [8]–[10] modules. While this structure offers interpretability and controllability, it suffers from error propagation and limited cross-task reasoning [11]. Recent advances in Multi-modal Large Language Models (MLLMs) [12]–[14] have demonstrated strong generalization and multi-task capabilities, inspiring a shift toward unified models that map raw sensor inputs to feasible trajectory planning results directly.

In this trend, Vision–Language–Action (VLA) models have emerged as a powerful framework, integrating perception, prediction, and planning in a unified manner [11], [15]–[19]. These models typically take RGB camera images as visual inputs and predict the vehicle’s future trajectory over several seconds at 0.5 s intervals. In terms of model design, the paradigm formulates trajectory prediction as a decoding task, where the MLLM-based planner either auto-regressing text-based waypoints [11], [20]–[24], or generating trajectories via an auxiliary planner\(, e.g.,\) a diffusion model,  [16], [17], [25], [26]. In terms of training, recent advanced AD-VLA models [15], [17], [23], [25], [27], [28] typically adopt a three-stage training pipeline: (i) Pretraining on Visual Question Answer (VQA) data [20], [27] to inject driving priors; (ii) Imitation Learning (IL) with Supervised Fine Tuning (SFT) on Chain-of-Thought (CoT) reasoning data and expert trajectories. These two stages are common across nearly all multi-stage VLA frameworks. (iii) More recently, the Reinforcement Learning (RL) stage with GRPO has been introduced by currently advanced works [11], [17], [23], [29] to further align AD-VLA’s predicted trajectories with human driving preferences.

Although AD-VLAs can explain their intents, meta-behaviors, e.g., turning left/right, and even their trajectory planning decisions in a linguistically interpretable manner, a fundamental gap between perception and planning exists: Although VQA or CoT reasoning data introduces general traffic priors, textual contexts alone can not truly equip VLAs with the spatial and behavioral knowledge required for autonomous driving. This is due to their supervision being inherently text-centric, focusing on semantic question answering rather than planning-related perception. Consequently, the learned representations emphasize linguistic semantics instead of motion and spatial dependencies essential for trajectory planning. Ultimately, the model learns to name what is seen rather than understand what should see and where should look for reliable trajectory generation. As shown in Figure 1 (a), attention learned from purely textual traffic-knowledge SFT is scattered across the scene with no clear spatial grounding, making it difficult to interpret. Quantitative analysis is further provided in Analysis 1 of Section [sec:ablation].

In this paper, we propose DriveTeach-VLA, a framework that operates in a dual-model manner consisting of a TGP-Prompter and a TGP-Planner, both built upon Qwen2.5-VL-3B. The overall pipeline of DriveTeach-VLA is shown in Figure 2. The TGP-Prompter undergoes vision-enhanced pretraining and is responsible for generating feasible driving region guidance, while TGP-Planner is conditioned on the guidance to predict future trajectory. To achieve “teach what to see”, i.e., which objects are relevant for driving, the TGP-Prompter is trained with Driving-aware Vision Distillation (DVD). Specifically, DVD first employs Grounding DINO [30] to detect traffic-critical objects. The detected bounding boxes are then overlaid on the images as visual prompts, forming so-called bbox-augmented images. Then we apply self-distillation [31] on the ViT encoder of the TGP-Prompter: a teacher ViT processes bbox-augmented images while a student ViT processes the corresponding raw images. Their representations are aligned in the latent space, enabling the student encoder to internalize traffic-relevant visual cues without relying on textual supervision. To achieve “teach where to look”, i.e., which spatial region is most relevant for planning in the current scene, we further introduce a 2D image Trajectory-Guided Prompt (2D-TGP), which projects trajectories from world (BEV) coordinates onto the image plane with a pinhole camera model [32]. These 2D-TGP trajectories can provide spatial guidance about feasible driving regions for the AD-VLA. Thus, the TGP-Prompter’s decoder output is supervised with ground-truth 2D-TGP trajectory derived from expert trajectories via SFT.

The TGP-Planner is initialized from the TGP-Prompter to obtain the traffic prior and trained to predict trajectories conditioned on the 2D-TGP. Following [11], [20], [24], we annotate step-by-step CoT reasoning data with Qwen2.5-VL-72B. The TGP-Planner learns to generate intermediate CoT reasoning steps and produce the future trajectory under the guidance of 2D-TGP. Training is performed with SFT followed by GRPO. To provide stable spatial guidance during training, we employ teacher forcing by supplying ground-truth 2D-TGP as the condition for the TGP-Planner in training. While in test, to avoid the information leak, the 2D-TGP is first generated by the TGP-Prompter and subsequently fed into the TGP-Planner to produce the future trajectory.

DriveTeach-VLA fully harnesses the intrinsic visual understanding ability of MLLMs and aligns it with driving-aware spatial reasoning for trajectory planning, achieving more precise trajectory prediction. Our main contributions are summarized as follows:

• We identify that text-centric training in existing VLA models often produces spatially ungrounded attention.

• DriveTeach-VLA introduces DVD and 2D-TGP to explicitly teach VLAs what to see and where to look. For improving what to see, DVD transfers traffic visual cues via self-distillation from bbox-augmented images. For improving where to look, 2D-TGP projects expert trajectories onto the image plane to provide spatial guidance for planning.

• DriveTeach-VLA achieves the State-of-The-Art (SoTA) performance on Navsim and nuScene benchmarks.

2 Related Work↩︎

AD-VLA Model. Recent AD-VLAs formulate trajectory prediction as a decoding task, where a MLLM planner generates a future few seconds’ trajectory conditioned on visual–language context. The paradigm follows two major branches. The first relies on VLM itself’s autoregressive scheme, where the model sequentially regresses the trajectory. ImpromptuVLA [20], AdaThinkDrive [23], and other previous models [21], [22] regress text form trajectory directly. Poutine [24] further explores a more suitable prompt and CoT reasoning steps for VLA’s text waypoint trajectory prediction. AutoVLA [11] follows this scheme but instead replaces text-form trajectory with action tokens that discretize each dimension of the trajectory into several bins. The second relies on an additional planner to produce continuous trajectories conditioned on the latent reasoning features of the LLM. AsyncDriver [33], RecogDrive [17], and Orion [16] are built upon VLM’s encoded features and introduce (query-based) MLP/Diffusion to predict the future trajectory. More recent works, such as DriveMoe [26] and DriveVLA-W0 [18], employ a Mixture-of-Expert (MoE) architecture that pairs the VLM with an additional action-expert transformer by mutual attention to process multi-view images or reduce the inference time. The proposed DriveTeach-VLA follows the text-waypoint trajectory autoregressive scheme to explore the native capabilities of the MLLM backbone in the autonomous driving domain.

AD-VLA Training. Some AD-VLAs directly finetune MLLMs for trajectory prediction [21], [22]. To achieve stronger generalization, recent studies have converged toward a multi-stage training pipeline including: (i) VQA Pretraining. VQA pairs are firstly pseudo-labeled by MLLM, such as Qwen2.5-VL-72B and used to pretrain VLA with instruction learning. These VQA pairs involve multiple traffic-relevant tasks including scene description, traffic condition reasoning, and action explanation. DriveLM [27] systematically organizes these tasks into a Perception–Prediction–Explanation–Behavior framework, which has since become the foundation for many subsequent VLA studies [15], [17], [23], [28]. (ii) SFT-IL. The model is supervised by expert trajectories and pseudo-labeled CoT reasoning steps with SFT to enable end-to-end trajectory generation from sensory inputs. (iii) RL. AutoVLA [11], RecogDrive [17], and AdaThinkDrive [23] introduce GRPO-based [34] reinforcement optimization. Especially, EVADrive [29] further proposes multi-objective RL framework, i.e., APO, for trajectory planning. The RL-stage aligns model prediction with human driving preference, including comfort, safety, and compliance with traffic regulations. DriveTeach-VLA replaces text-centric VQA pretraining stage with vision-centric DVD pretraining and augments IL and RL stages with 2D-TGP, enabling vision-grounded reasoning for reliable trajectory planning.

3 Method↩︎

In this section, we first formulate the pipeline of the proposed method in Section [sec:formulation]. Then, we introduce DVD pretraining and TGP-Guided learning/inference in Section [sec:dvd] and Section [sec:tgp], respectively.

3.1 Preliminary↩︎

BEV trajectory, 2D-TGP trajectory, and pinhole camera model. An expert trajectory with \(T\) time-steps in world (BEV) coordinates is denoted as \(\mathcal{T}_{w}=\{\mathbf{a}_t=(x_t, y_t, \psi_t ) \mid t=1,2,\ldots,T \}\), where \(x_t\) and \(y_t\) denote the position offsets in the ego-centric coordinate frame and \(\psi_t\) denotes the yaw (heading) angle of the ego vehicle. Given the front camera intrinsic matrix \(K\) and extrinsic matrix \(\left[ R \mid t \right]\), the \((x_t, y_t)\) in ego-centric coordinates can be projected onto the 2D image plane \((x_t^{I}, y_t^{I})\) using the pinhole camera model \(\pi(\cdot)\) [32]: \[~\label{eq:bev-to-tgp} \begin{align} &(x_t^{I}, y_t^{I}) = \pi\!\left(K, [R \mid t], (x_t, y_t)\right), \mathcal{T}_{I} = \{\mathbf{a}^{I}_{t}=(x^I_t, y^I_t, \psi_t) \mid t=1,2,\ldots,T \}, \\ &P_I = \text{Text}\left[(x^I_1, y^I_1), (x^I_2, y^I_2), \ldots, (x^I_T, y^I_T)\right], \end{align}\tag{1}\] where \(P_I\) is named ground-truth 2D-TGP, and \(\mathcal{T}_{I}\), combining the projected offset points with yaw angle, is named ground-truth 2D-TGP trajectory. The yaw angle \(\psi_t\) is retained from \(\mathcal{T}_w\) so that the complete BEV trajectory can be recovered from the 2D-TGP trajectory via the inverse projection \(\pi^{-1}\) (See Eq. 2 below).

In general, the \(\pi(\cdot)\) is not invertible due to the loss of depth information. However, in AD scenarios, trajectory points are typically assumed to lie on the ground plane \((z=0)\) and thus the projection becomes invertible: \[~\label{eq:tgp-to-bev} (x_t, y_t) = \pi^{-1}\!\left(K, [R|t], (x_t^{I}, y_t^{I})\right) \quad \text{s.t.}\quad z=0 ,\tag{2}\] which enables recovering a unique BEV position from the image-space trajectory and can be used to verify the quality of the estimated 2D-TGP trajectory (\(\psi_t\) should be predicted separately for 2D-TGP trajectory evaluation).

To establish a more intuitive understanding, we visualize the 2D-TGP trajectory on the image in Figure 4. By providing the MLLM with 2D-TGP coordinates, we can better align its intrinsic grounding ability with the VLA driving task.

Task formulation. Given observed camera images \(C\), ego-state \(S\), language instruction \(L\), AD-VLA models output BEV trajectory estimation \(\hat{\mathcal{T}}_w\) for planning: \[~\label{eq:AD-VLA-formulate} \hat{\mathcal{T}}_w = \text{AD-VLA}(C, S, L).\tag{3}\] Different from previous AD-VLAs formulating trajectory prediction as Eq. 3 , DriveTeach-VLA consists of two models, including TGP-Prompter and TGP-Planner, both of which are built upon Qwen2.5-VL-3B. TGP-Prompter firstly output 2D-TGP estimation \(\hat{\mathcal{T}}_{I}\) as spatial guidance, serving as the textual condition for TGP-Planner to estimate BEV trajectory: \[~\label{eq:driveteach-vla-formulation} \hat{P}_I, \hat{\mathcal{T}}_I=\text{TGP-Prompter}(C, S, L_{\text{TGP}}), \quad \hat{\mathcal{T}}_{w} =\text{TGP-Planner}(C, S, [L; \hat{P}_I]),\tag{4}\] where \(L_{\text{TGP}}\) and \(L\) is the language instruction for TGP-Prompter and TGP-Planner, detailed in Supp. Figure 8 and Supp. Figure 9, respectively. Such textual coordinates conditions \(\hat{P}_I\) are reasonable due to the intrinsic grounding ability of MLLMs, as 2D coordinates are naturally interpretable to the MLLM model.

Furthermore, we decouple the TGP-Prompter and TGP-Planner into two separate models to avoid instruction confusion, as 2D-TGP generation and BEV trajectory prediction have distinct optimization objectives despite sharing similar input modalities.

3.2 TGP-Prompter: DVD Pretraining↩︎

  DVD pretraining is specially designed for the TGP-Prompter to inject traffic priors without reliance on VQA pairs. Existing VLA models typically employ ViT encoders pretrained on natural images, which lack the domain knowledge about what should be seen during driving, e.g., lanes, traffic participants, and scene layout. To address this issue, DVD firstly introduces the bbox-augmented image self-distillation for the ViT encoder of the VLA model, enabling it to learn driving-critical visual cues. In addition, to achieve precise driving-aware spatial perception regarding where to look, DVD pretraining also includes SFT that trains the VLA decoder to estimate the 2D-TGP trajectory. See Figure 3 (left) for the overall training framework, and detailed elaborations are as follows:

Bbox-augmented image self-distillation for ViT encoder. We firstly employ an open-vocabulary detector Grounding Dino [30] to detect traffic-critical objects, including car, truck, bus, trailer, construction vehicle, pedestrian, motorcycle, bicycle, barrier, traffic element, and traffic light. The complete grounding object lists can be found in the Supp. Figure 6. By overlaying the detected bounding boxes on the raw camera image \(C\), we can obtain the bbox-augmented image \(C_{\text{bbox}}\). The bbox in Figure 4 presents the object detection results. The self-distillation involves a pair of student and teacher ViT encoders, both initialized from the same MLLM’s ViT encoder. The teacher ViT receives \(C_{\text{bbox}}\) while the student ViT is fed with \(C\): \[\text{ViT}_t(C_{\text{bbox}})=\left[v_1^t,\cdots,v_N^t\right], \quad \text{ViT}_s(C)=\left[v_1^s,\cdots,v_N^s\right],\] where \(N\) is the image patch token number, and \(\text{ViT}_s\)/\(v_i^s\), and \(\text{ViT}_t\)/\(v_i^t\) denotes student and teacher encoder/their patch features, respectively. In terms of feature alignment between \(v_i^t\) and \(v_i^s\) , block-wise alignment is employed to capture broader contextual information brought by the bounding boxes. Specifically, we partition the feature maps into \(K\) non-overlapping blocks \(\mathcal{B}_k\) (i.e., dividing the feature map into a grid of rows \(\times\) columns, e.g., \(2\times4\) yields \(K{=}8\) blocks) and compute block-wise alignment loss \(\mathcal{L}_{\text{distill}}\): \[\bar{v}_k^{t} = \frac{1}{|\mathcal{B}_k|}\sum_{i \in \mathcal{B}_k} v_i^{t}, \quad \bar{v}_k^{s} = \frac{1}{|\mathcal{B}_k|}\sum_{i \in \mathcal{B}_k} v_i^{s}, \quad \mathcal{L}_{\text{distill}} = \frac{1}{K}\sum_{k=1}^{K}\text{Smooth-L1}(\bar{v}_k^{t}, \bar{v}_k^{s}),\] where Smooth-L1 loss provides robustness to local spatial outliers [35].

2D-TGP trajectory SFT for MLLM decoder. We first convert expert trajectories to 2D-TGP trajectories according to Eq. 1 . Then the TGP-Prompter is supervised to predict the 2D-TGP trajectory via optimizing the SFT cross-entropy loss: \[\mathcal{L}_{{\text{TGP}}} = -\frac{1}{T} \sum_{t=1}^T \log P_\theta( \mathcal{T}_{I}^{(t)} \mid C, L_{\text{TGP}}, \mathcal{T}_{I}^{(<t)}),\] where \(P(\cdot)\) is the TGP-Prompter policy, \(L_{\text{TGP}}\) is language instruction (detailed in Supp. Figure 8), and \(\mathcal{T}_{I}^{(<t)}\) denotes the trajectory action sequence before step \(t\), and \(\mathcal{T}_{I}^{(t)}\) denotes the action at step \(t\).

Training Objective. The overall training loss is defined as: \[\mathcal{L}_{\text{DVD}} = \mathcal{L}_{\text{distill}} + \lambda_{\text{TGP}} \cdot \mathcal{L}_{\text{TGP}},\] where \(\lambda_{\text{TGP}}\) is a hyperparameter. During training, the encoded features from the student ViT are fed into the MLLM decoder to make a prediction. The weights of the student ViT and the MLLM decoder are updated through backpropagation, while the teacher ViT is updated with Exponential Moving Average (EMA) from student ViT following the classic self-distillation setup from DINO [31].

3.3 TGP-Planner: TGP-guided Learning and Reasoning↩︎

  The TGP-Planner is firstly initialized from the TGP-Prompter to inherit driving-related priors. It is then trained to generate step-by-step CoT reasoning and predict future trajectories in BEV space, conditioned on the 2D-TGP, the observed camera image, and the driving instruction. Detailed processes are as follows.

TGP-guided CoT Reasoning. Following Poutine [24], we generate pseudo-labeled reasoning data using the Qwen2.5-VL-72B model. The first three sub-tasks are identical to those in Poutine, while the fourth is modified to suit the 2D-TGP condition:

Task 1: Critical Object Detection. Determine whether any critical instance from a predefined object class list (e.g., hazards, cyclists, conflicting vehicles) may influence the ego vehicle’s future trajectory.

Task 2: Natural Language Explanation. Provide a concise explanation of the expert trajectory and the reasoning behind the expert driver’s behavior.

Task 3: Meta-Behavior Selection. Select a single behavior (e.g., turning, acceleration/deceleration, lane-following) that best describes the trajectory.

Task 4: Future Trajectory Prediction. Different from prior work, we introduce spatial guidance via 2D-TGP, where projected trajectory keypoints \(\{(x_1^I,y_1^I),\dots,(x_8^I,y_8^I)\}\) indicate regions along the feasible driving path that the model should attend to. Task 4 is modified as: given the observed camera image, ego-state, reasoning outputs, and 2D-TGP keypoints, predict the optimal 4-second future trajectory.

Supp. Figure 7 and Supp. Figure 9 details the prompts used for pseudo-labeling and TGP-Planner, respectively. We supervise TGP-Planner with above CoT reasoning data by SFT. After CoT-SFT training, we further align TGP-Planner’s trajectory prediction with human driving preferences using reinforcement learning, following AutoVLA [11]. At each rollout, the predicted trajectory \(\hat{\mathcal{T}}_w\) is evaluated by the PDM-Score (PDMS) [36]:

\[~\label{eq:pdms} \text{PDMS} = \prod_{c \in C} c \times \frac{\sum_{m \in M} w_m \cdot m}{\sum_{m \in M} w_m},\tag{5}\]

where \(C = \{\text{NC, DAC}\}\) (No-at-fault Collisions, Drivable Area Compliance) and \(M = \{\text{EP, TTC, C}\}\) (Ego Progress, Time to Collision, Comfort) with weights \(w_m = \{5, 5, 2\}\). We follow CuriousVLA [37] to conduct data filtering and rollout for GRPO [34].

Teacher forcing training and Inference pipeline. During CoT-SFT and GRPO-RL, to produce the most precise guidance for driving skills learning, we used the ground-truth 2D-TGP to condition TGP-Planner. While in test (inference), as shown in Figure 2 (left), the 2D-TGP condition is firstly generated by TGP-Prompter, which is subsequently fed into TGP-Planner for future trajectory prediction. Although this introduces a train–test gap, we show in Section [sec:ablation] that the gap is negligible (\(\sim\)​0.4 PDMS), confirming TGP-Prompter provides sufficiently accurate spatial guidance.

4 Experiment↩︎

1pt

4.1 Experimental Setup↩︎

Benchmarks. We evaluate our method on two public autonomous-driving benchmarks: non-reactive closed-loop NAVSIM [36] and open-loop nuScenes [47]. NAVSIM predicts future 4-second trajectory and is designed for intention-changing scenarios where the future motion of the ego vehicle cannot be reliably predicted from history alone. Following RecogDrive[17], we report PDMS results of DriveTeach-VLA on NAVSIM’s navtest. On nuScenes, following UniAD [38] and ST-P3 [45], we report the average L2 trajectory error and collision rate over a 3-second horizon.

Implementation Details. We follow CuriousVLA [37] to use the step-wise normalized text trajectory. Training uses AdamW (lr \(4\times10^{-5}\), weight decay \(0.05\)) with a cosine schedule and 0.10 warm-up. Models are trained with batch size 16 on 8\(\times\)H100 GPUs. DriveTeach-VLA is trained in three stages. (i) DVD pretraining:1 epoch, with block size set to \(2\times 4\) and trajectory-loss weight \(\lambda_{\text{TGP}}=0.1\). EMA momentum is set to \(0.996\). (ii) CoT-SFT: 6 epochs using Chain-of-Thought annotations generated by Qwen2.5-VL-72B. (iii) GRPO-RL: 180 update steps with a group size of 8. Especially, for fair comparison, CoT and GRPO are not employed for nuScene.

4.2 Comparison with the SoTA models.↩︎

NAVSIM Results. We benchmark against recently published end-to-end and VLA-based SoTA models, including UniAD, TransFuser, Hydra-MDP, DiffusionDrive, WoTE, RecogDrive, ImagiDrive-S, and AutoVLA. Among them, AutoVLA is the most similar to DriveTeach-VLA, as both adopt an autoregressive trajectory prediction paradigm. We report results on the Navtest split using PDMS metrics, and results are exhibited in Table ¿tbl:tab:sota95navsim?. DriveTeach-VLA achieves 90.4 PDMS on Navtest with a single inference pass.

NuScenes Results. Table 1 shows the comparison of DriveTeach-VLA with several SOTA methods using the L2 distance error (L2) and collision rate as the primary metrics. DriveTeach-VLA achieves the best performance on both metrics, with an L2 of 0.30 and a collision rate of 0.12, surpassing all prior methods. The model exhibits the highest precision in trajectory prediction with minimal collision occurrence, indicating superior performance in real-world driving scenarios.

4.3 Ablation and Analysis↩︎

Attention Analysis of Vision Encoders. We compare the attention maps of the TGP-Planner during trajectory decoding with a CoT-SFT trained Qwen2.5-VL-3B baseline. The qualitative observations in Figure 1 show that the baseline exhibits dispersed attention, while DriveTeach-VLA produces clear spatial grounding on traffic-critical objects. To quantify this phenomenon, we introduce a metric called Attention Mass (AM). The AM score for an observed camera image \(C\) is calculated as follows. Firstly, we compute the attention maps between the decoded waypoint tokens and the visual patch tokens. The attention maps are averaged across all waypoint tokens to obtain an aggregated attention map \(A\). Then, let \(\mathcal{R}\) denote the set of bounding boxes obtained from the bbox-augmented image \(C_{\text{bbox}}\) (Section [sec:dvd]), and let \(|r|\) denote the area of box \(r\). We sum the attention values of visual tokens lying inside all bounding boxes and normalize by the total box area: \[\label{eq:attn95mass} \mathrm{AM}(C) = \frac{1}{\sum_{r\in\mathcal{R}} |r|} \sum_{r\in\mathcal{R}} \sum_{(i,j)\in r} A_{i,j},\tag{6}\] where \(A_{i,j}\) denotes the average attention value at spatial position \((i,j)\) in the averaged attention map. We randomly sample 1k scenes from Navtrain and compute the mean and variance of their AM scores. Table ¿tbl:tab:vit95attention95stats? reports the results. The CoT-SFT baseline exhibits significantly lower AM than the TGP-Planner and also shows a smaller variance, indicating consistently weak attention on traffic-critical objects. This verifies that DVD pretraining effectively injects critical traffic-object priors into DriveTeach-VLA’s TGP-Planner.

Hyperparameter analysis of \(\lambda_{\text{TGP}}\) and \(\mathcal{B}_k\) As this evaluation only involves DVD pretraining, i.e., TGP-Prompter only, we transform the predicted 2D-TGP trajectories into BEV trajectories using Eq. 2 and calculate PDMS on Navtest. Table ¿tbl:tab:ablation95lambda? studies the hyperparameter \(\lambda_{\text{TGP}}\) balancing 2D-TGP learning and self-distillation in DVD pretraining. A small value (\(0.05\)) under-utilizes trajectory cues, while a large value (\(1.5\)) overemphasizes the auxiliary prediction task. A moderate value (\(0.1\)) achieves the best performance across NC, DAC, TTC, and PDMS.

Table [tbl:tab:ablation95block] analyzes how the block participation strategy \(\mathcal{B}_k\) in DVD affects planning quality. The 2\(\times\)​4 partition achieves the best PDMS among the tested configurations. This is likely because DVD relies on bbox-based visual prompts highlighting traffic-critical objects, and blocks with a larger receptive field during distillation better capture these highlighted regions and their surrounding context. In contrast, patch-level alignment dilutes the signal since most patches fall outside the highlighted bounding boxes.

Ablation on TGP-guided CoT and DVD. The incremental ablation results are reported in Table 2. Ablation begins based on a single Qwen2.5-VL-3B model (84.8 PDMS). We first establish a baseline using conventional VQA pretraining followed by CoT-SFT training (second row, 86.4 PDMS). Next, we introduce DVD pretraining combined with CoT-SFT. Under this setting, only the TGP-Prompter is trained: after completing DVD pretraining, the TGP-Planner model is initialized from TGP-Prompter and further optimized with CoT-SFT (87.1 PDMS). Subsequently, 2D-TGP guided reasoning is employed and brings performance to 88.2. Finally, the GRPO boosts performance to 90.4.

Furthermore, Table 2’s 3-4 rows isolate the performance of TGP-Prompter, and 5-8 rows ablate TGP-guided CoT reasoning with and without GRPO.

6pt

Furthermore, we analyze the effect of 2D-TGP on TGP-Planner’s prediction quality. Following Table ¿tbl:tab:tgp95gap?, we linearly bin samples by predicted \(\hat{\mathcal{T}}_I\)-vs-GT \({\mathcal{T}}_I\)’s L2 error and report PDMS in Figure 5. The small oracle gap (Table ¿tbl:tab:tgp95gap?: 90.4/90.8) is expected: most predicted TGPs are accurate. And when the error grows, PDMS drops gradually, showing robustness rather than TGP irrelevance.

DVD depends heavily on Grounding DINO detection quality. If Grounding DINO misses critical objects in novel scenes or domain-shifted data (e.g., unusual vehicles, occluded pedestrians), the distilled visual priors may be noisy or misleading. So Table [tbl:tab:dvd-robustness] adds random bbox drop and jitter bbox noise on the annotated bboxes and tests TGP-Prompter’s PDM-Score. The noise strength denotes the dropped-box ratio or the corner perturbation ratio w.r.t. box width/height. DVD shows robustness and remains helpful under moderate noise (20% drop/jitter: 86.8/87.1 vs. w/o DVD 86.2), while severe 40% noise can mislead guidance and hurt.

Efficiency and Dual Model Design Ablation. Although DriveTeach-VLA achieves SoTA performance, the dual-model architecture increases runtime latency. As shown in Table 3, Prompter-Planner design increases 1x runtime (about 1.14s on H100) during inference. To assess its necessity, we replace it with a multi-turn QA variant. As shown in Table 3, without GRPO, the Prompter-Planner design improves PDMS over the Prompter-only baseline (88.2 vs. 87.3), while the multi-turn QA variant performs worse (87.0 vs. 87.3). This suggests that jointly learning two instruction-following behaviors in one model remains challenging.

Despite the longer runtime, we also found something surprised. We compare DriveTeach-VLA with its most similar baseline, AutoVLA. Table [tbl:tab:efficiency] reports the runtime results, measured on one L20 GPU. AutoVLA reports results under two settings: predicting text waypoint and action token trajectories. Although action tokens require fewer tokens and are therefore more efficient, AutoVLA relies on long CoT reasoning, which introduces substantial token overhead. Consequently, even its action-token variant remains slower than DriveTeach-VLA. This indicates that DriveTeach-VLA uses fewer CoT reasoning tokens while still outperforming AutoVLA, highlighting the efficiency advantage of our vision-grounded spatial enhancement.

5 Conclusion↩︎

This work identifies a key limitation of existing VLA paradigms: their heavy reliance on text-centric pretraining, which leads to trajectory decoding without explicit semantic guidance. To address this issue, we propose DriveTeach-VLA, a VLA framework that decouples spatial guidance from trajectory planning through a dual-module architecture consisting of a TGP-Prompter and a TGP-Planner. Extensive experiments on NAVSIM and nuScenes demonstrate the effectiveness of the proposed approach, supported by comprehensive ablation studies. Furthermore, despite the dual-model design increases runtime, our efficiency analysis shows that visual spatial enhancement reduces the reliance on reasoning tokens. These results highlight the importance of vision-grounded spatial guidance for improving performance in autonomous driving systems.

6 Acknowledge↩︎

DriveTeach-VLA is funded by the National Key Research and Development Program of China, 2024YFE0217600, Xiongan AI Institute, DiDi and Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001120.

7 NAVSIM Best-of-N PDMS & EPDMS↩︎

Autoregressive model often exhibit strong exploration capability. In Table ¿tbl:tab:sota95navsim95bst95of95n?, we compare the best-of-N performance. For example, AutoVLA reports a best-of-N (BoN=6) PDMS of 92.1 [11]. But BoN strategy leaks test-set information and violates the NAVSIM evaluation rule. DrivorR [48] proposes an alternative approach by training a selector model to identify the best trajectory. Following this strategy, we perform N=12 times rollout for each scene and select the best trajectory using the DrivorR selector. This yields 92.7 PDMS, surpassing prior methods by a large margin.

1pt

8 NuScenes Results↩︎

3pt

9 System Prompt↩︎

DriveTeach-VLA involves four key system prompts, each serving a distinct purpose in DVD pretraining, CoT pseudo-labeling, language instruction for TGP-Prompter, and language instruction for TGP-Planner.

(1) Critical-object detection prompt. We first employ GroundingDINO to detect critical objects from the training set. These pseudo-labeled objects (Fig. 6) are used as the supervision signal for the teacher ViT in DVD stage. Here, the bounding box colors vary across categories. We follow the default GroundingDINO configuration to assign a distinct color to each class.

(2) CoT pseudo-labeling prompt. To obtain reliable reasoning supervision, we generate large-scale pseudo CoT annotations using the Qwen2.5-VL-72B model. This is essential: without providing high-quality reasoning traces, RL tends to collapse. The CoT annotation process covers three tasks—critical-object auditing, natural-language causal explanation, and meta-behavior selection. The system prompt used for generating these annotations is shown in Fig. 7.

(3) Language instruction for TGP-Prompter. During TGP-Prompter SFT learning, the model is required to predict 2D-TGP trajectory. The prompt is in Fig. 8 defining the format of 2D-TGP.

(4) Language instruction for TGP-Planner. This prompt incorporates several structured CoT subtasks to instruct TGP-Planner to reason step-by-step under the guidance of 2D-TGP. The Prompt is in Fig. 9.

References↩︎