1 Introduction↩︎

The integration of Vision-Language Models (VLMs) [1]–[5] has catalyzed a profound paradigm shift in autonomous driving, transitioning the field from traditional, modular perception-planning pipelines [6]–[9] toward end-to-end holistic reasoning frameworks [10], [11]. By leveraging extensive world knowledge and powerful visual-semantic alignment, VLMs demonstrate remarkable potential in interpreting complex traffic scenes and understanding nuanced dynamic interactions. However, deploying these models directly into safety-critical environments remains an open challenge. Current methodologies predominantly rely on standard Supervised Fine-Tuning (SFT) [12] to align pre-trained VLMs with driving-specific instructions. Unfortunately, standard SFT merely mimics the surface-level statistical distribution of human driving trajectories without genuinely internalizing deep logical deduction capabilities. Consequently, in complex dynamic scenarios, SFT models frequently suffer from severe reasoning hallucinations and conservative biases. As illustrated in 1 (a), a standard SFT model might easily misinterpret ambiguous visual cues, leading to suboptimal or overly cautious decisions, such as unnecessary deceleration, which disrupts traffic flow and fundamentally compromises safety.

To mitigate these inherent reasoning deficiencies, recent state-of-the-art methodologies have increasingly turned to augmenting VLMs with Chain-of-Thought (CoT) [13] processes and explicit external tool calls. Frameworks such as AgentThink [14] and OmniDrive-R1 [15] attempt to actively ground visual elements and verify intermediate logic by interleaving multi-modal reasoning with dynamic Application Programming Interface (API) calls, such as open-vocabulary detectors or depth estimators. However, as depicted in 1 (b), this tool-augmented paradigm fundamentally compromises the robustness of the end-to-end architecture. It introduces a fragile dependency pipeline where a single point of failure or a noisy output from an external module can cascade, causing the entire decision-making process to collapse or produce hazardous directives. Furthermore, generating hundreds of explicit reasoning tokens imposes prohibitive computational overhead, rendering these prolonged System-2 approaches impractical for real-time vehicular deployment.

To overcome the fragility of tool-augmented pipelines, we introduce a new Critique-Driven Multi-Turn Reinforcement Learning (RL) paradigm in the second stage of our CritiqueDriveVLM framework. Instead of relying on brittle external perception APIs, we construct a frozen Multi-Dimensional Verifier that strictly evaluates the model’s reasoning trajectories across perception, logic, and safety dimensions. As illustrated in 1 (c), this verifier acts as a logical guide during inference, providing targeted natural language critiques that allow the Teacher model to engage in a multi-turn interactive loop, iteratively refining its initial thoughts and correcting hallucinations. Concurrently, during the Group Relative Policy Optimization (GRPO) [16] training phase, the verifier provides fine-grained scalar rewards, while a step-decay multi-turn penalty explicitly forces the policy to internalize the logical deduction process. This cultivates a highly reliable Teacher model that achieves profound System-2 reasoning and robust decision-making through verifier-guided refinement.

While our Stage-2 Teacher achieves state-of-the-art accuracy, the heavy computational burden of explicit CoT reasoning limits its real-time applicability. Conventional knowledge distillation [17] typically forces a lightweight student either to mimic the teacher’s verbose CoT tokens, failing to resolve the latency bottleneck, or to directly replicate the final answer, severely truncating the required logical depth. To bridge this gap, we introduce a novel Latent Thought Distillation strategy in the third stage. Unlike tool-augmented methods that remain heavily bottlenecked by explicit API calls latency and fragile parsing pipelines, our proposed distillation operates entirely within the model’s internal latent space. Rather than imitating surface-level text, we directly align the Student’s latent representations with the Teacher’s fully converged hidden states at the end of its reasoning trajectory, specifically at the final </think> token. As depicted in 1 (d), this novel objective effectively compresses the complex reasoning capabilities of the System-2 Teacher into a fast, CoT-free System-1 Student. The distilled Student completely bypasses both the external verifier and the intermediate text generation, predicting the safe driving action with ultra-low latency, thereby achieving an optimal balance between deep reasoning rigor and low-latency inference efficiency.

The main contributions of this work can be summarized as follows:

1. We propose CritiqueDriveVLM, a novel three-stage framework specifically designed to bridge the gap between deep System-2 logic and low-latency System-1 execution for autonomous driving VLMs.

2. We introduce Critique-Driven Multi-Turn RL equipped with a multi-dimensional verifier and step-decay penalty, successfully suppressing visual hallucinations without relying on fragile external tools.

3. We develop Latent Thought Distillation to align the student’s hidden representations with the teacher’s fully converged reasoning states, enabling highly accurate CoT-free inference.

4. Extensive experiments on the DriveLMM-o1 [18] benchmark demonstrate the superiority of our approach. Our tool-free Teacher model achieves state-of-the-art reasoning accuracy, while our distilled Student model drastically slashes inference latency to low-latency levels, paving a practical pathway for robust autonomous driving.

2 Related Work↩︎

2.1 VLMs in Autonomous Driving↩︎

The integration of VLMs has shifted autonomous driving from modular [6], [19] pipelines to end-to-end holistic reasoning [10], [11], [20]–[23]. Pioneering works such as DriveVLM [24], [25] demonstrate the potential of VLMs in complex scene understanding and hierarchical planning. To systematically evaluate and advance these capabilities, benchmarks such as DriveLMM-o1 [18], [26] have been introduced, providing specialized datasets that emphasize step-wise visual reasoning and deep logical inference in complex dynamic scenarios. To mitigate reasoning hallucinations, recent state-of-the-art approaches rely on tool-augmented architectures and prolonged CoT. For instance, AgentThink [14] integrates CoT with dynamic tool calls, while OmniDrive-R1 [15] employs an interleaved Multi-modal CoT (iMCoT) mechanism for fine-grained active visual grounding. However, these methods incur exorbitant token consumption and latency bottlenecks [27]. In contrast, our proposed CritiqueDriveVLM eliminates dependencies on external APIs and explicit CoT generation, achieving robust low-latency inference.

2.2 Reinforcement Learning with Verifiable Rewards↩︎

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm to unlock the reasoning potential of Large Language Models [28], [29]. Pioneering works such as DeepSeekMath [16] and DeepSeekMathV2 [30] demonstrate that RL, particularly through algorithms such as GRPO, can significantly boost logical performance in structured domains by optimizing against objective, rule-based rewards. DeepSeek-R1 [31] further showcases how large-scale RLVR elicits emergent self-correction and sophisticated reasoning trajectories. Recently, AlphaDrive [32] introduces a two-stage GRPO-based RL framework tailored for autonomous driving, demonstrating that RL can effectively elicit emergent multimodal planning capabilities beyond standard SFT [33]–[36]. However, applying RLVR to complex, open-ended driving scenarios often suffers from unreliable or sparse reward signals, inevitably leading to severe hallucinations [37] during the CoT process. To address this, our framework introduces a multi-dimensional verifier to provide granular feedback during RL, effectively suppressing CoT hallucinations and establishing a highly reliable Teacher model capable of critique-driven refinement for subsequent distillation.

2.3 Knowledge Distillation↩︎

Knowledge Distillation [17], [38]–[46] is fundamentally designed to transfer capabilities from a cumbersome teacher model to a lightweight student. Early alignment methods, such as Self-Instruct [47], primarily relied on bootstrapping instruction-following data to mimic output distributions. Subsequent advancements, including Distilling Step-by-Step [48] and MiniLLM [49], explicitly distilled CoT rationales to enhance generation quality in smaller models. Recently, researchers have explored internalizing slow System 2 reasoning into fast System 1 intuition [50]–[54], while the distillation phase of DeepSeek-R1 [31] demonstrates that smaller models can successfully inherit complex reasoning capabilities from large RL-trained models. However, existing distillation paradigms typically force the student model to either mimic the teacher’s explicit CoT tokens or simply replicate the final response at the token level. In autonomous driving, generating even a shortened CoT trajectory incurs unacceptable latency, while standard token-level imitation fails to transfer the intricate spatial and logical reasoning deeply embedded in the teacher’s cognitive process. To address this, we propose a new Latent Thought Distillation approach that directly aligns the student’s hidden representations with the teacher’s fully converged reasoning states, achieving an optimal balance between reasoning depth and inference efficiency.

3 Method↩︎

In this section, we describe the proposed CritiqueDriveVLM framework, designed to tackle two critical bottlenecks in autonomous driving VLMs: the inherent unreliability of reasoning in complex scenarios, and the unacceptable inference latency of explicit CoT generation. As illustrated in 2, our framework is implemented through a three-stage training pipeline: Stage 1 (Warm-up SFT and Verifier Construction) establishes the foundational reasoning format and builds a multi-dimensional verifier; Stage 2 (Critique-Driven Multi-Turn RL) leverages verifier feedback and a multi-turn penalty to cultivate a highly reliable Teacher model capable of verifier-guided refinement; Finally, Stage 3 (Latent Thought Distillation) transfers the Teacher’s deep logical representations into a Student model, eliminating the latency of explicit CoT generation.

3.1 Warm-up SFT and Verifier Construction↩︎

Before initiating large-scale RL, it is essential to equip the base policy with a standardized CoT paradigm and construct an independent multi-dimensional verifier.

3.1.1 Warm-up SFT.↩︎

We perform a warm-up SFT on the base VLM using a subset of high-quality CoT data from the DriveLMM-o1 dataset [18]. The primary objective here is not to maximize the model’s reasoning capabilities, but to strictly enforce a structural output format. Given an input image \(V\) and a user instruction \(x\), the model is trained to consistently generate a complete response \(y\) formatted as \(y = \texttt{<think>}\;r\;\texttt{</think>}\;\texttt{<answer>}\;a\;\texttt{</answer>}\), where \(r\) denotes the intermediate reasoning process and \(a\) represents the final answer. This format initialization acts as a critical anchor, preventing mode collapse and format degradation when the model explores the expansive action space during the subsequent RL phase.

3.1.2 Multi-Dimensional Verifier Construction.↩︎

The core component of our multi-turn framework is an independent, frozen multi-dimensional verifier \(\mathcal{V}\), responsible for evaluating the model’s complete generated response \(y\). To train this verifier (initialized from a separate base model), we design a rigorous data generation pipeline. Positive samples are directly derived from ground truth annotations, which are inherently assigned maximum scalar scores across all dimensions with an empty natural language critique. To mine hard negatives, we deploy a baseline GRPO [16] model, trained solely with formatting and accuracy rewards, to perform inference on the training set. This yields a wealth of erroneous outputs exhibiting visual hallucinations or logical contradictions. These flawed responses are subsequently evaluated by a substantially more capable model (Qwen3-VL-235B [3]), which assigns discrete multi-dimensional scores as follows: \[R_{\text{verif}} = \{s_{\text{per}}, s_{\text{log}}, s_{\text{saf}}\}, \quad s \in \{0, 0.5, 1.0\},\] covering perception, logic, and safety. Concurrently, the Qwen3-VL-235B model generates targeted text-based critiques \(c\) addressing the specific errors. Following a stringent data filtering and human verification process, this curated dataset is used to train the verifier. During the RL stage, the function \(\mathcal{V}(V, x, y) \rightarrow (R_{\text{verif}}, c)\) provides the policy model with granular scalar feedback and, if the response fails to meet requirements, it also generates a text-based critique prompt to guide the subsequent reasoning turn.

3.2 Critique-Driven Multi-Turn RL↩︎

To train a highly reliable Teacher model \(\pi_\theta\) capable of profound logical reasoning and verifier-guided refinement, we introduce a multi-turn RL paradigm. Unlike traditional RL settings that rely solely on binary task-completion rewards, our framework leverages the frozen multi-dimensional verifier \(\mathcal{V}\) (constructed in Stage 1) to provide fine-grained textual and scalar feedback. The definitions of the composite reward components are detailed in 1.

3.2.1 Multi-Turn Interaction.↩︎

We formulate the reasoning process as a sequential interactive loop with a maximum allowed turn limit \(K\). At the initial turn \(t=1\), given an input image \(V\) and a prompt instruction \(x_1 = x\), the policy model generates a complete response \(y_1\). The verifier \(\mathcal{V}\) then evaluates \(y_1\) to output a score set \(R_{\text{verif}}\) and a text critique \(c_1\). If \(R_{\text{verif}}\) achieves a perfect score across all dimensions, or the interaction reaches the maximum limit (\(t = K\)), the loop terminates at turn \(T\). Otherwise, the verifier outputs a critique \(c_t\), which is appended to the dialogue history to construct a new user prompt: \(x_{t+1} = \text{Concat}(x_t, y_t, c_t)\). The model then generates a refined response \(y_{t+1}\) based on the critique history [55], [56].

3.2.2 Composite Reward and Efficiency Penalty.↩︎

The reward is calculated based on the terminal response \(y_T\) at the concluding turn \(T\). As outlined in 1, the base reward \(R_{\text{base}}\) aggregates formatting, accuracy, and process-aware evaluations as follows: \[R_{\text{base}}(y_T) = R_{\text{fmt}}(y_T) + R_{\text{acc}}(y_T) + \alpha (s_{\text{per}} + s_{\text{log}} + s_{\text{saf}}),\] where \(\alpha\) is a weighting coefficient for the verifier scores.

While multi-turn interaction significantly enhances reasoning accuracy, autonomous driving systems demand extremely low latency. To prevent the model from becoming overly reliant on external critiques, we introduce a step-decay multi-turn penalty \(\mathcal{P}_{\text{mt}}(T)\). This penalty explicitly forces the model to internalize its reasoning process and promotes first-attempt correctness. The final terminal reward is formulated as: \[R_{\text{final}} = R_{\text{base}}(y_T) - \mathcal{P}_{\text{mt}}(T).\] In practical implementation, to optimally balance exploration efficiency and computational overhead, we set the maximum turn limit to \(K=2\). Thus, the model is penalized with a constant decay value if it requires a second turn to correct its errors.

3.2.3 GRPO Optimization Objective.↩︎

To avoid the massive memory overhead associated with maintaining an equivalent-sized Critic network in the standard Proximal Policy Optimization (PPO) algorithm [57], we employ GRPO in this work. For a given instruction \(x\) drawn from the training distribution \(P(x)\), we sample a group of \(G\) terminal responses, denoted as \(\{y_1, y_2, \dots, y_G\}\), from the old policy \(\pi_{\theta_{\text{old}}}\). The policy is optimized by maximizing the following GRPO objective: \[\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{x \sim P(x), \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^G \mathcal{L}_i - \beta \mathbb{D}_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right],\] where the group-wise clipped loss \(\mathcal{L}_i\) is defined as follows: \[\mathcal{L}_i = \min \left( \rho_i A_i, \operatorname{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i \right),\] where the importance sampling ratio \(\rho_i\) and the normalized advantage \(A_i\) are given by: \[\begin{align} \rho_i &= \frac{\pi_\theta(y_i|x)}{\pi_{\theta_{\text{old}}}(y_i|x)}, \\ A_i &= \frac{R_{\text{final}}^{(i)} - \operatorname{mean}(R_{\text{final}})}{\operatorname{std}(R_{\text{final}})}. \end{align}\] The hyperparameter \(\epsilon\) clips this ratio to ensure training stability. Additionally, \(\beta\) controls the strength of the Kullback-Leibler (KL) divergence penalty \(\mathbb{D}_{\text{KL}}\), which restricts the policy \(\pi_\theta\) from deviating excessively from the reference model \(\pi_{\text{ref}}\) established during the warm-up SFT stage.

3.3 Latent Thought Distillation↩︎

To resolve the prohibitive latency of explicit CoT generation, we propose Latent Thought Distillation, internalizing the Teacher’s slow System-2 reasoning into the Student’s fast System-1 intuition without outputting CoT text, inspired by [50], [51].

We initialize the Student model directly from the original base VLM. Furthermore, we construct the distillation dataset by strictly formatting the DriveLMM-o1 training data into pure prompt-answer pairs, entirely removing any explicit CoT text. This setup forces the Student to rely solely on its internal latent representations to maintain logical depth.

Our strategy aligns the hidden representation of the Student with the final reasoning state of the Teacher. Let \(\pi_T\) and \(\pi_S\) be the frozen Teacher and trainable Student, respectively. For a given input, \(\pi_T\) generates its multi-turn reasoning trajectory. We extract the deep hidden state at the last </think> token of the trajectory, denoted as \(\mathbf{h}_{T}^{\text{think}} \in \mathbb{R}^{d}\). Simultaneously, \(\pi_S\) skips the intermediate reasoning steps to directly predict the final output. We extract its corresponding hidden state at the <answer> token, denoted as \(\mathbf{h}_{S}^{\text{answer}} \in \mathbb{R}^{d}\).

To ensure the Student’s internal logic mimics the Teacher’s thought process, we employ Cosine Similarity as the alignment objective: \[\mathcal{L}_{\text{align}} = 1 - \frac{\mathbf{h}_{S}^{\text{answer}} \cdot \mathbf{h}_{T}^{\text{think}}}{\|\mathbf{h}_{S}^{\text{answer}}\|_2 \|\mathbf{h}_{T}^{\text{think}}\|_2}.\] This objective strictly focuses on aligning the semantic direction of the reasoning logic in high-dimensional space. The Student model is subsequently optimized through a joint loss function: \[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{align}},\] where \(\mathcal{L}_{\text{CE}}\) is the standard auto-regressive cross-entropy loss for supervised answer prediction, and \(\lambda\) is the balancing coefficient. Through this unified objective, the Student inherits the Teacher’s logical rigor within its latent space, achieving efficient and accurate prediction.

4 Experiments↩︎

In this section, we comprehensively evaluate the effectiveness of the proposed CritiqueDriveVLM framework in complex autonomous driving scenarios. The remainder of this section is organized as follows: We first detail the experimental setup in 4.1. Then, we present the main quantitative results on the DriveLMM-o1 benchmark [18], highlighting the state-of-the-art reasoning capabilities of our Stage-2 Teacher model in 4.2. Next, we analyze the inference efficiency and the impact of Latent Thought Distillation under CoT-free constraints in 4.3. We further conduct ablation studies to validate the critique-driven multi-turn reinforcement learning mechanisms in 4.4. Finally, we provide a qualitative analysis of our model’s decision-making process in 4.5.

4.1 Experimental Setup↩︎

4.1.1 Datasets.↩︎

To ensure a fair comparison and strictly validate the effectiveness of our framework, we exclusively utilize the DriveLMM-o1 [18] benchmark, which is built upon the foundational nuScenes [58] dataset. Specifically, we employ its training split comprising over 18,000 VQA pairs, where each sample provides step-by-step reasoning for complex driving scenarios. This unified dataset is consistently applied across all three stages of our framework: Warm-up SFT, Critique-Driven RL, and Latent Thought Distillation.

4.1.2 Evaluation Metrics.↩︎

We adopt the official evaluation protocol of the DriveLMM-o1 benchmark, reporting the Overall Reasoning score and Multiple Choice Quality (MCQ) as primary metrics. Additionally, we evaluate fine-grained task-specific driving and scene detail metrics (, Risk Assessment, Traffic Rule Adherence, Scene Awareness, Relevance, and Missing Details). Further metric calculation details are provided in Appendix A.

4.1.3 Implementation Details.↩︎

We employ Qwen3-VL-8B [3] as our base model for both the Teacher and the Student, keeping the vision encoder frozen across all training phases. During Stage 1, the Warm-up SFT is conducted using Low-Rank Adaptation (LoRA) [59] with a learning rate of \(1 \times 10^{-4}\). For Stage 2, the GRPO [16] fine-tuning utilizes a learning rate of \(2 \times 10^{-6}\) with 4 rollouts per question (\(G=4\)), and the maximum multi-turn interaction limit is set to \(K=2\). In Stage 3, the Latent Thought Distillation is also optimized using LoRA with a learning rate of \(2 \times 10^{-5}\), and the balancing coefficient for the alignment loss is set to \(\lambda = 0.5\). Across all three training phases, the models are optimized for 2 epochs with a global batch size of 128. All experiments are conducted on a single server equipped with 8 NVIDIA A100 GPUs, and inference time is tested on a single A100 GPU. Additional implementation details and hyperparameter settings are elaborated in Appendix B.

2.8pt

4.2 Main Results↩︎

We evaluate the comprehensive driving and reasoning capabilities of our framework against a wide range of state-of-the-art VLMs on the DriveLMM-o1 benchmark. As shown in ¿tbl:tab:main?, the comparative baselines include prominent general-purpose models (, GPT-4o [60], InternVL2.5-8B [4], and the Qwen-VL series [3], [5]) as well as domain-specific autonomous driving models (, DriveLMM-o1 [18], AgentThink [14], and OmniDrive-R1 [15]).

Our Stage-2 Teacher establishes a new state-of-the-art across multiple critical metrics. Notably, it achieves the highest Overall Reasoning score of 80.48% and an MCQ of 76.54%, significantly outperforming strong driving-specific counterparts like OmniDrive-R1 (73.62%) and AgentThink (71.35%). Unlike baselines relying on complex external APIs or interleaved multi-modal CoT, our Teacher achieves superior accuracy entirely through internalized logic cultivated during the critique-driven RL phase, avoiding explicit tool-use overhead.

Delving into the fine-grained metrics, our model demonstrates a significant advantage in Traffic Rule Adherence (86.21%) and Relevance (85.46%). While OmniDrive-R1 marginally leads in Risk Assessment (82.31%) and Scene Awareness and Object Understanding (83.75%), and AgentThink slightly edges out in the Missing Details metric (79.56%), our Teacher model maintains highly competitive and robust scores in these areas (81.63%, 82.27%, and 79.17%, respectively). This comprehensive and well-rounded performance profile proves that providing granular, multi-dimensional verifier feedback during RL effectively suppresses visual hallucinations and conservative biases, resulting in highly reliable end-to-end driving decisions.

2.5pt

4.3 Efficiency and Latent Thought Distillation↩︎

Real-time autonomous driving systems impose stringent constraints on inference latency. While explicit CoT generation and tool-use mechanisms significantly enhance reasoning accuracy, they inherently incur unacceptable computational overhead. As shown in the System-2 section of 2, traditional reasoning models suffer from extreme verbosity. AgentThink incurs a staggering latency of 5416 ms due to explicit tool invocations. Even our highly capable Stage-2 Teacher requires 3482 ms to articulate its internal reasoning process. To eliminate this latency bottleneck, we must evaluate models under a strict CoT-free constraint.

However, when standard baselines are forced to predict final answers directly without CoT, their performance drops precipitously. As shown in the System-1 section of 2, the Qwen3-VL-8B (SFT) model without distillation severely truncates the logical depth, achieving an MCQ of only 61.73%. To bridge this capability gap, our Latent Thought Distillation integrates an alignment objective (\(\mathcal{L}_\text{align}\)). By explicitly aligning the Student’s latent representations with the Teacher’s fully converged reasoning state, our Stage-3 Student achieves a superior MCQ of 68.59% using an average of only 28.83 tokens. This drastically slashes the absolute inference time to 416 ms, an 88% reduction compared to the Stage-2 Teacher (3482 ms), demonstrating that our proposed distillation method successfully internalizes deep System-2 reasoning capabilities into the fast System-1 latent space, achieving an optimal balance between reasoning accuracy and low-latency execution.

4.4 Ablation on Critique-Driven Multi-Turn RL↩︎

To validate the contribution of each core mechanism in Critique-Driven Multi-Turn RL (Stage 2), we ablate our reward design in 3. Starting from the Warm-up SFT baseline (62.86% MCQ), applying standard single-turn GRPO with only Accuracy Rewards yields a marginal improvement (64.49% MCQ). The addition of Multi-dimensional Verifier Rewards further boosts the MCQ to 67.46% while steadily improving fine-grained metrics such as Risk Assessment and Traffic Rule Adherence, indicating that providing granular scalar feedback effectively suppresses visual hallucinations. Crucially, introducing the Multi-Turn Interaction combined with a step-decay penalty significantly elevates the overall MCQ to 76.54%. This substantial improvement confirms our hypothesis: transitioning from a single-turn setup to a multi-turn loop allows the model to iteratively refine its reasoning based on verifier critiques, while the multi-turn penalty prevents over-reliance on these external cues, forcing the policy to internalize the logical deduction process for correctness.

3.5pt

4.5 Qualitative Analysis↩︎

To intuitively demonstrate the efficacy of our CritiqueDriveVLM framework, 3 shows a comparative case study in a complex driving scenario involving a pedestrian crossing. As illustrated, the standard SFT Baseline suffers from severe visual hallucination; it explicitly reasons that there are “no visible pedestrians” and dangerously concludes that no evasive action is required (Option F). In contrast, our Stage-2 Teacher overcomes this failure through its critique-driven multi-turn reasoning mechanism. While its initial draft mirrors the Baseline’s dangerous oversight, the external verifier injects targeted critique feedback, alerting the model to the overlooked pedestrian. This prompts the Teacher to successfully re-examine the visual evidence, correct its flawed trajectory, and output the safe action to “Come to a complete stop” (Option A). Crucially, our Stage-3 Student directly and instantly predicts the correct, safe action (Option A) without any explicit reasoning overhead. This confirms that our Latent Thought Distillation effectively compresses the Teacher’s rigorous System-2 risk assessment capabilities directly into the Student’s fast System-1 latent space.

5 Conclusion↩︎

In this paper, we presented CritiqueDriveVLM, a unified three-stage framework designed to resolve the reasoning fragility and inference latency bottlenecks in autonomous driving VLMs. By introducing Critique-Driven Multi-Turn RL, our Teacher model internalized logical deduction to mitigate visual hallucinations, achieving state-of-the-art accuracy without relying on fragile external tools. Furthermore, our Latent Thought Distillation successfully compressed these deep System-2 reasoning capabilities into a CoT-free System-1 Student, preserving decision accuracy while reducing inference latency to low-latency levels (416 ms). In the future, we plan to extend this framework to multi-camera video streams for temporal logic distillation.

Acknowledgements↩︎

This work is partly supported by the Funds for the Beijing Natural Science Foundation under Grant L243027, the NSFC Project under Grant 62572072 and Hainan Provincial Natural Science Foundation.

References↩︎