1 Introduction↩︎

Vehicle-to-everything-aided autonomous driving (V2X-AD) can significantly address the inherent limitations of single-vehicle autonomous driving, such as restricted field of view, occlusion blind spots, and unpredictability of other road users’ behaviors. However, existing V2X collaborative driving methods uniformly adopt a two-stage paradigm: (1) multi-agent feature fusion is performed at the perception stage with fixed fusion weights; (2) the fused features are then passed to single-vehicle planning. This paradigm suffers from fusion timing mismatch—fusion strategies designed for perception tasks (e.g., fixed attention weights optimized for detection) may not be suitable for planning, where scenario-dependent adaptive weighting is required. Representative collaborative perception and prediction methods [1]–[3] all perform fusion before the planning stage, leaving no opportunity for planning-aware adaptive fusion.

CoDriving [4], the state-of-the-art end-to-end V2X system, exemplifies this limitation: multi-agent fusion occurs at the perception stage with fixed weights that cannot adapt to planning requirements (e.g., roadside features are critical in occluded intersections but less valuable on open highways). Additionally, roadside features contain 75%–90% background information redundant for planning.

Our Core Insight: The key issue is when and how to fuse multi-agent features. We propose to defer fusion to the planning stage, where an autoregressive decoder can dynamically learn adaptive fusion weights based on the current driving context, rather than using fixed weights determined at the perception stage.

Our Approach. As illustrated in Figure 2, our system features:

• Heterogeneous Feature Processing: Targeting feature differences between vehicle and infrastructure agents, we design a dual-stream processing pipeline with planning-oriented channel-wise purification on roadside features and MoE-enhanced tokenizers.

• Autoregressive Adaptive Fusion: Employ LLaMA-style autoregressive decoder to achieve adaptive fusion of multi-agent features, dynamically learning importance weights of ego and road tokens through self-attention mechanism.

• End-to-End Planning Optimization: Directly optimize planning tasks through end-to-end training, achieving complete pipeline optimization from feature extraction to trajectory generation.

Contributions:

• We propose a planning-stage fusion framework that defers multi-agent feature fusion to the planning stage, enabling adaptive fusion weights learned through the autoregressive decoder.

• We design dual-stream feature processing with channel-wise purification and MoE-enhanced tokenization for heterogeneous vehicle-infrastructure features.

• We introduce autoregressive decoder that dynamically learns scenario-dependent fusion weights, replacing fixed fusion at the perception stage.

• Experiments on V2Xverse demonstrate significant improvements over both single-vehicle and perception-stage fusion baselines.

2 Related Work↩︎

Collaborative Perception and Prediction. V2X collaborative perception and prediction methods [1]–[3], [5] mostly perform multi-agent fusion before planning, typically with fixed fusion strategies optimized for perception or forecasting. Roadside-specific methods [6], [7] further study infrastructure-view representations. These designs provide strong upstream perception, but leave limited room for planning-aware adaptive fusion.

End-to-End Autonomous Driving. End-to-end driving methods [8]–[10] demonstrate planning-oriented optimization, while UniV2X [11] extends this direction to V2X cooperation. Related roadside representation and road-scene understanding works [12]–[16] improve infrastructure-side modeling. Our work focuses on planning-stage adaptive fusion with robustness analysis under latency and noise.

Collaborative Driving. Collaborative driving systems such as Coopernaut [17] and CoDriving [4] perform fusion at the perception stage with fixed weights, creating an information bottleneck before planning. Our work defers fusion to the planning stage with adaptive weights learned by autoregressive decoder.

MoE and Autoregressive Models. MoE [18] enables conditional computation with expert specialization; DriveMoE [19] introduces it to autonomous driving. LLaMA [20] demonstrates autoregressive generation capabilities. We leverage both for adaptive multi-agent fusion.

3 Method↩︎

3.1 Problem Formulation↩︎

Consider \(N\) collaborative agents (vehicles + roadside units), given observations \(\{X_i\}^N_{i=1}\), the goal is to maximize each vehicle’s driving performance while satisfying communication bandwidth constraint \(B\): \[\max_{\theta,P} \sum_{i=1}^{N} d(\Phi_\theta(X_i, D_i, \{P_{j\rightarrow i}\}^N_{j=1})) \quad \text{s.t.} \quad \sum_{j\neq i} \|P_{j\rightarrow i}\| \leq B\] where \(d(\cdot)\) is the driving performance metric, \(\Phi_\theta\) is the end-to-end driving network, and \(P_{j\rightarrow i}\) is the communication message from agent \(j\) to \(i\).

3.2 Overall Architecture↩︎

As shown in Figure 3, our collaborative driving system adopts a dual-stream processing architecture. The pipeline consists of three main stages:

Multi-Agent Feature Extraction: Vehicle and roadside agents separately collect LiDAR sensor data, extract spatial features through BEV encoder [21], [22], and generate occupancy maps.

Dual-Stream Tokenization: The ego stream directly feeds BEV features through MotionNetwork for temporal aggregation, then applies attention-based tokenizer to extract compact ego tokens. The road stream first performs channel-wise purification to filter out planning-irrelevant background channels, then processes the filtered features through a separate MotionNetwork, and finally employs MoE-enhanced tokenizer to generate road tokens with expert specialization.

Adaptive Multi-Agent Fusion: Construct token sequence and dynamically learn fusion weights through autoregressive decoder to generate collaborative waypoints.

3.3 Multi-Agent Feature Extraction and Tokenization↩︎

3.3.1 Dual-Stream Temporal Feature Aggregation↩︎

Inspired by CoDriving, we configure independent MotionNetworks for vehicle and roadside agents. Each MotionNetwork receives historical \(T\) frames of BEV features \(F^{(t)} \in \mathbb{R}^{T\times C\times H\times W}\) and occupancy maps \(O^{(t)} \in \mathbb{R}^{T\times 6\times H\times W}\), performs spatiotemporal joint encoding through 3D convolution, and outputs compressed motion-aware features \(M \in \mathbb{R}^{256\times H'\times W'}\).

3.3.2 Planning-Oriented Road Feature Purification↩︎

Roadside units have fixed viewpoints and wide coverage, with features often containing 75%–90% background information irrelevant to current planning tasks. We introduce channel-wise top-\(k\) filtering before the roadside MotionNetwork:

1. Channel Activation: \(a_c = \text{mean}_{h,w}(F_c^{\text{road}}(h,w))\)

2. Top-K Selection: Retain top \(\alpha\%\) channels (\(\alpha=0.1\)), zero out others

3. Temporal Fusion: Filtered features enter MotionNetwork

Ego vehicle features already focus on forward key regions and do not require purification.

3.3.3 MoE-Enhanced Heterogeneous Tokenization↩︎

We design FlexibleTokenizer to compress spatial features into a compact token set. A naive tokenizer with a single shared MLP struggles to simultaneously model ego-centric and infrastructure-centric views as well as diverse traffic patterns (highway, intersections, dense crowds). To address this, we instantiate independent tokenizers with separate parameters for ego and road, and further introduce MoE in the MLP layers so that different experts specialize on different driving maneuvers and scene patterns.

Motivation for MoE Architecture. Theoretically, MoE with \(N\) experts offers two advantages over a single MLP: (1) Conditional computation—activating only Top-\(K\) experts reduces inference cost while maintaining model capacity; (2) Specialization—different experts can specialize on distinct input patterns (e.g., sharp turns vs. straight driving), avoiding the mode-averaging effect where a single model learns a compromise representation suboptimal for all scenarios. However, these advantages only materialize if the router learns meaningful specialization; otherwise, the model degenerates to a weighted average of similar experts.

Tokenization Pipeline. Features are projected and downsampled to \(F' \in \mathbb{R}^{d\times H''\times W''}\), flattened with positional encoding: \(X = \text{Flatten}(F') + \text{PE} \in \mathbb{R}^{H''W''\times d}\). Multi-head attention [23] pooling with \(k\) learnable queries \(Q \in \mathbb{R}^{k\times d}\) aggregates spatial features: \[\text{Tokens} = \text{MultiHeadAttention}(Q, X, X) \in \mathbb{R}^{k\times d}\]

MoE Integration with Trajectory-Guided Routing. We employ MoE layers with Top-3 routing among 6 experts. To prevent expert collapse, we introduce trajectory-guided supervision: K-means clustering on trajectory curvature identifies 6 maneuver patterns, providing soft labels for the router. The gating network computes \(p = \text{Softmax}(\mathcal{R}(h))\), selecting Top-\(K\) experts for token projection.

After processing, the outputs are \(\text{ego\_tokens} \in \mathbb{R}^{B\times k_e\times d}\) and \(\text{road\_tokens} \in \mathbb{R}^{B\times k_r\times d}\).

3.4 Autoregressive Decoder for Multi-Agent Adaptive Fusion↩︎

3.4.1 From Fixed Fusion to Adaptive Collaboration↩︎

CoDriving’s fixed attention weights cannot adapt to scenarios—roadside features are critical in occluded intersections but less important on open roads. We propose autoregressive decoder for adaptive fusion, dynamically learning importance weights through self-attention.

3.4.2 LLaMA-Style Parallel Autoregressive Decoder↩︎

We adopt LLaMA-style [20] parallel autoregressive decoder for efficient trajectory generation.

Token Sequence Construction. To enable joint reasoning over navigation intent and multi-agent perception, we construct a unified token sequence by concatenating the target embedding, ego tokens, road tokens, and learnable planning queries: \[S = [\text{target\_embed}; \text{ego\_tokens}; \text{road\_tokens}; \text{plan\_queries}]\] where \(S \in \mathbb{R}^{(1+k_e+k_r+T_f)\times d}\), with \(k_e\) and \(k_r\) denoting the number of ego and road tokens respectively, and \(T_f\) representing the number of future waypoints to predict.

Hybrid Causal Mask. We design a hybrid attention mask that allows bidirectional attention within the context region (target + ego + road tokens) for comprehensive scene understanding, while enforcing causal attention in the planning region to ensure temporally coherent trajectory generation where each waypoint can only attend to previous waypoints and the full context. Concretely, the mask is applied to decoder self-attention after concatenating context tokens and planning queries into a single sequence. Context tokens attend to each other bidirectionally, while each planning query attends to the full context and only preceding planning queries. This preserves causal waypoint generation while retaining parallel decoding efficiency.

Layer-wise Processing. The concatenated sequence is processed through multiple transformer decoder layers with the hybrid causal mask: \[S^{(l+1)} = \text{TransformerLayer}(S^{(l)}, S^{(l)}, \text{mask}=M_{\text{causal}})\] This enables the planning queries to dynamically aggregate information from both ego and road tokens through attention weights learned end-to-end.

Waypoint Generation. After transformer processing, the planning tokens are projected to 2D coordinates through a linear layer: \[W = \text{Linear}(S_{\text{plan}}) \in \mathbb{R}^{T_f\times 2}\] Each row of \(W\) represents the \((x, y)\) offset of a future waypoint relative to the ego vehicle’s current position.

3.4.3 Collaborative Information Flow↩︎

In the decoder, multi-agent information flows through self-attention where ego and road tokens interact bidirectionally for mutual perception enhancement. Planning queries dynamically aggregate features from both streams through learned attention weights, adaptively emphasizing ego or road information based on driving context. The causal mask ensures sequential waypoint generation with temporal coherence.

3.5 Training Strategy↩︎

Two-Stage Training. We first freeze the pre-trained BEV encoder and train planning components (MotionNetwork, tokenizers, decoder), then optionally fine-tune end-to-end with smaller learning rate.

Loss Function. The training objective combines trajectory regression and MoE auxiliary losses: \[\mathcal{L} = \mathcal{L}_{\text{waypoint}}(W_{\text{pred}}, W_{\text{gt}}) + \lambda_{\text{router}}(t) \mathcal{L}_{\text{router}} + \lambda_{\text{moe}} \mathcal{L}_{\text{balance}}\] where \(\mathcal{L}_{\text{waypoint}}\) is L1 loss for trajectory regression.

Preventing Expert and Router Collapse. Without explicit guidance, MoE training faces two failure modes: (1) Expert collapse—all experts learn similar representations, degenerating the model into a weighted average of redundant functions; (2) Router collapse—the router consistently selects a fixed subset of experts, leaving others untrained. To prevent these failures, \(\mathcal{L}_{\text{router}}\) provides trajectory-guided supervision from K-means clustering on curvature (\(K\)=6 maneuvers), and \(\mathcal{L}_{\text{balance}}\) encourages balanced expert utilization. We apply curriculum annealing: \[\lambda_{\text{router}}(t) = \lambda_0 \cdot \max(0, 1 - t/T_{\text{anneal}})\] where \(\lambda_0\)=0.03. This strategy applies strong supervision early to escape poor local minima, then progressively reduces guidance to allow autonomous expert selection.

4 Experiments↩︎

4.1 Experimental Setup↩︎

We conduct experiments on V2Xverse [4], a V2X simulation platform built on CARLA with closed-loop evaluation support. We evaluate on 67 routes across 8 towns with safety-critical scenarios including occluded intersections and unexpected pedestrians. We report closed-loop metrics including Driving Score (DS), Route Completion (RC), and Infraction Score (IS), where DS is the benchmark’s overall driving score, RC measures route completion, and IS reflects infraction-related safety. We also report open-loop trajectory errors including Average Displacement Error (ADE) and Final Displacement Error (FDE), measured in meters. Baselines include single-agent methods (TCP [10], TransFuser [9]) and collaborative methods (CoDriving [4], Coopernaut [17], V2X-ViT [2]).

4.2 Main Results↩︎

Table 2 summarizes both open-loop trajectory metrics (ADE/FDE) and closed-loop driving metrics (DS/RC/IS) on V2Xverse.

Our method achieves state-of-the-art performance across both evaluation paradigms. In open-loop evaluation, we surpass CoDriving by 3.39% in ADE and 1.42% in FDE, validating the effectiveness of our planning-oriented feature processing. In closed-loop driving, we achieve the highest driving score (79.72, +3.33%) and infraction score (0.88, +7.32%), demonstrating that our autoregressive adaptive fusion produces safer trajectories with fewer traffic violations. Compared with CoDriving, the slight RC decrease (-1.29%) together with improved DS and IS suggests a safety-efficiency trade-off: our planner behaves more cautiously in some edge cases, sacrificing a small amount of completion for better infraction-related outcomes. The improvement in IS particularly validates our dual-stream design: channel-wise purification filters planning-irrelevant background from roadside features, while MoE tokenizers enable specialized processing for different driving maneuvers.

4.3 Robustness Analysis↩︎

We evaluate robustness under real-world communication challenges.

Pose Estimation Noise. Table 3 evaluates performance under Gaussian pose noise. Our method demonstrates superior robustness: under severe noise (0.6), we degrade by only 4.01% ADE compared to CoDriving’s 5.50%. This stems from our autoregressive fusion, which dynamically down-weights misaligned roadside features.

Communication Latency. Network congestion can cause roadside information to arrive 200–600ms late. Table 4 shows our autoregressive fusion handles stale information gracefully: under 600ms latency, we degrade by only 2.34% ADE vs. CoDriving’s 2.75%. The adaptive attention learns to rely more on ego features when roadside information becomes temporally misaligned.

These results validate that our autoregressive fusion dynamically re-weights ego and roadside tokens to compensate for degraded collaborative information.

4.4 Ablation Studies↩︎

To validate the effectiveness of each component, we conduct ablation studies. Table 5 shows the results.

Our method builds on CoDriving with three key components:

• MoE-Enhanced Tokenizers: Specialized experts yield 1.84% ADE and 0.91% FDE gains through conditional expert selection for different driving maneuvers.

• Channel-wise Purification: Roadside feature purification contributes 2.34% ADE and 1.36% FDE improvements by suppressing planning-irrelevant background channels.

• Autoregressive Decoder: Replacing fixed fusion brings the largest gain (4.85% ADE, 3.88% FDE) via dynamic re-weighting between ego and roadside tokens.

Together, these components achieve 4.21% ADE and 3.28% FDE improvement over CoDriving.

4.5 Visualization and Analysis↩︎

4.5.1 Trajectory Visualization↩︎

Figure 4 illustrates a challenging yield-and-turn scenario. When a cyclist emerges from an occluded region, our method: (1) promptly stops to yield, leveraging roadside features that detect the cyclist earlier; (2) smoothly resumes after the cyclist passes. This validates that our fusion dynamically adjusts ego vs. roadside importance based on the evolving context.

4.5.2 MoE Expert Activation Patterns↩︎

Figure 5 reveals meaningful expert specialization: straight driving activates Experts 1, 2, 5, while yielding with left turns triggers Experts 0, 4, 1. This confirms our trajectory-guided routing prevents expert collapse—each expert specializes on distinct maneuvers rather than degenerating into redundant copies.

4.5.3 Feature Purification Effect↩︎

Figure 6 shows that after Top-10% channel selection, feature activation concentrates on planning-critical regions: driving corridor, dynamic agents, and conflict zones. Background regions (distant buildings, parked vehicles) are suppressed, explaining the 2.34% ADE improvement in our ablation study.

5 Conclusion↩︎

We propose a planning-oriented V2X collaborative driving system that defers multi-agent fusion to the planning stage, enabling adaptive weights learned through an autoregressive decoder. Our dual-stream pipeline with channel-wise purification and MoE-enhanced tokenizers effectively handles heterogeneous vehicle-infrastructure features. Experiments demonstrate state-of-the-art performance with 3.33% driving score and 7.32% infraction score improvements over CoDriving, along with stronger robustness under noise and latency.

Limitations and Future Work. Our system assumes fixed communication topology; future work can explore dynamic agent selection based on relevance. Explicitly modeling communication delays and incorporating uncertainty estimation would further enhance real-world applicability.

References↩︎

---

1. This work was supported by Key-Area Research and Development Program of Guangdong Province (2023B0909040001) and Shenzhen Science and Technology Program (KJZD20231023094701003).↩︎