1 Introduction↩︎

Learning generalizable action policies is a central challenge in autonomous driving, where a planner trained on one dataset must safely transfer to unseen cities, traffic patterns, sensor distributions, and simulation environments. Standard end-to-end planners directly map historical observations to future ego trajectories, which is efficient but can overfit dataset-specific driving priors such as common trajectory patterns, road layouts, or benchmark-specific behaviors [@hu2023planning; @chitta2022transfuser; @liao2025diffusiondrive]. World Action Models (WAMs) offer a promising alternative by coupling future prediction with action generation [@ye2026world]. Compared with sparse trajectory supervision, future world modeling provides dense signals about scene geometry, object motion, traffic-agent interactions, and temporal causality, helping the model learn both how the world evolves and how the ego vehicle should act [@lecun2022path; @ha2018world; @wang2024_drive-WM; @zheng2024genad]. Recent driving world models and embodied video-action models further show that future visual prediction can improve planning, simulation, transferability, and zero-shot policy generalization [@gao2024_vista; @ye2026world; @jang2025dreamgen; @lingbot-va2026; @li2025unified; @zhu2025unified].

However, it remains unclear which architecture best transfers video-learned world-dynamics priors to trajectory generation. Fast-WAM [@yuan2026fast] shows that future video prediction can be more useful as training-time world-dynamics supervision than as mandatory test-time imagination, since bypassing video rollout significantly accelerates action inference. This raises a complementary training-side question: if future videos and ego trajectories describe the same future driving state, should they be denoised by separate modules, or by a shared generative backbone? As shown in Fig. 1, existing WAMs often use cascaded or dual-DiT designs that separate video and trajectory generation [@li2025imagidrive; @yuan2026fast; @liu2026driveva; @ye2026gigaworldpolicy; @wang2025prophetdwm]. Such separation lets video-learned dynamics regularize planning only indirectly, while the trajectory branch can still overfit dataset-specific action priors. We instead treat future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising.

Our key finding is that architecture matters: video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone [@pai2025mimic; @liang2025video; @kim2026cosmos; @chen2025large]. Based on this insight, we propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer [@peebles2023DIT]. Unlike cascaded or two-DiT WAMs [@lingbot-va2026; @won2025dual], UNIVERSE places historical observations, future video latents, and ego-trajectory tokens in one shared DiT. Because video and trajectory losses update the same attention layers and feed-forward blocks, dense video dynamics directly shape the parameters used for trajectory denoising, encouraging a shared world-action representation. This design also improves efficiency over the Two-DiT variant, reducing trajectory inference latency, inference memory, and training memory, as later shown in Table [tbl:tab:efficiency]. Naively placing future video and trajectory tokens into one DiT, however, may lead to inefficient and entangled generation. Coupled future-token denoising forces trajectory inference to retain multi-frame video generation, causing unnecessary memory and latency overhead. Meanwhile, mutual attention entangles video and trajectory denoising, allowing errors from one modality to interfere with the other. We therefore introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking future video-action attention. Thus, UNIVERSE achieves coupling-by-parameters and decoupling-by-visibility: both objectives train one shared DiT, while future targets remain causally separated and independently instantiable at inference.

This mask also enables efficient deployment. Because trajectory tokens are trained without depending on future video tokens, the video branch can be safely removed at test time with minimal distribution shift. UNIVERSE therefore supports trajectory-only inference by directly denoising future ego trajectories from historical observations, reducing inference time by \(4.3\times\) while preserving the generalization benefit of video co-training. The same model can also support video-only rollout for future-scene simulation and joint video-action rollout for interpretable planning and safety analysis by changing the inference mask and instantiated token groups. To test this hypothesis, we build controlled UNIVERSE variants that ablate video co-training, the Modality-Decoupling Visibility Mask, and shared DiT parameterization. Together, they clarify how dense video supervision, causal modality decoupling, and shared parameterization contribute to robust action generalization.

Overall, our contributions are summarized as follows:

• We propose UNIVERSE, a unified video-action model that co-trains future video latents and ego-trajectory tokens within one shared Diffusion Transformer, allowing dense video supervision to directly regularize trajectory denoising.

• We introduce a Modality-Decoupling Visibility Mask that shares historical context across modalities while blocking mutual attention between future video and trajectory tokens, preventing causal leakage in unified video-action training.

• We enable efficient trajectory-only inference by removing future-video denoising at test time, achieving a \(4.3\times\) speedup while maintaining the generalization benefit of video co-training. Experiments on NAVSIM, nuScenes, and Bench2Drive demonstrate state-of-the-art closed-loop planning and strong zero-shot action generalization.

2 Related Work↩︎

2.1 World Models for Autonomous Driving Video Generation↩︎

World modeling has become a major direction for autonomous-driving video generation, aiming to synthesize temporally coherent and controllable future observations for simulation, data augmentation, and downstream decision-making. Early works such as DriveGAN [@kim2021drivegan] and MagicDrive [@gao2023magicdrive] explore controllable driving-scene generation. Recent methods further improve multi-view consistency, 3D geometric control, long-horizon generation, and scenario diversity. Representative examples include DriveDreamer [@wang2024drivedreamer], DriveDreamer-2 [@zhao2025drivedreamer], Panacea [@wen2024panacea], VISTA [@gao2024vista], GAIA-1 [@hu2023gaia], GAIA-2 [@russell2025gaia], MagicDrive-V2 [@gao2025magicdrive], MiLA [@wang2025mila], CoGen [@ji2025cogen], Genesis [@guo2025genesis], and OmniNWM [@li2025omninwm]. Another line of work studies world models in BEV, occupancy, LiDAR, or 4D representations, such as OccWorld [@zheng2024occworld], DriveWorld [@min2024driveworld], DrivingWorld [@hu2024drivingworld], WoVoGen [@lu2024wovogen], UniScene [@li2025uniscene], and related occupancy-centric driving simulators. These methods have significantly advanced the fidelity, controllability, and temporal consistency of driving-scene generation. However, they are primarily designed as scene simulators or data generators, with the main objective of improving visual realism, geometric consistency, or controllable rollout. Less attention has been paid to how video-learned world priors can be effectively transferred to trajectory generation for action generalization. In contrast, our work takes a planning-oriented perspective: UNIVERSE uses future video prediction as dense training-time supervision for trajectory generation, rather than treating video generation only as a standalone simulation task.

2.2 World–Action Models for Autonomous Driving Control↩︎

Beyond pure scene generation, world models have increasingly been incorporated into autonomous-driving planning and control. Representative methods include Driving into the Future [@wang2024driving], DrivingGPT [@chen2025drivinggpt], latent-world-model-assisted driving [@li2024enhancing], Doe-1 [@zheng2024doe], ReSim [@yang2025resim], WoTE [@li2025end_wote], DriveLaW [@xia2025drivelaw], Epona [@zhang2025epona], Policy World Model [@zhao2025forecasting], and DriveVLA-W0 [@li2025drivevlaw0]. These methods show that modeling future scene evolution can provide richer supervisory signals for motion planning, closed-loop control, and cross-domain transfer. Meanwhile, recent embodied WAMs and video-action models, such as DreamZero [@ye2026world], GigaWorld-Policy [@ye2026gigaworldpolicy], LingBot-VA [@lingbot-va2026], Unified Video Action Model [@li2025unified], Unified World Models [@zhu2025unified], and DreamGen [@jang2025dreamgen], further suggest that joint video-action learning can improve generalization beyond the training domain. Existing WAMs differ mainly in how future video prediction is coupled with control. Imagine-then-plan methods retain video rollout in the online loop, offering interpretable future imagination but incurring diffusion or autoregressive latency. DriveVA [@liu2026driveva] jointly predicts future videos and ego trajectories with a DiT decoder, while Fast-WAM [@yuan2026fast] shows that video prediction can serve primarily as training-time supervision, enabling trajectory inference without explicit rollout. In contrast, UNIVERSE keeps unified video-action training, removes test-time future-video denoising, and studies whether shared video-action parameters better transfer world-dynamics priors to planning than separated video-action modules.

3 Method↩︎

3.1 Unified Diffusion Transformer↩︎

Problem Definition. As shown in Fig. 2, given \(l\)-frame history observations \(\mathcal{O}_s=\{\mathbf{F}_{s-l+1},\ldots,\mathbf{F}_{s}\}\), current ego velocity \((v_x,v_y)\), and optional text instruction \(\mathcal{T}\), UNIVERSE predicts future actions and future visual rollouts either jointly or as a single modality. The outputs are defined as \[\mathcal{A}_{s+1:s+K} = \{\boldsymbol{a}_{s+i}\in\mathbb{R}^3\}_{i=1}^{K}, \qquad \mathcal{F}_{s+1:s+N} = \{\mathbf{F}_{s+j}\}_{j=1}^{N}, \label{eq:problem95output}\tag{1}\] where each action \(\boldsymbol{a}_{s+i}\) encodes the ego-vehicle \((x,y)\) position and yaw angle, and \(\mathcal{F}_{s+1:s+N}\) describes the anticipated visual evolution of the same driving segment. In practice, we predict latent visual and action representations following [@zhang2025epona], without using history trajectories as input.

Architecture. A frozen text encoder from Wan2.2-TI2V-5B [@wan2025] encodes the instruction \(\mathcal{T}\) into latent tokens \(\mathbf{T}_{s}\in\mathbb{R}^{L_T\times d}\). A 3D-causal VAE adapted from Wan2.2-TI2V-5B encodes history observations into visual latents \(\mathcal{H}^{\mathrm{his}}_s=\{\mathbf{H}_{s-l+1},\ldots,\mathbf{H}_{s}\}\), where \(\mathbf{H}\in\mathbb{R}^{L_V\times d}\). The ego velocity is embedded by an MLP, and the self-attention context tokens are constructed as \[\mathbf{T}_s=\mathrm{Enc}_{\mathrm{text}}(\mathcal{T}), \qquad \mathcal{H}^{\mathrm{his}}_s=\mathrm{VAE}(\mathcal{O}_s), \qquad \mathbf{E}_s=\mathrm{MLP}(v_x,v_y), \qquad \mathcal{C}_s=\mathcal{H}^{\mathrm{his}}_s\cup \mathbf{E}_s . \label{eq:context95tokens}\tag{2}\] Text tokens \(\mathbf{T}_{s}\) are injected through cross-attention.

For generation, future video latent tokens \(\mathcal{V}_{s+1:s+N}\) and future trajectory tokens \(\mathcal{A}_{s+1:s+K}\) are projected into the same hidden dimension \(d\) with modality-specific input projections, modality embeddings, and temporal positional embeddings. The DiT blocks, including self-attention and feed-forward layers, are shared by both modalities, while only the input projections and final prediction heads are modality-specific. The unified prediction process is written as \[(\widehat{\mathbf{v}}^{\mathrm{vid}}, \widehat{\mathbf{v}}^{\mathrm{act}}) = \bigl( h_{\mathrm{vid}}(\mathbf{Z}^{\mathrm{vid}}), h_{\mathrm{act}}(\mathbf{Z}^{\mathrm{act}}) \bigr), \quad [\mathbf{Z}^{\mathrm{vid}},\mathbf{Z}^{\mathrm{act}}] = \mathrm{DiT}_{\theta} \bigl( \mathcal{V}_{s+1:s+N}, \mathcal{A}_{s+1:s+K}, \mathcal{C}_s, \mathbf{T}_s \bigr), \label{eq:unified95dit}\tag{3}\] where \(h_{\mathrm{vid}}\) and \(h_{\mathrm{act}}\) are modality-specific prediction heads for future video latents and trajectory tokens, respectively. Thus, future-video prediction updates the same generative backbone used for trajectory generation instead of regularizing a separated planner.

3.2 Modality-Decoupling Visibility Mask↩︎

Generating future videos and trajectories in one transformer may cause future-target leakage: trajectory tokens may exploit future visual tokens unavailable during trajectory-only inference, while video tokens may become conditioned on future ego motion. We therefore introduce a binary self-attention mask \(M\in\{0,1\}^{L\times L}\) over context tokens \(\mathcal{C}_s\), future video tokens \(\mathcal{V}\), and future trajectory tokens \(\mathcal{A}\), where \(M(i,j)=1\) allows token \(i\) to attend to token \(j\) and \(M(i,j)=0\) blocks this attention. Text tokens are excluded since they are injected through cross-attention.

The mask allows future video and trajectory tokens to attend to historical context, i.e., \(M(i,j)=1\) for \(i\in\mathcal{V}\cup\mathcal{A}\) and \(j\in\mathcal{C}_s\), but prevents context tokens from attending back to future targets. It also blocks future video-action attention in both directions, i.e., \(M(i,j)=0\) for \(i\in\mathcal{V},j\in\mathcal{A}\) or \(i\in\mathcal{A},j\in\mathcal{V}\), while preserving full attention within \(\mathcal{C}_s\), \(\mathcal{V}\), and \(\mathcal{A}\). This design shares historical context across modalities, prevents causal leakage between future targets, and maintains intra-modality coherence for video generation and trajectory denoising.

3.3 Flexible Inference Controller↩︎

The proposed mask enables flexible inference by instantiating only the requested future token groups. For trajectory-only rollout, future video tokens are removed and the model denoises \(\mathcal{A}_{s+1:s+K}\) conditioned on \(\mathcal{C}_s\) and \(\mathbf{T}_s\). For video-only rollout, future trajectory tokens are removed and the model denoises \(\mathcal{V}_{s+1:s+N}\) from the same conditions. Because trajectory tokens are already blocked from future video tokens during training, removing the video branch at inference does not change their attention dependency, allowing trajectory-only inference to bypass future-video diffusion and improve efficiency by \(4.3\times\).

For joint video-action rollout, both future token groups are instantiated and denoised within the same shared DiT under the Modality-Decoupling Visibility Mask. Although future video and trajectory tokens remain mutually blocked, they are aligned through the shared denoising backbone, common historical context, and joint co-training, enabling interpretable rollout while preserving the causal structure used for trajectory-only deployment.

3.4 Training Objective↩︎

We train UNIVERSE with flow matching on both future trajectory tokens and future video latents. For a clean target sample \(\mathbf{Y}_0^m\) from modality \(m\in\{\mathrm{act},\mathrm{vid}\}\), we sample a flow time \(p\sim\mathcal{U}(0,1)\) and Gaussian noise \(\boldsymbol{\epsilon}^m\sim\mathcal{N}(0,I)\). The noisy sample and target velocity are \[\mathbf{Y}_p^m=(1-p)\boldsymbol{\epsilon}^m+p\mathbf{Y}_0^m, \qquad \dot{\mathbf{Y}}_p^m=\mathbf{Y}_0^m-\boldsymbol{\epsilon}^m .\]

The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask: \[(\hat{\mathbf{v}}_{\theta}^{\mathrm{act}}, \hat{\mathbf{v}}_{\theta}^{\mathrm{vid}}) = f_{\theta} ( \mathbf{Y}_p^{\mathrm{act}}, \mathbf{Y}_p^{\mathrm{vid}}, p, \mathcal{C}_s, \mathbf{T}_s; M ),\] where \(f_{\theta}\) denotes the shared DiT with modality-specific prediction heads. The final objective is \[\mathcal{L} = \mathcal{L}_{\mathrm{act}} + \lambda_{\mathrm{vid}}\mathcal{L}_{\mathrm{vid}}, \qquad \mathcal{L}_{m} = \mathbb{E} \left[ \left\| \hat{\mathbf{v}}_{\theta}^{m} - \dot{\mathbf{Y}}_p^{m} \right\|_2^2 \right], \quad m\in\{\mathrm{act},\mathrm{vid}\}.\] Here, \(\lambda_{\mathrm{vid}}\) balances the two objectives. By default, \(\lambda_{\mathrm{vid}}>0\), so future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation.

3.5 UNIVERSE Variants↩︎

We instantiate several variants to isolate the effects of video co-training, mask-based causal decoupling, and shared DiT parameterization. All variants use the same observations, trajectory targets, tokenization, training schedule, and evaluation protocol unless otherwise specified.

UNIVERSE. The full model co-trains future video latents and trajectory tokens within one shared DiT under the Modality-Decoupling Visibility Mask.

UNIVERSE w.o. Mask. Following joint video-action generation designs [@ye2026world; @shen2025videovla], we remove the Modality-Decoupling Visibility Mask, allowing future video and trajectory tokens to attend to each other. This keeps the unified DiT and joint objective, but requires coupled video-action generation at inference.

UNIVERSE w.o. video co-train. This variant keeps the architecture, tokenization, mask, and inference procedure unchanged, but removes video supervision by setting \(\lambda_{\mathrm{vid}}=0\), namely \(\mathcal{L}=\mathcal{L}_{\mathrm{act}}\).

UNIVERSE Two-DiT. Following Fast-WAM [@yuan2026fast], we replace the shared DiT with two separated DiTs for video and trajectory denoising. They share the same conditions and losses but not transformer parameters, preserving video co-training while removing shared video-action parameterization.

4 Experiments↩︎

4.1 Datasets and Implementation Details↩︎

NAVSIM v1. Following prior works [@zhang2025epona; @xia2025drivelaw], we adopt NAVSIM v1 [@dauner2024navsim], built upon OpenScene [@contributors2023openscene], as our main closed-loop planning benchmark. It reports No Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C.), and Ego Progress (EP), aggregated as \(\mathrm{PDMS}=\mathrm{NC}\times \mathrm{DAC}\times (5\mathrm{EP}+5\mathrm{TTC}+2\mathrm{C.})/12\). Unless otherwise specified, our model only uses front-view camera input without map, box, depth, occupancy, or LiDAR supervision.

nuScenes. For cross-dataset evaluation, we directly test the NAVSIM-trained model on the nuScenes validation split of 150 scenes from the official 1,000-scene dataset [@caesar2020nuscenes]. Following prior planning works [@hu2023planning; @jiang2023vad], we report L2 displacement error and collision rate at 1 s, 2 s, and 3 s horizons. We also report FID and FVD for future-video generation quality following driving world-model literature [@wang2024drivedreamer; @gao2024vista].

Bench2Drive. Bench2Drive [@jia2024bench2drive] is a CARLA v2 benchmark [@dosovitskiy2017carla] with diverse interactive scenarios and routes. We use it for real-to-simulation zero-shot transfer by directly evaluating the NAVSIM-trained model on its validation split, where the model must handle shifts in visual appearance, vehicle dynamics, and agent behaviors [@hu2023simulation].

Implementation Details. Training is conducted on NVIDIA H20 GPUs with AdamW, a learning rate of \(10^{-4}\), weight decay of \(0.01\), and distributed bf16 mixed-precision training. We first train with a batch size of \(80\) for \(20\)k steps for fast convergence, and then continue training for another \(10\)k steps with an effective batch size of \(640\) using gradient accumulation. We use a \(1\)k-step linear warm-up from \(10^{-3}\) of the base learning rate, followed by a constant learning-rate schedule. The training objective combines a flow-matching loss for future video supervision and a trajectory prediction loss. During inference, we use the trajectory-only rollout mode with \(2\) flow sampling steps unless otherwise stated. Wan2.2-TI2V-5B [@wan2025] is used as the pre-trained video generation backbone. Each training sample contains \(l=4\) history frames and \(N=8\) future frames at a resolution of \(832\times480\).

6pt

4.2 Comparison Results on NAVSIM v1↩︎

Table 1 reports closed-loop planning results on NAVSIM v1. UNIVERSE achieves a PDMS of \(91.0\), outperforming prior traditional end-to-end planners and recent world-model-based planners. Compared with DiffusionDrive [@liao2025diffusiondrive], our method improves PDMS from \(88.1\) to \(91.0\). Compared with WoTE [@li2025end_wote], which uses both camera and LiDAR inputs, UNIVERSE still obtains a higher PDMS using only the front-view camera. The gain is mainly reflected in DAC, TTC, and EP, indicating that the unified video-action DiT improves both safety and progress in closed-loop planning.

These results support our central hypothesis: future-video supervision is more effective for planning when it directly updates the same generative parameters used for trajectory denoising. Instead of using future videos only as an auxiliary output or a separate imagination module, UNIVERSE co-trains video latents and trajectory tokens in one shared DiT, allowing dense visual dynamics to regularize action generation. The UNIVERSE variant rows are included to preview the controlled comparison results; since these variants are designed to isolate architectural factors rather than serve as standalone baselines, we defer their detailed discussion to Sec. 4.5.

1.5pt

4.3 Zero-Shot Generalization on nuScenes and Bench2Drive↩︎

From NAVSIM to nuScenes. To evaluate cross-dataset action generalization, we directly test the NAVSIM-trained UNIVERSE on nuScenes without any fine-tuning. As shown in Table 2, UNIVERSE achieves an average L2 error of \(0.83\) m and an average collision rate of \(0.03\%\). Compared with PWM [@zhao2025forecasting], which is also trained on NAVSIM and evaluated zero-shot, UNIVERSE reduces the average L2 error from \(3.99\) m to \(0.83\) m and the average collision rate from \(0.36\%\) to \(0.03\%\). Compared with DriveVLA-W0 [@li2025drivevlaw0], UNIVERSE also improves the average L2 error from \(1.43\) m to \(0.83\) m and the average collision rate from \(0.77\%\) to \(0.03\%\). These results suggest that unified video-action co-training learns transferable planning priors rather than merely fitting NAVSIM-specific trajectories.

From NAVSIM to Bench2Drive. We further evaluate zero-shot transfer from real-world logs to simulation on Bench2Drive. As reported in Table 2, UNIVERSE obtains an average L2 error of \(1.33\) m and an average collision rate of \(1.19\%\). Compared with PWM [@zhao2025forecasting], UNIVERSE reduces the average L2 error from \(2.80\) m to \(1.33\) m and the average collision rate from \(3.76\%\) to \(1.19\%\). Fig. 4 further provides a qualitative zero-shot example: when a stationary vehicle blocks the lane ahead, UNIVERSE predicts a reasonable bypass maneuver, while the Two-DiT variant tends to stop behind the obstacle. This indicates stronger driving-intention understanding under real-to-simulation domain shift.

3.2pt

4.4 Future-Video Generation Quality↩︎

Although motion planning is our primary goal, UNIVERSE also generates high-quality future videos. As shown in Table 3, it achieves the best FID of \(6.7\) and FVD of \(81.7\) on nuScenes among the compared driving video-generation methods. This indicates that the video branch learns realistic and temporally coherent scene evolution, while the shared DiT allows such video dynamics modeling to regularize trajectory generation.

5pt

4.5 Controlled Comparison with UNIVERSE Variants↩︎

The UNIVERSE variants are designed to answer which architectural factors are responsible for the observed action generalization. All variants use the same observations, trajectory targets, tokenization, training schedule, and evaluation protocol, but differ in one key factor: whether future-video supervision is used, whether future video and trajectory tokens are causally decoupled by the visibility mask, and whether video and trajectory denoising share the same DiT parameters. We compare them jointly across in-domain planning, zero-shot transfer, video-action consistency, inference latency, and memory consumption.

Dense video supervision improves transfer. Removing future-video supervision reduces NAVSIM PDMS from \(91.0\) to \(88.2\) in Table 1. The degradation becomes more evident under domain shift: on nuScenes, the average L2 error increases from \(0.83\) m to \(1.02\) m and the collision rate increases from \(0.03\%\) to \(0.35\%\); on Bench2Drive, the collision rate increases from \(1.19\%\) to \(1.57\%\). This confirms that future-video prediction is not merely an auxiliary visualization task, but provides dense dynamics supervision that improves transferable action generation.

Shared denoising backbone matters. UNIVERSE Two-DiT preserves both video and trajectory objectives but separates them into two diffusion transformers. Although it still benefits from video co-training at the system level, its NAVSIM PDMS drops to \(89.6\), and its zero-shot performance is consistently weaker than the shared-DiT UNIVERSE. On nuScenes, the average L2 error increases from \(0.83\) m to \(0.92\) m and the collision rate from \(0.03\%\) to \(0.21\%\); on Bench2Drive, the collision rate increases from \(1.19\%\) to \(1.82\%\). Qualitatively, Fig. 4 shows that the Two-DiT variant tends to stop behind a stationary obstacle, while UNIVERSE predicts a bypass trajectory, indicating stronger driving-intention understanding from shared video-action denoising. It also produces the largest video-action consistency error in Table ¿tbl:tab:dpvo95consistency? and the highest training peak memory in Table [tbl:tab:efficiency]. These results show that video co-training is most effective when video and trajectory losses update the same generative backbone.

Visibility decoupling enables flexible deployment. The no-mask variant obtains comparable in-domain performance with a PDMS of \(90.9\), but this does not translate into better deployment behavior. Because future video and trajectory tokens can attend to each other, the model relies on coupled video-action rollout and cannot remove one future modality without changing the learned dependency structure. This leads to weaker zero-shot safety: the collision rate increases from \(0.03\%\) to \(0.06\%\) on nuScenes and from \(1.19\%\) to \(1.79\%\) on Bench2Drive. It also increases trajectory inference latency from \(376\) ms to \(1623\) ms, since future-video denoising must be retained during rollout. Therefore, the visibility mask is important not only for causal validity, but also for flexible trajectory-only deployment.

video-action consistency and efficiency. The qualitative results in Fig. 3 show that generated future videos and predicted trajectories evolve consistently. To quantify this alignment, we use DPVO [@teed2023deep] to reconstruct ego-motion from ground-truth and generated future videos, align the reconstructed trajectories to their references with a similarity transform, and report the average L2 error over the future \(4\) s horizon. As shown in Table ¿tbl:tab:dpvo95consistency?, UNIVERSE achieves an average error of \(0.13\), close to the GT-reference error of \(0.08\), while the no-mask and Two-DiT variants increase the error to \(0.15\) and \(0.19\). Meanwhile, Table [tbl:tab:efficiency] shows that UNIVERSE achieves the lowest latency, inference memory, and training memory among the variants. Together, these results support the coupling-by-parameters and decoupling-by-visibility design: video and trajectory objectives should share the same DiT parameters, while their future tokens should remain causally separated.

4.6 Additional Ablation on Horizon and Sampling↩︎

Table ¿tbl:tab:ablation? studies the number of future video frames and flow sampling steps. Using \(4\) future frames provides insufficient long-horizon supervision, while \(8\) frames achieves the best PDMS of \(91.0\). Further increasing the horizon to \(12\) frames brings no additional gain. For sampling, one step is unstable, whereas two steps already recover full performance and three steps offer negligible improvement. We therefore use \(8\) future frames and \(2\) sampling steps by default.

5 Conclusion↩︎

In this paper, we propose UNIVERSE, a unified video-action diffusion model for autonomous driving. Rather than simply adding future-video prediction, UNIVERSE lets dense video supervision update the same DiT parameters used for trajectory denoising, enabling video-learned scene dynamics and temporal priors to directly regularize action generation.

To ensure causal validity and efficient deployment, we introduce the Modality-Decoupling Visibility Mask, which shares historical context while blocking mutual attention between future video and trajectory tokens. This design supports joint video-action rollout, video-only simulation, and efficient trajectory-only planning. Experiments show that UNIVERSE achieves state-of-the-art NAVSIM performance and strong zero-shot generalization to nuScenes and Bench2Drive, while providing \(4.3\times\) faster trajectory inference and lower inference/training memory than coupled or Two-DiT variants. These results demonstrate that shared-parameter video-action co-training with mask-modulated modality decoupling is effective for generalizable autonomous-driving planners.

6 Supplementary Material↩︎

This supplementary material provides implementation details, extended analyses, and additional qualitative results that complement the main paper. It is organized as follows:

• Additional architecture details specify the block form of the Modality-Decoupling Visibility Mask and summarize the token configurations for trajectory-only, video-only, and joint video-action inference.

• DPVO-based verification describes how we use an external visual-odometry system to measure whether generated future videos and predicted trajectories imply consistent ego motion.

• Additional nuScenes comparison extends the main zero-shot evaluation by comparing UNIVERSE with methods trained or fine-tuned on nuScenes.

• Additional zero-shot visualization presents qualitative results on unseen nuScenes and Bench2Drive scenes, including diverse driving behaviors, nighttime conditions, and foggy simulation scenes.

• Limitations discusses the current scope of evaluation, including the front-view setting and the training cost introduced by the video-generation backbone.

• Licenses for existing assets summarizes the public datasets, benchmarks, pretrained models, and evaluation tools used in this work, together with their corresponding usage terms.

• Broader impacts discusses potential benefits, deployment cautions, and the computational cost of video-action world models for autonomous-driving planning.

7 Additional Architecture Details↩︎

7.1 Visibility Mask Implementation↩︎

The main paper introduces the Modality-Decoupling Visibility Mask conceptually. Here we provide its explicit block form. Let the self-attention token groups be historical context tokens \(C\), future video tokens \(V\), and future trajectory tokens \(A\). The binary visibility mask is \[\mathbf{M} = \begin{array}{c|ccc} & C & V & A \\ \hline C & 1 & 0 & 0 \\ V & 1 & 1 & 0 \\ A & 1 & 0 & 1 \end{array},\] where rows denote query tokens, columns denote key/value tokens, \(1\) means attention is allowed, and \(0\) means attention is blocked.

This mask enforces three constraints. First, both future modalities read the same historical context through \(V\!\rightarrow\!C\) and \(A\!\rightarrow\!C\) attention. Second, context tokens remain read-only with respect to future targets, since \(C\!\rightarrow\!V\) and \(C\!\rightarrow\!A\) are blocked. This prevents context tokens from acting as an indirect information bridge between future video and trajectory tokens. Third, future video and trajectory tokens are mutually invisible through blocked \(V\!\rightarrow\!A\) and \(A\!\rightarrow\!V\) attention. Thus, video and trajectory denoising share model parameters but remain causally separated at the future-token level.

Full intra-modality attention is preserved within \(V\) and within \(A\). This is valid because UNIVERSE predicts future video and trajectory chunks with diffusion-style denoising rather than autoregressive next-step decoding. Bidirectional attention within \(V\) helps maintain temporal coherence across generated frames, while bidirectional attention within \(A\) helps produce smooth trajectory chunks.

7.2 Inference Token Configurations↩︎

At inference time, UNIVERSE uses the same trained DiT weights but instantiates different future token groups. Table 4 summarizes the token configurations for each inference mode.

4pt

The key property is dependency consistency. During training, trajectory tokens are blocked from attending to future video tokens. Therefore, removing \(V\) during trajectory-only inference does not change the attention dependency of the trajectory branch. This differs from naively coupled video-action generation, where removing future video tokens at test time would alter the learned dependency structure and introduce a train–test mismatch.

8 DPVO-based Verification of video-action Consistency↩︎

We use DPVO [@teed2023deep] as an external visual-odometry system to verify whether generated videos and predicted trajectories describe consistent ego motion. For each sample, DPVO is run on both the ground-truth future video and the future video generated by UNIVERSE. This produces two reconstructed trajectories, denoted as DPVO(gt img) and DPVO(pred img). We compare DPVO(gt img) with the ground-truth ego trajectory, and DPVO(pred img) with the trajectory predicted by UNIVERSE.

Since monocular visual odometry is ambiguous up to scale, rotation, and translation, we align each reconstructed trajectory to its reference using a 2D similarity transform. Let \(\tau=\{\mathbf{p}_t\}_{t=1}^{T}\) be the reference trajectory and \(\hat{\tau}=\{\hat{\mathbf{p}}_t\}_{t=1}^{T}\) be the DPVO-reconstructed trajectory on the ground plane. We solve \[(s^\star,R^\star,\mathbf{t}^\star) = \arg\min_{s,R,\mathbf{t}} \sum_{t=1}^{T} \left\| sR\hat{\mathbf{p}}_t+\mathbf{t}-\mathbf{p}_t \right\|_2^2 ,\] and compute the aligned trajectory as \(\tilde{\mathbf{p}}_t=s^\star R^\star\hat{\mathbf{p}}_t+\mathbf{t}^\star\). The final consistency error is the average L2 distance over the future 4s horizon: \[\mathrm{Avg.\;L2}(\tilde{\tau},\tau)= \frac{1}{T}\sum_{t=1}^{T} \left\|\tilde{\mathbf{p}}_t-\mathbf{p}_t\right\|_2 . \label{eq:supp95avg95l295consistency}\tag{4}\]

As reported in Table ¿tbl:tab:dpvo95consistency?, UNIVERSE achieves an average DPVO-based error of \(0.13\), close to the GT-reference error of \(0.08\). This indicates that the ego motion implied by the generated future videos is consistent with the trajectory predicted by the model. Fig. 5 and Fig. 7 further shows that DPVO(pred img) closely follows the predicted trajectory across diverse zero-shot driving scenarios, supporting the video-action consistency of the unified generative process.

9 Additional Comparison on nuScenes↩︎

Table ¿tbl:tab:supp95plan95nusc? provides an extended comparison on the nuScenes planning benchmark. Different from Table 2 in the main paper, which focuses on zero-shot transfer from NAVSIM to nuScenes and Bench2Drive, this table additionally includes methods trained or fine-tuned on nuScenes. Although UNIVERSE is directly evaluated on nuScenes without any fine-tuning, it achieves lower L2 error and collision rate than the listed nuScenes-trained or fine-tuned baselines. This further suggests that unified video-action co-training learns transferable planning priors instead of only fitting the source-domain trajectory distribution.

3pt

10 Additional Zero-Shot Visualization↩︎

Fig. 6 provides additional qualitative results on unseen datasets. The model is trained on NAVSIM and directly evaluated on nuScenes and Bench2Drive without fine-tuning. For nuScenes, DPVO [@teed2023deep] reconstructs ego trajectories from both ground-truth and generated future videos. The close alignment between predicted trajectories, generated video evolution, and DPVO-reconstructed motion indicates that the video and trajectory branches learn consistent future dynamics even under domain shift. For Bench2Drive, the generated future frames and trajectories also remain temporally coherent despite the real-to-simulation visual and dynamics gap.

11 Limitations↩︎

Although UNIVERSE achieves strong closed-loop planning and zero-shot generalization, several limitations remain. First, we mainly evaluate front-view camera input, which keeps the setting controlled but may miss side or rear-view cues in highly interactive scenes. Extending UNIVERSE to multi-view inputs is an important future direction. Second, trajectory-only inference is efficient, but training still relies on a large video-generation backbone, increasing memory and compute compared with purely trajectory-supervised planners. Lighter video tokenizers, smaller DiT backbones, or distillation may reduce this cost.

12 Licenses for Existing Assets↩︎

We use public datasets, benchmarks, models, and evaluation tools according to their official licenses and terms of use.

NAVSIM and OpenScene. We use NAVSIM for closed-loop planning evaluation. The NAVSIM repository states that its code and assets are released under Apache-2.0 unless otherwise specified, while datasets used by NAVSIM inherit their own distribution licenses. OpenScene code is released under Apache-2.0, whereas the OpenScene data are based on nuPlan and distributed under Creative Commons Attribution-NonCommercial-ShareAlike and the nuPlan Dataset License Agreement for Non-Commercial Use. We follow these licenses and use the data only for research evaluation.

nuScenes. We use nuScenes for cross-dataset zero-shot evaluation and future-video generation evaluation. nuScenes is used under its official data license and terms of use, commonly associated with CC BY-NC-SA 4.0 for non-commercial research use. We use the official validation split only for research evaluation.

Bench2Drive and CARLA. We use Bench2Drive for real-to-simulation zero-shot transfer evaluation. The Bench2Drive dataset page states that its assets and code are released under Apache-2.0 unless specified otherwise. Bench2Drive is built on CARLA; CARLA-specific code is released under the MIT license, CARLA-specific assets under CC-BY, and third-party dependencies such as Unreal Engine follow their own licenses. We follow the corresponding licenses and use the benchmark only for research evaluation.

Wan2.2-TI2V-5B. We use Wan2.2-TI2V-5B as the pretrained video-generation backbone. The model card releases the model under Apache-2.0. We follow its license and release terms.

DPVO. We use DPVO only as an evaluation tool for ego-motion reconstruction from ground-truth and generated videos. It is not used as a training signal. The official DPVO repository is released under the MIT license.

Baselines. For all compared baselines, we cite the corresponding papers and follow the stated benchmark protocols. When results are taken from prior literature, we keep the original evaluation settings as closely as possible. Released code, checkpoints, and assets remain under the licenses of their original authors.

13 Broader Impacts↩︎

This work studies video-action world models for autonomous-driving planning. Its potential benefit is improved action generalization under domain shift, while trajectory-only inference reduces the cost of test-time video rollout. Training large video-action models also requires substantial compute, motivating future work on efficient training, compression, and distillation.