July 07, 2026
Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present Point as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that Point as Skeleton improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.
End-to-end autonomous driving (E2E-AD) [1]–[5] has attracted increasing attention in recent years. A central challenge in evaluating E2E-AD systems is to provide sensor observations that respond to the actions selected by the driving policy. Open-loop benchmarks such as nuScenes [6] provide realistic sensor data, but the observations are fixed by the recorded trajectory and cannot react to policy-induced actions. In contrast, closed-loop simulators such as CARLA [7]–[9] provide interactive feedback, but their rendered sensor appearance can still differ from real-world camera data. This motivates generative sensor simulation, where a learned generative model synthesizes camera observations from simulator-provided ego states, actor states, and scene conditions.
However, closed-loop simulation goes beyond standard driving-scene reconstruction [10] and generation [11], [12]. The ego vehicle may deviate from the logged trajectory, while the planner and traffic simulator update online. The generator therefore needs to render multi-view observations under off-log motions and repeatedly refreshed conditions. This introduces two additional requirements:
(I) Spatiotemporal Extrapolation. Closed-loop policies may deviate from the logged motion in speed or heading, exposing vehicles and pedestrians that are not observed in the recorded images. This is challenging for classical reconstruction pipelines [13]. Recent methods [14], [15] use generative models as reconstruction post-processing to improve visual plausibility, but they require scene- or frame-specific processing, reducing their ease of deployment.
(II) Instantaneous Interactivity. Closed-loop simulation updates the planner action at every step, requiring the simulator to synthesize the next observation from the latest scene state. Reconstruction-based renderers can be repeatedly queried once a scene is built. However, reconstruction-generation hybrids [16], [17] and urban-scene generators [18], [19] usually assume pre-defined conditions over a full clip. This calls for frame-wise autoregressive generation with step-wise condition updates.
Existing studies [20], [21] show that directly applying full-clip video generators to frame-wise autoregressive rollout causes severe error accumulation (Fig. 3). We therefore propose Point as Skeleton, which uses point-cloud assets accumulated once offline as stabilizing conditions for autoregressive generation. The resulting simulator uses an autoregressive video generator as the visual backbone, while the point skeleton provides editable geometric cues to reduce error accumulation. To realize this framework, we make the following designs.
Reset-and-Roll autoregressive generation. We observe that Rolling Diffusion [22] inference is better suited to highly dynamic driving scenes, but its input-output timestamps are not aligned with the simulator interface. A simulator step expects the generator to consume the current scene condition and immediately return the next observation, whereas rolling diffusion emits the earliest observation in its rolling window after using lookahead conditions. Reset-and-Roll bridges this gap by allowing rolling-style denoising while resetting the committed latent state before future-conditioned information is passed to the next simulation step.
Point-skeleton conditions. We design two point-cloud conditions to stabilize autoregressive generation. The painted point cloud [23] provides pixel-level appearance guidance and suppresses visual degradation during long rollout. However, painted points may suffer from reconstruction inconsistency when actor instances are only partially scanned, motivating a parallel depth condition as a stable geometric anchor. The template-based depth branch replaces foreground actors with category-level point templates, providing stable depth guidance for autoregressive rollout.
nuPlan SimGen. We implement a closed-loop generative interface on nuPlan [24]. The interface attaches our visual generator to the nuPlan traffic simulation core, allowing the ego vehicle to change speed and heading according to user-defined commands or nuPlan planners. It enables evaluation of autoregressive generation under trajectories that deviate from the original log and provides projected point-based semantic labels for perception-oriented evaluation.
Conditional diffusion models have shown strong controllability in visual generation [25], [26]. As conditioning signals extend from text and semantic maps to 3D boxes, HD maps, ego motion, and camera geometry, these models have become increasingly applicable to driving-scene generation [27]–[30]. A line of work studies driving video generation as predictive world modeling [11], [12], [18], [19], [31]–[33]. These methods synthesize future scene observations under given actions or scene-level conditions, and introduce driving-specific designs such as structured trajectory tokens [12], action-controllable future prediction [31], and depth or occupancy conditions [18], [19]. Their main focus is to generate plausible future clips under a specified condition sequence.
More recent works further explore autoregressive driving world models. Epona [34] formulates driving world modeling as next-token prediction and jointly studies generation and planning. OmniNWM [35] extends this direction toward panoramic navigation world modeling with state, action, and reward awareness. These methods indicate a shift from fixed-length video synthesis toward longer-horizon and more interactive world modeling. However, their design objective still differs from closed-loop simulation. They are generally designed to predict future pixels and actor motions from historical observations and action-related inputs, rather than to render observations under externally updated scene layouts. As a result, fine-grained control over individual participants is less direct than in explicit layout-conditioned simulation, where 3D boxes, HD maps, and simulator states can be updated at every step.
Driving scene simulation aims to provide sensor observations for interactive evaluation of autonomous-driving policies. Classical simulators such as CARLA [7] support closed-loop interaction, but their rendered appearance often differs from real-world camera data. Reconstruction-based methods improve visual fidelity by reconstructing urban scenes from recorded logs [10], [36], yet they are most reliable near the logged trajectory. When the ego vehicle deviates from the recorded motion, unseen regions and weakly observed actors become difficult to render faithfully.
Recent methods introduce generative priors to alleviate this limitation. FreeVS [16] improves free-viewpoint synthesis under shifted driving trajectories, ReCamDriving [14] conditions generation on reconstructed scenes, and R3D2 [15] refines projected dynamic instances with generative models. These reconstruction-generation hybrids improve visual plausibility beyond logged views, but they still rely on scene-specific reconstruction, object-level editing, or post-processing modules. Therefore, extending them to continuous closed-loop rollout with online condition updates is not straightforward.
Generative simulation systems move one step further by using learned world models as simulator engines. DriveArena [37] builds a closed-loop generative simulation platform, where generated observations are consumed by a driving agent, the resulting trajectory updates the scene layout, and the updated layout is fed back to the generator. Closed-source systems such as Waymo World Model [38] and GAIA-3 [39] further demonstrate controllable autonomy evaluation with action-, layout-, and scenario-conditioned generation. These systems suggest that driving simulation is moving from graphics- or reconstruction-centered pipelines toward generation-centered closed-loop simulators. Nevertheless, open autoregressive video-generation designs that can be readily integrated with closed-loop driving systems remain limited.
Overview. We build a frame-wise generative rendering interface for closed-loop-compatible autonomous driving simulation: each updated simulator state defines the next scene layout, and the generator synthesizes the corresponding camera observation. Our method includes (i) an autoregressive video diffusion model with diffusion-forcing training and Reset-and-Roll inference for stable rollouts (Sec. 3.1), (ii) a Point Cloud Skeleton projected as color and depth conditions for view-consistent geometry (Sec. 3.2), and (iii) nuPlan-SimGen, an interactive generative rendering interface and evaluation protocol for ego-deviated trajectories (Sec. 3.3).
In closed-loop simulation, the generator acts as a step-wise visual renderer. At simulation step \(t\), the planner outputs a short-horizon trajectory as discrete ego waypoints. The simulator commits only the next executable waypoint as the current ego state, while the remaining waypoints are treated as short-horizon forecasts. We therefore distinguish between committed states and forecast states. Committed states include the generated history, the current ego state, the current actor states, and the simulator layout \(\mathcal{L}^{\mathrm{sim}}_t\) at step \(t\). Forecast states include future ego waypoints and their corresponding future layouts \(\hat{\mathcal{L}}_{t+1:t+K}\), which may be revised after the policy receives the newly generated observation.
As shown in the left part of Fig. 1, the traffic simulator places the committed ego and actor states on the map to construct the current layout \(\mathcal{L}^{\mathrm{sim}}_t\). Future layouts are constructed in the same format, but they remain mutable forecasts rather than simulator states to be inherited. The video generator is therefore invoked at every simulation step, conditioned on the generated history, the current committed layout, and the forecast layouts within the generation window.
This setting casts generative simulation as a frame-wise autoregressive video generation problem: each synthesized observation is returned to the policy and affects the next simulator query. Without dedicated training or rollout design, autoregressive diffusion suffer from severe error accumulation [20], [21], making standard autonomous-driving video generators difficult to use as interactive renderers, as shown in Fig. 3-(2). Existing works alleviate this issue by noising frames during training and inference [20], [35], or by simulating inference-time histories during training [34]. However, when deployed as closed-loop renderers, full-sequence inference still suffers from two failure modes.
Causal full-sequence. As shown in Fig. 2, if the generator avoids future layouts, the next-frame distribution is marginalized over possible future motions: \[p(x_{t+1}\mid h_t)= \int p(x_{t+1}\mid h_t,f_t)p(f_t\mid h_t)\,df_t ,\] where \(h_t\) denotes the committed history and current simulator layout, and \(f_t\) denotes future layout trajectories. This removes explicit motion cues during denoising; thus low-speed or occluded actors are often generated conservatively, causing weak or static motion in rollout (Fig. 3-(3),(4)).
Non-causal full-sequence. Lookahead layouts improve dynamics, but full-sequence non-causal inference entangles the current output with unrealized forecasts. In closed-loop rollout, the lookahead used at step \(t\) may change after the policy receives the generated observation. Once observations or latent states produced under the old forecast are reused in later queries, future-conditioned information becomes part of the committed rollout state, causing state mismatch and accumulated errors.
We therefore adopt rolling-window diffusion, which we find better suited to step-wise high-dynamic rendering. Instead of denoising a full future clip at once, rolling diffusion maintains a FIFO latent window: each roll injects a new noisy latent, denoises the window, and emits one frame. This rolling form naturally matches the simulator’s step-wise query pattern and avoids committing an entire future-conditioned sequence. It also allows predicted layouts to guide near-future dynamics, which improves instance motion compared with causal full-sequence inference, as shown in Fig. 3-(5). However, naive rolling still exposes the inherited latent state to predicted layouts, so future-conditioned information can be carried into the next simulator query.
Reset-and-Roll aligns rolling inference with the closed-loop simulator interface as shown in Fig. 2. At each step, the window contains \((L-1)\) history frames and one current-step latent. After the first denoising update, before injecting any lookahead layout, we cache the current latent and timestep. This cached latent has only seen the committed history and the current simulator layout \(\mathcal{L}^{\mathrm{sim}}_t\). We then continue rolling denoising with predicted layouts \(\hat{\mathcal{L}}_{t+1:t+K}\) to render the current observation with stronger dynamic cues. Before moving to the next simulator step, all lookahead-exposed latents are discarded. The next query is formed with the newly realized layout and the cached latent at the same diffusion scheduler timestep. Thus, predicted layouts are used for temporary denoising in the current query, but future-conditioned latent states are not committed across steps.
To address the error accumulation observed in AR generation (Fig. 3-(5)), we introduce the Point Cloud Skeleton as a training-inference-consistent scene representation. It accumulates offline LiDAR observations into static background and track-indexed foreground assets, which are recomposed by the simulator and projected to camera views at each rollout step. The projected skeleton provides sparse but state-aligned current-view evidence, turning each step into conditional rendering rather than pure history-based extrapolation. Thus, the primary role of the skeleton is to provide a consistent state representation during rollout, rather than to reconstruct the scene at the pixel level.
As illustrated in Fig. 4, we colorize each LiDAR frame by projecting points onto calibrated images, obtaining \(P_t^c\). Using 3D boxes and track IDs, we split it into background and foreground points: \[P_t^c = P_t^{\mathrm{bg}} \cup P_t^{\mathrm{fg}}, \quad P_t^{\mathrm{fg}}=\bigcup_{i\in\mathcal{A}_t} P_{t,i}^{\mathrm{fg}}, \quad P_t^{\mathrm{bg}}\cap P_t^{\mathrm{fg}}=\emptyset ,\] where \(\mathcal{A}_t\) denotes the actors observed at frame \(t\). We accumulate background points in the global frame, and normalize each actor’s foreground points into canonical coordinates: \[P_{\mathrm{global}}^{\mathrm{bg}} = \bigcup_{t=1}^{N} T_{\mathrm{global}}(t) P_t^{\mathrm{bg}}, \quad P_{\mathrm{actor}(i)}^{\mathrm{fg}} = \bigcup_{t:i\in\mathcal{A}_t} B_{t,i}^{-1} P_{t,i}^{\mathrm{fg}} .\] Here \(T_{\mathrm{global}}(t)\) maps frame-\(t\) points to the global frame, and \(B_{t,i}\) denotes the 3D box pose of actor \(i\). Thus, \(B_{t,i}^{-1}\) recenters and aligns foreground points into canonical coordinates, forming a track-indexed actor asset. During simulation, the simulator places each actor according to box pose, merges it with \(P_{\mathrm{global}}^{\mathrm{bg}}\), and projects the composed skeleton to each camera view as the color map condition.
The projected skeleton is used through two complementary branches. The color branch preserves pixel-level appearance from accumulated colorized points, providing degraded but useful appearance anchors for autoregressive rendering. However, foreground actors are often only partially scanned in the logs. Under trajectory drift, their color projections may become sparse or view-inconsistent, creating a training-inference gap that is amplified during rollout. We therefore introduce a depth branch to provide more stable geometry. We cluster training-log 3D boxes by category, and aggregate LiDAR points within each cluster to build category-level point templates for common traffic participants, such as sedan, SUV, and pedestrian. During simulation, these templates are placed according to simulator-provided actor boxes and projected as depth maps. The color map supplies appearance anchors, while the template-depth map supplies geometry. We concatenate both maps with the box&hdmap layout as conditions for the driving world model.
To connect generative models with practical closed-loop simulation, we implement nuPlan-SimGen, a nuPlan [24] plugin for closed-loop generative rendering. The plugin attaches the visual generator to the nuPlan traffic simulator: after each planner action, the simulator updates ego and actor states, and the generator renders the corresponding camera observations. Enabled by our template-based depth map, nuPlan-SimGen also supports a simulator-state alignment proxy under deviated ego trajectories. We project movable-object point templates to obtain vehicle pseudo masks and compare them with vehicle masks predicted by an off-the-shelf segmentor, reporting projected-mask IoU (Fig. 5).
As shown in Fig. 5, nuPlan-SimGen evaluates generative rendering under ego-trajectory drift. During the first \(N\) steps, the ego vehicle and surrounding actors follow the logged trajectories, where \(N\) is the number of reference frames required by the generator. Starting from frame \(N\), a predefined lane-change trajectory takes over the ego motion. This trajectory gradually adjusts the ego speed toward a target value of 10m/s and applies a yaw deviation from the recorded path. At each simulator step, the next executable waypoint is committed as the current layout, while the remaining waypoints are treated as future layouts. Surrounding actors can either continue along their logged tracks or be controlled by nuPlan’s IDM policy; for consistency, our evaluation keeps them on the logged tracks. Although their motions are unchanged, the deviated ego pose alters relative viewpoints and visibility, which may bring actors outside the logged camera view into the generated field of view.
| Method | Backbone | Condition | Resolution | \(\mathrm{FID}_{(\downarrow)}\) | \(\mathrm{FVD}_{(\downarrow)}\) |
|---|---|---|---|---|---|
| Panacea+ [41] | SD1.5 | Layout\(+\)Text | 256\(\times\)512 | 15.50 | 103.00 |
| MagicDrive [28] | SD1.5 | Layout\(+\)Text | 224\(\times\)400 | 16.20 | 217.94 |
| DriveArena [37] | SD1.5 | Layout\(+\)Text\(+\)Prev | 224\(\times\)400 | 16.03 | 185.32 |
| MagicDiT [29] | Sora-DiT | Layout\(+\)Text\(+\)Video | 224\(\times\)400 | 20.91 | 97.21 |
| Epona [34] | Flux-DiT | Ego Traj\(+\)Text\(+\)Video | 512\(\times\)1024 | 7.50 | 82.80 |
| Ours (Image) | SD1.5 | Layout\(+\)Point\(+\)Prev | 224\(\times\)400 | 7.09 | 69.47 |
| Ours (Video) | SD3.5 | Layout\(+\)Point\(+\)Video | 256\(\times\)448 | 5.97 | 58.3 |
16pt
We conduct experiments on nuScenes and nuPlan-mini. On nuScenes, we use 700 training and 150 validation scenes, reporting FID, FVD, and UniAD scores [1], [37]. On nuPlan-mini, we use all 64 sequences for model training and select 300 customized-trajectory clips (specific in appendix) for nuPlan-SimGen evaluation. This benchmark evaluates trajectory deviation within logged scenes, rather than scene-level generalization to unseen logs or maps. We report FID and Mask2Former vehicle segmentation scores [42].
We implement two variants of Point as Skeleton: an SDv1.5-based single-frame generator [43] for comparison with image generators [20], [37], and an SDv3.5-based autoregressive video generator for comparison with video generators [34], [44]. The SDv1.5 variant is trained for 200 epochs on nuScenes and 150 epochs on nuPlan, with batch sizes of 16 and 8, respectively. The SDv3.5 variant is trained for 15k iterations with batch size 16 on both datasets. All models use diffusion forcing [40]. We generate 2 Hz images and 6 Hz videos on nuScenes using ASAP labels [45], and 5 Hz videos on nuPlan. More implementation details are provided in the appendix.
We first evaluate Point as Skeleton on nuScenes. In video generation, the model conditions on 2s history and generates a 4s clip at 6 Hz, yielding 24 autoregressive steps. The image variant is evaluated for 16 autoregressive steps at 2 Hz. Table 1 shows the best FID and FVD under this evaluation protocol, indicating improved visual quality and temporal consistency. We further evaluate generated sequences with UniAD. Table 2 shows stronger downstream detection and segmentation, with competitive planning performance. These results suggest that Point as Skeleton better preserves perception-relevant information while maintaining competitive planning performance.
| Source Data | 3DOD \(\uparrow\) | Segmentation \(\uparrow\) | \(\text{L2}\) \(\downarrow\) | ||||||
| mAP | NDS | Lanes | Drivable | Divider | Crossing | 1.0s | 2.0s | 3.0s | |
| ori nuScenes [6] | 37.98 | 49.85 | 31.31 | 69.14 | 25.93 | 14.36 | 0.51 | 0.98 | 1.65 |
| MagicDrive [28] | 12.92 | 28.36 | 21.95 | 51.46 | 17.10 | 6.57 | 0.57 | 1.14 | 1.95 |
| Panacea [30] | 13.72 | 27.73 | 18.23 | 52.37 | 17.21 | 6.32 | 0.58 | 1.14 | 1.97 |
| DriveArena [37] | 16.06 | 30.03 | 26.14 | 59.37 | 20.79 | 8.92 | 0.56 | 1.10 | 1.89 |
| Ours (Image) | 15.82 | 32.87 | 26.37 | 57.66 | 21.59 | 9.18 | 0.53 | 1.06 | 1.84 |
| Ours (Video) | 25.59 | 40.93 | 28.76 | 63.76 | 23.69 | 12.04 | 0.56 | 1.10 | 1.83 |
8pt
As shown in Fig. 5, Point as Skeleton can synthesize long-horizon interactive multi-view camera sequences. It offers a simple way to handle unobserved actor parts during temporal extrapolation and allows new actions to be inserted at each rollout step. We further evaluate long-horizon generation on nuScenes in Fig. 6. We roll out 100 consecutive frames at 6 Hz, covering nearly the full nuScenes validation sequence length of 17 seconds. Compared with prior methods [16], [34], [37], our design is better suited to simulation-oriented rollouts. It also remains more stable than autoregressive generation without point-skeleton conditioning, especially when the ego vehicle moves beyond initially observed regions (T40–70) or encounters rare street layouts (T80–90).
Figure 8:
.
Ablation of Conditions. We ablate Point Skeleton conditioning on nuScenes and nuPlan. On nuScenes, we compare box & map, relative depth with denser near-range bins, raw metric depth, color, and metric depth+color for 100-frame AR generation, as shown in Fig. 8. Metric depth provides stronger guidance than relative depth, while color further stabilizes synthesis quality; combining color and metric depth achieves the best result. The FVD drop around frame 90 is caused by tail-flush generation, where the rolling window stops and stays fixed until all frames are fully denoised. On nuPlan, we study short-clip arbitrary-trajectory generation in Tab. [tab:ablation95exp] using image generation model. Here, \(\tilde{P}\) denotes a previous latent perturbed by 50–200 diffusion steps during training. Diffusion forcing improves robustness, and projected color conditioning further improves both FID and IoU. Directly appending metric depth to color brings only marginal gains, as the depth signal can be masked by dominant appearance and history cues. However, with foreground-instance augmentation, color+depth achieves the best overall result, indicating that depth provides complementary geometric guidance when color cues degrade during autoregressive inference.
As shown in Fig. 7, foreground vehicles are more prone to projection artifacts than static backgrounds when the ego pose and viewing direction change. This introduces a training-validation gap: during training, point projections are usually aligned with logged camera views, whereas arbitrary simulation trajectories can expose unseen views of nearby agents. To mitigate this mismatch, we augment foreground instances during training by corrupting 80% of instance assets with either voxelization or simulated camera projections, and mixing them with clean assets. Since the sparse point clouds in nuScenes naturally reduce this gap, we evaluate this augmentation mainly on nuPlan. The last block of Tab. [tab:ablation95exp] and Fig. 7 show that instance augmentation improves both quantitative performance and visual consistency.
We present Point as Skeleton, a point-cloud-conditioned generative simulator for closed-loop-compatible autonomous driving rendering. Reset-and-Roll separates committed simulator states from transient forecasts, allowing rolling denoising to use lookahead guidance without carrying future-conditioned latents across simulation steps. The point-cloud skeleton further decomposes logged LiDAR observations into static background and track-indexed foreground assets, which are projected as color and template-depth conditions to anchor autoregressive rollout. Experiments on nuScenes and nuPlan show improved visual realism and geometric alignment, demonstrating the potential of integrating autoregressive generative models into closed-loop driving rollout.
All our videos are generated in a frame-by-frame autoregressive manner. Before generating each frame, the motion trajectory can be specified by the user, which enables interactive generation. The nuScenes videos are generated at 6 Hz, while the nuPlan videos are generated at 5 Hz. The details explanation of the videos are shown in the Fig. 11 and Fig. 12. Note that the stored video loses some degree of visual clarity.
Figure 9:
.
Figure 10:
.
Without future layout cues, the next-frame conditional is a marginal over future layouts: \[p(x_{t+1}\mid h_t)=\int p(x_{t+1}\mid h_t,f_t)\,p(f_t\mid h_t)\,df_t . \label{eq:app95causal95marginal}\tag{1}\]
Let \(z_s\) denote the diffusion latent at noise level \(s\). Under the standard forward perturbation, we have \[\tag{2} \begin{equation} z_s=\sqrt{\bar\alpha_s}\,x_{t+1}+\sqrt{1-\bar\alpha_s}\,\epsilon, \qquad \epsilon\sim\mathcal{N}(0,I) . \tag{3} \end{equation} \begin{equation} p(z_s\mid h_t)=\int p(z_s\mid h_t,f_t)\,p(f_t\mid h_t)\,df_t , \tag{4} \end{equation}\] where \(p(z_s\mid h_t,f_t)\) is the distribution induced by forward diffusion from \(p(x_{t+1}\mid h_t,f_t)\).
In conditional diffusion score matching, the optimal denoising direction is the conditional score \(s^{*}(z_s;h_t):=\nabla_{z_s}\log p(z_s\mid h_t)\), which for the above mixture satisfies \[\nabla_{z_s}\log p(z_s\mid h_t) = \mathbb{E}_{f_t\sim p(f_t\mid z_s,h_t)} \big[\nabla_{z_s}\log p(z_s\mid h_t,f_t)\big] . \label{eq:app95score95average}\tag{5}\] Eq. 5 shows that, without future layout cues, the optimal denoising direction becomes a posterior-weighted average over future-conditioned modes. When \(p(f_t\mid z_s,h_t)\) is multi-modal (e.g., at low-speed or stop-and-go where multiple motion hypotheses are plausible), mode-wise denoising directions may partially cancel, yielding conservative updates that favor persistence of current instances and leading to static agents in rollouts. Conversely, when future motion is near-deterministic, \(p(f_t\mid z_s,h_t)\) concentrates and the averaging effect diminishes.
Autoregressive Generator. For the nuPlan-mini dataset, we use a fixed prompt template with a variable weather description: “weather. A driving scene in city”, where the weather term is instantiated for each sample. For nuScenes, we adopt the text annotations from OpenDWM [33]. The model is trained using the AdamW optimizer with a learning rate of 6e-5. For the video setting, we use an autoregressive window of 2 seconds, corresponding to 12 frames at 6 Hz, with 11 initial reference frames. For the image setting, only a 2-frame window is used. Note that the FVD for long-video generation is computed from saved MP4 files, which may introduce a slight loss in visual quality due to compression.
Point Cloud Skeleton. We apply voxel downsampling with a voxel size of 0.01m to the stacked city background point clouds, while the tracklet actors remain unchanged. For depth-map synthesis, we mainly use vehicle-size prototype templates as foreground depth hints, while during nuPlan-SimGenEval we use tracklet actor first if the points number is higher than 20000. During the simulation, we only consider point clouds within a 100m radius of the ego vehicle. We implement CUDA-based rasterization to accelerate point projection in nuPlan-SimGenEval.
Interactive Divergence Track Construction. For nuPlan-mini, we uniformly sample data instances such that each video sequence of approximately 400 s can be divided into about 30 clips. We first pretrain the layout-based autoregressive model on the text-annotated nuPlan-mini set using all available splits, and then finetune it on 12 subsets with point-skeleton. Finally, we select 10 subsets for customized-trajectory evaluation. In the trajectory-customization test, we generate camera data corresponding to a right-lane-change maneuver of the ego vehicle. The point mask for segment GT applies Gaussian blur with a kernel size of 3 to make the projected point cloud denser in the image plane.
For the IoU evaluation, we assign a category label to each decoupled actor point when constructing the point-cloud skeleton. During rasterized point-to-image projection, these point-wise labels serve as point-template pseudo labels for evaluating alignment with simulator-provided actor geometry. For evaluation, we use an official Mask2Former checkpoint trained on the Cityscapes dataset and report segmentation scores on the Vehicle category. The public Mask2Former checkpoint is fixed across all methods, so the metric is used as a consistent geometry-alignment proxy rather than image-grounded semantic ground truth. The evaluation subsets used for customized-trajectory nuPlan-SimGenEval are:
2021.05.12.22.28.35_veh-35_00620_01164
2021.05.12.22.00.38_veh-35_01008_01518
2021.05.12.23.36.44_veh-35_02035_02387
2021.05.12.23.36.44_veh-35_01133_01535
2021.05.12.23.36.44_veh-35_00152_00504
2021.05.25.14.16.10_veh-35_01690_02183
2021.06.09.12.39.51_veh-26_01943_02303
2021.06.09.14.58.55_veh-35_01894_02311
2021.06.09.17.23.18_veh-38_00773_01140
2021.06.14.16.32.09_veh-35_05038_05402