Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

Songbur Wong\(^{1}\), Xiaosong Jia\(^{3}\), Junqi You\(^{1}\), Bo Zhang\(^{4}\), Pei Xu\(^{2}\)
Renqiu Xia\(^{1}\), Yuping Qiu\(^{2}\), Shaofeng Zhang\(^{5}\), Zelin Zhao\(^{6}\), Xuechao Yan\(^{2}\)
Yuchen Zhou\(^{3}\), Yurui Chen\(^{2}\), Wen Guo\(^{2}\), Hang Xu\(^{2}\), Junchi Yan\(^{1}\)
\(^{1}\)Shanghai Jiao Tong University \(^{2}\)Yinwang Intelligent Technology Co., Ltd.
\(^{3}\)Fudan University \(^{4}\)Shanghai Artificial Intelligence Laboratory
\(^{5}\)University of Science and Technology of China \(^{6}\)Georgia Institute of Technology
 Corresponding authors


Abstract

Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present Point as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that Point as Skeleton improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.

1 Introduction↩︎

End-to-end autonomous driving (E2E-AD) [1][5] has attracted increasing attention in recent years. A central challenge in evaluating E2E-AD systems is to provide sensor observations that respond to the actions selected by the driving policy. Open-loop benchmarks such as nuScenes [6] provide realistic sensor data, but the observations are fixed by the recorded trajectory and cannot react to policy-induced actions. In contrast, closed-loop simulators such as CARLA [7][9] provide interactive feedback, but their rendered sensor appearance can still differ from real-world camera data. This motivates generative sensor simulation, where a learned generative model synthesizes camera observations from simulator-provided ego states, actor states, and scene conditions.

However, closed-loop simulation goes beyond standard driving-scene reconstruction [10] and generation [11], [12]. The ego vehicle may deviate from the logged trajectory, while the planner and traffic simulator update online. The generator therefore needs to render multi-view observations under off-log motions and repeatedly refreshed conditions. This introduces two additional requirements:

Figure 1: Point as Skeleton Generative Simulator. We utilize offline logs to build a foreground-background decoupled Point Cloud Skeleton. At each time step, the evaluated E2E-AD model predicts an action, and the traffic simulator updates the ego state and surrounding actor states accordingly. Based on the updated actor information, the Point Cloud Skeleton is edited and projected back to the camera views as multiple conditions. Finally, the generator synthesizes the latest sensor observations, which are fed back to the E2E-AD model for the next-step decision.

(I) Spatiotemporal Extrapolation. Closed-loop policies may deviate from the logged motion in speed or heading, exposing vehicles and pedestrians that are not observed in the recorded images. This is challenging for classical reconstruction pipelines [13]. Recent methods [14], [15] use generative models as reconstruction post-processing to improve visual plausibility, but they require scene- or frame-specific processing, reducing their ease of deployment.

(II) Instantaneous Interactivity. Closed-loop simulation updates the planner action at every step, requiring the simulator to synthesize the next observation from the latest scene state. Reconstruction-based renderers can be repeatedly queried once a scene is built. However, reconstruction-generation hybrids [16], [17] and urban-scene generators [18], [19] usually assume pre-defined conditions over a full clip. This calls for frame-wise autoregressive generation with step-wise condition updates.

Existing studies [20], [21] show that directly applying full-clip video generators to frame-wise autoregressive rollout causes severe error accumulation (Fig. 3). We therefore propose Point as Skeleton, which uses point-cloud assets accumulated once offline as stabilizing conditions for autoregressive generation. The resulting simulator uses an autoregressive video generator as the visual backbone, while the point skeleton provides editable geometric cues to reduce error accumulation. To realize this framework, we make the following designs.

Reset-and-Roll autoregressive generation. We observe that Rolling Diffusion [22] inference is better suited to highly dynamic driving scenes, but its input-output timestamps are not aligned with the simulator interface. A simulator step expects the generator to consume the current scene condition and immediately return the next observation, whereas rolling diffusion emits the earliest observation in its rolling window after using lookahead conditions. Reset-and-Roll bridges this gap by allowing rolling-style denoising while resetting the committed latent state before future-conditioned information is passed to the next simulation step.

Point-skeleton conditions. We design two point-cloud conditions to stabilize autoregressive generation. The painted point cloud [23] provides pixel-level appearance guidance and suppresses visual degradation during long rollout. However, painted points may suffer from reconstruction inconsistency when actor instances are only partially scanned, motivating a parallel depth condition as a stable geometric anchor. The template-based depth branch replaces foreground actors with category-level point templates, providing stable depth guidance for autoregressive rollout.

nuPlan SimGen. We implement a closed-loop generative interface on nuPlan [24]. The interface attaches our visual generator to the nuPlan traffic simulation core, allowing the ego vehicle to change speed and heading according to user-defined commands or nuPlan planners. It enables evaluation of autoregressive generation under trajectories that deviate from the original log and provides projected point-based semantic labels for perception-oriented evaluation.

2 Related Works↩︎

2.1 Driving World Models↩︎

Conditional diffusion models have shown strong controllability in visual generation [25], [26]. As conditioning signals extend from text and semantic maps to 3D boxes, HD maps, ego motion, and camera geometry, these models have become increasingly applicable to driving-scene generation [27][30]. A line of work studies driving video generation as predictive world modeling [11], [12], [18], [19], [31][33]. These methods synthesize future scene observations under given actions or scene-level conditions, and introduce driving-specific designs such as structured trajectory tokens [12], action-controllable future prediction [31], and depth or occupancy conditions [18], [19]. Their main focus is to generate plausible future clips under a specified condition sequence.

More recent works further explore autoregressive driving world models. Epona [34] formulates driving world modeling as next-token prediction and jointly studies generation and planning. OmniNWM [35] extends this direction toward panoramic navigation world modeling with state, action, and reward awareness. These methods indicate a shift from fixed-length video synthesis toward longer-horizon and more interactive world modeling. However, their design objective still differs from closed-loop simulation. They are generally designed to predict future pixels and actor motions from historical observations and action-related inputs, rather than to render observations under externally updated scene layouts. As a result, fine-grained control over individual participants is less direct than in explicit layout-conditioned simulation, where 3D boxes, HD maps, and simulator states can be updated at every step.

2.2 Driving Scene Simulation↩︎

Driving scene simulation aims to provide sensor observations for interactive evaluation of autonomous-driving policies. Classical simulators such as CARLA [7] support closed-loop interaction, but their rendered appearance often differs from real-world camera data. Reconstruction-based methods improve visual fidelity by reconstructing urban scenes from recorded logs [10], [36], yet they are most reliable near the logged trajectory. When the ego vehicle deviates from the recorded motion, unseen regions and weakly observed actors become difficult to render faithfully.

Recent methods introduce generative priors to alleviate this limitation. FreeVS [16] improves free-viewpoint synthesis under shifted driving trajectories, ReCamDriving [14] conditions generation on reconstructed scenes, and R3D2 [15] refines projected dynamic instances with generative models. These reconstruction-generation hybrids improve visual plausibility beyond logged views, but they still rely on scene-specific reconstruction, object-level editing, or post-processing modules. Therefore, extending them to continuous closed-loop rollout with online condition updates is not straightforward.

Generative simulation systems move one step further by using learned world models as simulator engines. DriveArena [37] builds a closed-loop generative simulation platform, where generated observations are consumed by a driving agent, the resulting trajectory updates the scene layout, and the updated layout is fed back to the generator. Closed-source systems such as Waymo World Model [38] and GAIA-3 [39] further demonstrate controllable autonomy evaluation with action-, layout-, and scenario-conditioned generation. These systems suggest that driving simulation is moving from graphics- or reconstruction-centered pipelines toward generation-centered closed-loop simulators. Nevertheless, open autoregressive video-generation designs that can be readily integrated with closed-loop driving systems remain limited.

3 Method↩︎

Overview. We build a frame-wise generative rendering interface for closed-loop-compatible autonomous driving simulation: each updated simulator state defines the next scene layout, and the generator synthesizes the corresponding camera observation. Our method includes (i) an autoregressive video diffusion model with diffusion-forcing training and Reset-and-Roll inference for stable rollouts (Sec. 3.1), (ii) a Point Cloud Skeleton projected as color and depth conditions for view-consistent geometry (Sec. 3.2), and (iii) nuPlan-SimGen, an interactive generative rendering interface and evaluation protocol for ego-deviated trajectories (Sec. 3.3).

3.1 Autoregressive Framework↩︎

Figure 2: Reset-and-Roll Framework. Unlike full-sequence denoising [20], [34], [37], Reset-and-Roll supports past-to-future conditioned denoising, while preventing the non-causal guidance from accumulating errors across simulation steps. In our implementation, a world prediction model [33] trained with standard diffusion forcing [40] is sufficient for Reset-and-Roll inference.

In closed-loop simulation, the generator acts as a step-wise visual renderer. At simulation step \(t\), the planner outputs a short-horizon trajectory as discrete ego waypoints. The simulator commits only the next executable waypoint as the current ego state, while the remaining waypoints are treated as short-horizon forecasts. We therefore distinguish between committed states and forecast states. Committed states include the generated history, the current ego state, the current actor states, and the simulator layout \(\mathcal{L}^{\mathrm{sim}}_t\) at step \(t\). Forecast states include future ego waypoints and their corresponding future layouts \(\hat{\mathcal{L}}_{t+1:t+K}\), which may be revised after the policy receives the newly generated observation.

As shown in the left part of Fig. 1, the traffic simulator places the committed ego and actor states on the map to construct the current layout \(\mathcal{L}^{\mathrm{sim}}_t\). Future layouts are constructed in the same format, but they remain mutable forecasts rather than simulator states to be inherited. The video generator is therefore invoked at every simulation step, conditioned on the generated history, the current committed layout, and the forecast layouts within the generation window.

This setting casts generative simulation as a frame-wise autoregressive video generation problem: each synthesized observation is returned to the policy and affects the next simulator query. Without dedicated training or rollout design, autoregressive diffusion suffer from severe error accumulation [20], [21], making standard autonomous-driving video generators difficult to use as interactive renderers, as shown in Fig. 3-(2). Existing works alleviate this issue by noising frames during training and inference [20], [35], or by simulating inference-time histories during training [34]. However, when deployed as closed-loop renderers, full-sequence inference still suffers from two failure modes.

Causal full-sequence. As shown in Fig. 2, if the generator avoids future layouts, the next-frame distribution is marginalized over possible future motions: \[p(x_{t+1}\mid h_t)= \int p(x_{t+1}\mid h_t,f_t)p(f_t\mid h_t)\,df_t ,\] where \(h_t\) denotes the committed history and current simulator layout, and \(f_t\) denotes future layout trajectories. This removes explicit motion cues during denoising; thus low-speed or occluded actors are often generated conservatively, causing weak or static motion in rollout (Fig. 3-(3),(4)).

Non-causal full-sequence. Lookahead layouts improve dynamics, but full-sequence non-causal inference entangles the current output with unrealized forecasts. In closed-loop rollout, the lookahead used at step \(t\) may change after the policy receives the generated observation. Once observations or latent states produced under the old forecast are reused in later queries, future-conditioned information becomes part of the committed rollout state, causing state mismatch and accumulated errors.

Figure 3: Qualitative Observation. Results are generated frame-wise autoregressively on the first nuScenes-val sequence. Note that sequence (3) and (5) share the same checkpoint from diffusion forcing training, and sequence (2) and (4) used official model. Both sequences (3) and (4) use full-sequence causal AR and show vehicle stagnation. Sequence (5) shows static background moves with the ego vehicle, suggesting weak geometric consistency in frame-wise autoregressive generation.

We therefore adopt rolling-window diffusion, which we find better suited to step-wise high-dynamic rendering. Instead of denoising a full future clip at once, rolling diffusion maintains a FIFO latent window: each roll injects a new noisy latent, denoises the window, and emits one frame. This rolling form naturally matches the simulator’s step-wise query pattern and avoids committing an entire future-conditioned sequence. It also allows predicted layouts to guide near-future dynamics, which improves instance motion compared with causal full-sequence inference, as shown in Fig. 3-(5). However, naive rolling still exposes the inherited latent state to predicted layouts, so future-conditioned information can be carried into the next simulator query.

Reset-and-Roll aligns rolling inference with the closed-loop simulator interface as shown in Fig. 2. At each step, the window contains \((L-1)\) history frames and one current-step latent. After the first denoising update, before injecting any lookahead layout, we cache the current latent and timestep. This cached latent has only seen the committed history and the current simulator layout \(\mathcal{L}^{\mathrm{sim}}_t\). We then continue rolling denoising with predicted layouts \(\hat{\mathcal{L}}_{t+1:t+K}\) to render the current observation with stronger dynamic cues. Before moving to the next simulator step, all lookahead-exposed latents are discarded. The next query is formed with the newly realized layout and the cached latent at the same diffusion scheduler timestep. Thus, predicted layouts are used for temporary denoising in the current query, but future-conditioned latent states are not committed across steps.

3.2 Point Cloud Skeleton↩︎

To address the error accumulation observed in AR generation (Fig. 3-(5)), we introduce the Point Cloud Skeleton as a training-inference-consistent scene representation. It accumulates offline LiDAR observations into static background and track-indexed foreground assets, which are recomposed by the simulator and projected to camera views at each rollout step. The projected skeleton provides sparse but state-aligned current-view evidence, turning each step into conditional rendering rather than pure history-based extrapolation. Thus, the primary role of the skeleton is to provide a consistent state representation during rollout, rather than to reconstruct the scene at the pixel level.

As illustrated in Fig. 4, we colorize each LiDAR frame by projecting points onto calibrated images, obtaining \(P_t^c\). Using 3D boxes and track IDs, we split it into background and foreground points: \[P_t^c = P_t^{\mathrm{bg}} \cup P_t^{\mathrm{fg}}, \quad P_t^{\mathrm{fg}}=\bigcup_{i\in\mathcal{A}_t} P_{t,i}^{\mathrm{fg}}, \quad P_t^{\mathrm{bg}}\cap P_t^{\mathrm{fg}}=\emptyset ,\] where \(\mathcal{A}_t\) denotes the actors observed at frame \(t\). We accumulate background points in the global frame, and normalize each actor’s foreground points into canonical coordinates: \[P_{\mathrm{global}}^{\mathrm{bg}} = \bigcup_{t=1}^{N} T_{\mathrm{global}}(t) P_t^{\mathrm{bg}}, \quad P_{\mathrm{actor}(i)}^{\mathrm{fg}} = \bigcup_{t:i\in\mathcal{A}_t} B_{t,i}^{-1} P_{t,i}^{\mathrm{fg}} .\] Here \(T_{\mathrm{global}}(t)\) maps frame-\(t\) points to the global frame, and \(B_{t,i}\) denotes the 3D box pose of actor \(i\). Thus, \(B_{t,i}^{-1}\) recenters and aligns foreground points into canonical coordinates, forming a track-indexed actor asset. During simulation, the simulator places each actor according to box pose, merges it with \(P_{\mathrm{global}}^{\mathrm{bg}}\), and projects the composed skeleton to each camera view as the color map condition.

Figure 4: Construction and Utilization of Point Cloud Skeleton. We separately accumulate foreground and background skeleton using point cloud and images from recorded logs. During simulation, actor assets are recomposed with the background according to simulator states and projected to camera views. The skeleton provides two conditioning inputs: a color map for appearance anchors and a template-based depth map for view-consistent geometry of moving agents.

The projected skeleton is used through two complementary branches. The color branch preserves pixel-level appearance from accumulated colorized points, providing degraded but useful appearance anchors for autoregressive rendering. However, foreground actors are often only partially scanned in the logs. Under trajectory drift, their color projections may become sparse or view-inconsistent, creating a training-inference gap that is amplified during rollout. We therefore introduce a depth branch to provide more stable geometry. We cluster training-log 3D boxes by category, and aggregate LiDAR points within each cluster to build category-level point templates for common traffic participants, such as sedan, SUV, and pedestrian. During simulation, these templates are placed according to simulator-provided actor boxes and projected as depth maps. The color map supplies appearance anchors, while the template-depth map supplies geometry. We concatenate both maps with the box&hdmap layout as conditions for the driving world model.

3.3 nuPlan-SimGen↩︎

To connect generative models with practical closed-loop simulation, we implement nuPlan-SimGen, a nuPlan [24] plugin for closed-loop generative rendering. The plugin attaches the visual generator to the nuPlan traffic simulator: after each planner action, the simulator updates ego and actor states, and the generator renders the corresponding camera observations. Enabled by our template-based depth map, nuPlan-SimGen also supports a simulator-state alignment proxy under deviated ego trajectories. We project movable-object point templates to obtain vehicle pseudo masks and compare them with vehicle masks predicted by an off-the-shelf segmentor, reporting projected-mask IoU (Fig. 5).

Figure 5: nuPlan-SimGen. Starting from GT reference frames, nuPlan-SimGen evaluates generative rendering when a predefined lane-change trajectory takes over the ego motion. Projected point templates provide vehicle pseudo masks for projected-mask IoU under novel trajectories where no image ground truth is available.

As shown in Fig. 5, nuPlan-SimGen evaluates generative rendering under ego-trajectory drift. During the first \(N\) steps, the ego vehicle and surrounding actors follow the logged trajectories, where \(N\) is the number of reference frames required by the generator. Starting from frame \(N\), a predefined lane-change trajectory takes over the ego motion. This trajectory gradually adjusts the ego speed toward a target value of 10m/s and applies a yaw deviation from the recorded path. At each simulator step, the next executable waypoint is committed as the current layout, while the remaining waypoints are treated as future layouts. Surrounding actors can either continue along their logged tracks or be controlled by nuPlan’s IDM policy; for consistency, our evaluation keeps them on the logged tracks. Although their motions are unchanged, the deviated ego pose alters relative viewpoints and visibility, which may bring actors outside the logged camera view into the generated field of view.

Table 1: Comparison on nuScenes Dataset.
Method Backbone Condition Resolution \(\mathrm{FID}_{(\downarrow)}\) \(\mathrm{FVD}_{(\downarrow)}\)
Panacea+ [41] SD1.5 Layout\(+\)Text 256\(\times\)512 15.50 103.00
MagicDrive [28] SD1.5 Layout\(+\)Text 224\(\times\)400 16.20 217.94
DriveArena [37] SD1.5 Layout\(+\)Text\(+\)Prev 224\(\times\)400 16.03 185.32
MagicDiT [29] Sora-DiT Layout\(+\)Text\(+\)Video 224\(\times\)400 20.91 97.21
Epona [34] Flux-DiT Ego Traj\(+\)Text\(+\)Video 512\(\times\)1024 7.50 82.80
Ours (Image) SD1.5 Layout\(+\)Point\(+\)Prev 224\(\times\)400 7.09 69.47
Ours (Video) SD3.5 Layout\(+\)Point\(+\)Video 256\(\times\)448 5.97 58.3

16pt

4 Experiments↩︎

4.1 Datasets and Implementation Details↩︎

We conduct experiments on nuScenes and nuPlan-mini. On nuScenes, we use 700 training and 150 validation scenes, reporting FID, FVD, and UniAD scores [1], [37]. On nuPlan-mini, we use all 64 sequences for model training and select 300 customized-trajectory clips (specific in appendix) for nuPlan-SimGen evaluation. This benchmark evaluates trajectory deviation within logged scenes, rather than scene-level generalization to unseen logs or maps. We report FID and Mask2Former vehicle segmentation scores [42].

We implement two variants of Point as Skeleton: an SDv1.5-based single-frame generator [43] for comparison with image generators  [20], [37], and an SDv3.5-based autoregressive video generator for comparison with video generators [34], [44]. The SDv1.5 variant is trained for 200 epochs on nuScenes and 150 epochs on nuPlan, with batch sizes of 16 and 8, respectively. The SDv3.5 variant is trained for 15k iterations with batch size 16 on both datasets. All models use diffusion forcing [40]. We generate 2 Hz images and 6 Hz videos on nuScenes using ASAP labels [45], and 5 Hz videos on nuPlan. More implementation details are provided in the appendix.

4.2 Main Results↩︎

4.2.1 Quantitative Analysis↩︎

We first evaluate Point as Skeleton on nuScenes. In video generation, the model conditions on 2s history and generates a 4s clip at 6 Hz, yielding 24 autoregressive steps. The image variant is evaluated for 16 autoregressive steps at 2 Hz. Table 1 shows the best FID and FVD under this evaluation protocol, indicating improved visual quality and temporal consistency. We further evaluate generated sequences with UniAD. Table 2 shows stronger downstream detection and segmentation, with competitive planning performance. These results suggest that Point as Skeleton better preserves perception-relevant information while maintaining competitive planning performance.

Table 2: UniAD’s [1] Different Tasks in nuScenes Generation.
Source Data 3DOD \(\uparrow\) Segmentation \(\uparrow\) \(\text{L2}\) \(\downarrow\)
mAP NDS Lanes Drivable Divider Crossing 1.0s 2.0s 3.0s
ori nuScenes [6] 37.98 49.85 31.31 69.14 25.93 14.36 0.51 0.98 1.65
MagicDrive [28] 12.92 28.36 21.95 51.46 17.10 6.57 0.57 1.14 1.95
Panacea [30] 13.72 27.73 18.23 52.37 17.21 6.32 0.58 1.14 1.97
DriveArena [37] 16.06 30.03 26.14 59.37 20.79 8.92 0.56 1.10 1.89
Ours (Image) 15.82 32.87 26.37 57.66 21.59 9.18 0.53 1.06 1.84
Ours (Video) 25.59 40.93 28.76 63.76 23.69 12.04 0.56 1.10 1.83

8pt

Figure 6: Visualization of Long Horizon Rollout. FreeVS [16] is reproduced in a non-causal full-sequence setting. Our method is better suited to simulation scenarios than the compared methods.

4.2.2 Qualitative Analysis.↩︎

As shown in Fig. 5, Point as Skeleton can synthesize long-horizon interactive multi-view camera sequences. It offers a simple way to handle unobserved actor parts during temporal extrapolation and allows new actions to be inserted at each rollout step. We further evaluate long-horizon generation on nuScenes in Fig. 6. We roll out 100 consecutive frames at 6 Hz, covering nearly the full nuScenes validation sequence length of 17 seconds. Compared with prior methods [16], [34], [37], our design is better suited to simulation-oriented rollouts. It also remains more stable than autoregressive generation without point-skeleton conditioning, especially when the ego vehicle moves beyond initially observed regions (T40–70) or encounters rare street layouts (T80–90).

Figure 7: 3D Instance Augmentation Pipeline.

Figure 8: image.

4.3 Ablation Study↩︎

Ablation of Conditions. We ablate Point Skeleton conditioning on nuScenes and nuPlan. On nuScenes, we compare box & map, relative depth with denser near-range bins, raw metric depth, color, and metric depth+color for 100-frame AR generation, as shown in Fig. 8. Metric depth provides stronger guidance than relative depth, while color further stabilizes synthesis quality; combining color and metric depth achieves the best result. The FVD drop around frame 90 is caused by tail-flush generation, where the rolling window stops and stays fixed until all frames are fully denoised. On nuPlan, we study short-clip arbitrary-trajectory generation in Tab. [tab:ablation95exp] using image generation model. Here, \(\tilde{P}\) denotes a previous latent perturbed by 50–200 diffusion steps during training. Diffusion forcing improves robustness, and projected color conditioning further improves both FID and IoU. Directly appending metric depth to color brings only marginal gains, as the depth signal can be masked by dominant appearance and history cues. However, with foreground-instance augmentation, color+depth achieves the best overall result, indicating that depth provides complementary geometric guidance when color cues degrade during autoregressive inference.

4.3.0.1 Instance augmentation.

As shown in Fig. 7, foreground vehicles are more prone to projection artifacts than static backgrounds when the ego pose and viewing direction change. This introduces a training-validation gap: during training, point projections are usually aligned with logged camera views, whereas arbitrary simulation trajectories can expose unseen views of nearby agents. To mitigate this mismatch, we augment foreground instances during training by corrupting 80% of instance assets with either voxelization or simulated camera projections, and mixing them with clean assets. Since the sparse point clouds in nuScenes naturally reduce this gap, we evaluate this augmentation mainly on nuPlan. The last block of Tab. [tab:ablation95exp] and Fig. 7 show that instance augmentation improves both quantitative performance and visual consistency.

5 Conclusion↩︎

We present Point as Skeleton, a point-cloud-conditioned generative simulator for closed-loop-compatible autonomous driving rendering. Reset-and-Roll separates committed simulator states from transient forecasts, allowing rolling denoising to use lookahead guidance without carrying future-conditioned latents across simulation steps. The point-cloud skeleton further decomposes logged LiDAR observations into static background and track-indexed foreground assets, which are projected as color and template-depth conditions to anchor autoregressive rollout. Experiments on nuScenes and nuPlan show improved visual realism and geometric alignment, demonstrating the potential of integrating autoregressive generative models into closed-loop driving rollout.

6 Visualization↩︎

All our videos are generated in a frame-by-frame autoregressive manner. Before generating each frame, the motion trajectory can be specified by the user, which enables interactive generation. The nuScenes videos are generated at 6 Hz, while the nuPlan videos are generated at 5 Hz. The details explanation of the videos are shown in the Fig. 11 and Fig. 12. Note that the stored video loses some degree of visual clarity.

7 Discussion↩︎

Figure 9: image.

Figure 10: image.

Figure 11: Long Video Generation. A 17-second generation produces a 100-frame video, including 89 autoregressive steps after excluding the 11 initial frames. This long video demonstrates that the point-as-skeleton representation supports a generation length sufficient to complete a full sequence of actions.
Figure 12: nuPlan Plugin. Starting from step 16, we perform a rollout every 0.2 seconds for trajectory control and generate one frame at each rollout. This video demonstrates that our method can be seamlessly integrated with modern driving planning systems.

8 Derivations↩︎

8.0.0.1 Causal full-sequence tends to static instances.

Without future layout cues, the next-frame conditional is a marginal over future layouts: \[p(x_{t+1}\mid h_t)=\int p(x_{t+1}\mid h_t,f_t)\,p(f_t\mid h_t)\,df_t . \label{eq:app95causal95marginal}\tag{1}\]

Let \(z_s\) denote the diffusion latent at noise level \(s\). Under the standard forward perturbation, we have \[\tag{2} \begin{equation} z_s=\sqrt{\bar\alpha_s}\,x_{t+1}+\sqrt{1-\bar\alpha_s}\,\epsilon, \qquad \epsilon\sim\mathcal{N}(0,I) . \tag{3} \end{equation} \begin{equation} p(z_s\mid h_t)=\int p(z_s\mid h_t,f_t)\,p(f_t\mid h_t)\,df_t , \tag{4} \end{equation}\] where \(p(z_s\mid h_t,f_t)\) is the distribution induced by forward diffusion from \(p(x_{t+1}\mid h_t,f_t)\).

In conditional diffusion score matching, the optimal denoising direction is the conditional score \(s^{*}(z_s;h_t):=\nabla_{z_s}\log p(z_s\mid h_t)\), which for the above mixture satisfies \[\nabla_{z_s}\log p(z_s\mid h_t) = \mathbb{E}_{f_t\sim p(f_t\mid z_s,h_t)} \big[\nabla_{z_s}\log p(z_s\mid h_t,f_t)\big] . \label{eq:app95score95average}\tag{5}\] Eq. 5 shows that, without future layout cues, the optimal denoising direction becomes a posterior-weighted average over future-conditioned modes. When \(p(f_t\mid z_s,h_t)\) is multi-modal (e.g., at low-speed or stop-and-go where multiple motion hypotheses are plausible), mode-wise denoising directions may partially cancel, yielding conservative updates that favor persistence of current instances and leading to static agents in rollouts. Conversely, when future motion is near-deterministic, \(p(f_t\mid z_s,h_t)\) concentrates and the averaging effect diminishes.

9 Implementation Details↩︎

Autoregressive Generator. For the nuPlan-mini dataset, we use a fixed prompt template with a variable weather description: “weather. A driving scene in city”, where the weather term is instantiated for each sample. For nuScenes, we adopt the text annotations from OpenDWM [33]. The model is trained using the AdamW optimizer with a learning rate of 6e-5. For the video setting, we use an autoregressive window of 2 seconds, corresponding to 12 frames at 6 Hz, with 11 initial reference frames. For the image setting, only a 2-frame window is used. Note that the FVD for long-video generation is computed from saved MP4 files, which may introduce a slight loss in visual quality due to compression.

Point Cloud Skeleton. We apply voxel downsampling with a voxel size of 0.01m to the stacked city background point clouds, while the tracklet actors remain unchanged. For depth-map synthesis, we mainly use vehicle-size prototype templates as foreground depth hints, while during nuPlan-SimGenEval we use tracklet actor first if the points number is higher than 20000. During the simulation, we only consider point clouds within a 100m radius of the ego vehicle. We implement CUDA-based rasterization to accelerate point projection in nuPlan-SimGenEval.

Interactive Divergence Track Construction. For nuPlan-mini, we uniformly sample data instances such that each video sequence of approximately 400 s can be divided into about 30 clips. We first pretrain the layout-based autoregressive model on the text-annotated nuPlan-mini set using all available splits, and then finetune it on 12 subsets with point-skeleton. Finally, we select 10 subsets for customized-trajectory evaluation. In the trajectory-customization test, we generate camera data corresponding to a right-lane-change maneuver of the ego vehicle. The point mask for segment GT applies Gaussian blur with a kernel size of 3 to make the projected point cloud denser in the image plane.

For the IoU evaluation, we assign a category label to each decoupled actor point when constructing the point-cloud skeleton. During rasterized point-to-image projection, these point-wise labels serve as point-template pseudo labels for evaluating alignment with simulator-provided actor geometry. For evaluation, we use an official Mask2Former checkpoint trained on the Cityscapes dataset and report segmentation scores on the Vehicle category. The public Mask2Former checkpoint is fixed across all methods, so the metric is used as a consistent geometry-alignment proxy rather than image-grounded semantic ground truth. The evaluation subsets used for customized-trajectory nuPlan-SimGenEval are:

  • 2021.05.12.22.28.35_veh-35_00620_01164

  • 2021.05.12.22.00.38_veh-35_01008_01518

  • 2021.05.12.23.36.44_veh-35_02035_02387

  • 2021.05.12.23.36.44_veh-35_01133_01535

  • 2021.05.12.23.36.44_veh-35_00152_00504

  • 2021.05.25.14.16.10_veh-35_01690_02183

  • 2021.06.09.12.39.51_veh-26_01943_02303

  • 2021.06.09.14.58.55_veh-35_01894_02311

  • 2021.06.09.17.23.18_veh-38_00773_01140

  • 2021.06.14.16.32.09_veh-35_05038_05402

References↩︎

[1]
Y. Hu et al., “Planning-oriented autonomous driving,” 2023, pp. 17853–17862.
[2]
Q. Li, X. Jia, S. Wang, and booktitle=ECCV. Yan Junchi, “Think2Drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in CARLA-v2),” 2024 ## Benc.
[3]
Z. Yang et al., “DriveMoE: Mixture-of-experts for vision-language-action model in end-to-end autonomous driving.” 2025 , eprint={2505.16278}, archivePrefix={arXiv}, primaryClass={cs.CV}, [Online]. Available: https://arxiv.org/abs/2505.16278.
[4]
X. Jia, J. You, Z. Zhang, and booktitle=The. T. I. C. on L. R. Junchi Yan, “DriveTransformer: Unified transformer for scalable end-to-end autonomous driving,” 2025, [Online]. Available: https://openreview.net/forum?id=M42KR4W9P5.
[5]
X. Jia, Y. Gao, L. Chen, J. Yan, P. L. Liu, and booktitle=ICCV. Li Hongyang, “DriveAdapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving,” 2023.
[6]
H. Caesar et al., “Nuscenes: A multimodal dataset for autonomous driving,” 2020, pp. 11621–11631.
[7]
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and booktitle=Conference. on robot learning Koltun Vladlen, “CARLA: An open urban driving simulator,” 2017 , organization={PMLR}, pp. 1–16.
[8]
B. Zhang et al., “Resimad: Zero-shot 3d domain transfer for autonomous driving with source reconstruction and target simulation,” arXiv preprint arXiv:2309.05527, 2023.
[9]
X. Jia, Z. Yang, Q. Li, Z. Zhang, and booktitle=NEURIPSDATA. Junchi Yan, “Bench2Drive : Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,” 2024.
[10]
Y. Yan et al., “Street gaussians: Modeling dynamic urban scenes with gaussian splatting,” 2024 , organization={Springer}, pp. 156–173.
[11]
X. Wang, Z. Zhu, G. Huang, X. Chen, J. Zhu, and booktitle=European. C. on C. V. Lu Jiwen, “DriveDreamer: Towards real-world-drive world models for autonomous driving,” 2024 , organization={Springer}, pp. 55–72.
[12]
J. Yang et al., “Generalized predictive model for autonomous driving,” 2024.
[13]
Z. Chen et al., “Omnire: Omni urban scene reconstruction,” arXiv preprint arXiv:2408.16760, 2024.
[14]
Y. Li et al., “ReCamDriving: LiDAR-free camera-controlled novel trajectory video generation,” arXiv preprint arXiv:2512.03621, 2025.
[15]
W. Ljungbergh et al., arXiv preprint arXiv:2506.07826, 2025.
[16]
Q. Wang, L. Fan, Y. Wang, Y. Chen, and Z. Zhang, “Freevs: Generative view synthesis on free driving trajectory,” arXiv preprint arXiv:2410.18079, 2024.
[17]
S. Wang, H. Sun, B. Wang, H. Ye, and X. Yu, “Mirage: One-step video diffusion for photorealistic and coherent asset editing in driving scenes,” arXiv preprint arXiv:2512.24227, 2025.
[18]
T. Yan et al., “DrivingSphere: Building a high-fidelity 4D world for closed-loop simulation,” arXiv preprint arXiv:2411.11252, 2024.
[19]
J. Guo et al., “DiST-4D: Disentangled spatiotemporal diffusion with metric depth for 4D driving scene generation,” arXiv preprint arXiv:2503.15208, 2025.
[20]
J. You, X. Jia, Z. Zhang, Y. Zhu, and J. Yan, “Bench2Drive-r: Turning real world data into reactive closed-loop autonomous driving benchmark by generative model,” arXiv preprint arXiv:2412.09647, 2024 ## LLM4A.
[21]
B. Chen, D. Martı́ Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann, “Diffusion forcing: Next-token prediction meets full-sequence diffusion,” Advances in Neural Information Processing Systems, vol. 37, pp. 24081–24125, 2024.
[22]
D. Ruhe, J. Heek, T. Salimans, and E. Hoogeboom, “Rolling diffusion models.” 2024 , eprint={2402.09470}, archivePrefix={arXiv}, primaryClass={cs.LG}, [Online]. Available: https://arxiv.org/abs/2402.09470.
[23]
S. Vora, A. H. Lang, B. Helou, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Beijbom Oscar, “Pointpainting: Sequential fusion for 3d object detection,” 2020, pp. 4604–4612.
[24]
H. Caesar et al., “Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles,” arXiv preprint arXiv:2106.11810, 2021.
[25]
L. Zhang, A. Rao, and booktitle=Proceedings. of the I. international conference on computer vision Agrawala Maneesh, “Adding conditional control to text-to-image diffusion models,” 2023, pp. 3836–3847.
[26]
Z. Tan, Q. Xue, X. Yang, S. Liu, and X. Wang, “Ominicontrol2: Efficient conditioning for diffusion transformers,” arXiv preprint arXiv:2503.08280, 2025.
[27]
K. Yang, E. Ma, J. Peng, Q. Guo, D. Lin, and K. Yu, “Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout,” arXiv preprint arXiv:2308.01661, 2023.
[28]
R. Gao et al., “Magicdrive: Street view generation with diverse 3d geometry control,” arXiv preprint arXiv:2310.02601, 2023.
[29]
R. Gao, K. Chen, B. Xiao, L. Hong, Z. Li, and Q. Xu, “MagicDriveDiT: High-resolution long video generation for autonomous driving with adaptive control,” arXiv preprint arXiv:2411.13807, 2024.
[30]
Y. Wen et al., “Panacea: Panoramic and controllable video generation for autonomous driving,” 2024, pp. 6902–6912.
[31]
S. Gao et al., “Vista: A generalizable driving world model with high fidelity and versatile controllability,” 2024.
[32]
G. Zhao et al., “Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,” 2025, vol. 39, pp. 10412–10420.
[33]
R. Chen et al., “Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving,” arXiv preprint arXiv:2412.04842, 2024.
[34]
K. Zhang et al., 2025 , eprint={2506.24113}, archivePrefix={arXiv}, primaryClass={cs.CV}, [Online]. Available: https://arxiv.org/abs/2506.24113.
[35]
B. Li et al., “OmniNWM: Omniscient driving navigation world models,” arXiv preprint arXiv:2510.18313, 2025.
[36]
J. Yang et al., “Emernerf: Emergent spatial-temporal scene decomposition via self-supervision,” arXiv preprint arXiv:2311.02077, 2023.
[37]
X. Yang et al., “Drivearena: A closed-loop generative simulation platform for autonomous driving,” arXiv preprint arXiv:2408.00415, 2024.
[38]
Waymo, “The waymo world model: A new frontier for autonomous driving simulation , howpublished = https://waymo.com/blog/2026/02/the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation.” 2026.
[39]
“Gaia-3 , howpublished = https://wayve.ai/thinking/gaia-3/.” 2026.
[40]
K. Song, B. Chen, M. Simchowitz, Y. Du, R. Tedrake, and V. Sitzmann, arXiv preprint arXiv:2502.06764, 2025.
[41]
Y. Wen et al., “Panacea+: Panoramic and controllable video generation for autonomous driving.” 2024 , eprint={2408.07605}, archivePrefix={arXiv}, primaryClass={cs.CV}, [Online]. Available: https://arxiv.org/abs/2408.07605.
[42]
B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Girdhar Rohit, “Masked-attention mask transformer for universal image segmentation,” 2022, pp. 1290–1299.
[43]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Ommer Björn, “High-resolution image synthesis with latent diffusion models,” 2022, pp. 10684–10695.
[44]
J. Mei et al., “Dreamforge: Motion-aware autoregressive video generation for multi-view driving scenes,” arXiv preprint arXiv:2409.04003, 2024.
[45]
X. Wang et al., “Are we ready for vision-centric driving streaming perception? The asap benchmark,” 2023, pp. 9600–9610.