1 Introduction↩︎

The generation of realistic and controllable synthetic environments is a fundamental challenge in computer vision, with critical applications in autonomous driving, embodied AI, and digital twins [1]–[7]. In the context of autonomous driving, the ability to synthesize long-duration, multi-view video sequences is essential for training and validating perception and planning systems. However, generating long-horizon driving scenes remains a significant challenge. Unlike short-range clips, long-term generation requires maintaining strict temporal consistency, physical plausibility, and geometric stability over extended trajectories, where even minor errors can quickly compound into significant visual distortions or structural collapses.

Existing generative approaches for driving scenes can be broadly categorized into two groups, yet both face limitations in long-horizon scenarios. The first category consists of Implicit-Autoregressive methods [8]–[10], which typically take a short video clip or a few frames as input and use video diffusion model to autoregressively predict future frames, as depicted in Fig. 1 (a). While these models can produce high-fidelity appearance and plausible motion, they often rely on implicit latent features without an explicit intermediate geometric representation. Consequently, during long-term prediction, these methods are prone to accumulated errors and “hallucinations”, where the generated content gradually deviates from the underlying 3D structure or loses spatial coherence. The second category focuses on Explicit-Prior-Guided approaches [11]–[18] (, Fig. 1 (b)), requiring a specific BEV map/HD map as input. Although these provide better structural constraints, they often struggle to render long-horizon video due to the limited range of input prior. Furthermore, maintaining alignment between generated geometry and visual appearance remains a difficult open problem.

To address these limitations, we introduce InfiniVerse (see Fig. 1 (c)), a unified framework for long-range, controllable synthesis of dynamic urban environments from a single multi-view observation. Unlike prior works that rely on implicit transitions or rigid HD maps, our approach reconstructs an explicit 3D occupancy representation from the initial frame. This representation serves as a structural foundation for autoregressively extending the scene along arbitrary user-defined trajectories. To ensure high-fidelity results, we employ a video diffusion model to translate the extended occupancy grids into consistent multi-view video sequences.

The core of our method is a bidirectional “sketch-and-refine” loop that establishes a closed-loop feedback between the 3D occupancy and 2D video domains. Specifically, we treat the coarse occupancy as a “sketch” to guide video generation, and then re-project the generated high-fidelity frames back into the 3D space as image-conditioned feedback. By making 2D appearance and 3D geometry mutually constraining, our framework significantly reduces drift and suppresses hallucinations unsupported by the underlying scene structure. This reciprocal mechanism enables the creation of long-horizon, temporally consistent, and geometrically aligned driving scenarios.

In summary, our key contributions are:

• We propose InfiniVerse, a unified pipeline that generates geometrically accurate and semantically coherent long-range 3D scenes from a single frame.

• We leverage a hierarchical sketch-and-refine paradigm that progressively refines the occupancy representation, enhancing scene consistency and controllability over long trajectories.

• Extensive experiments on the Waymo Open Dataset and nuScenes demonstrate that InfiniVerse achieves new state-of-the-art performance in both generation quality and duration.

2 Related work↩︎

2.1 3D Urban Scene Generation↩︎

Recent progress [19] has been driven by learning-based methods that leverage powerful priors from large-scale 3D datasets, enabling diffusion model based 3D reconstructions [20], [21] even under severe ambiguity. Due to lack of high-quality 3D assets, several simulated environments played a vital role in autonomous driving, with platforms like CARLA [22] and Waymo’s Simulation Engine[23] providing handcrafted 3D assets and rule-based control. However, these systems require significant manual effort and lack flexibility for arbitrary trajectory exploration.

More recent methods attempt to automate urban scene generation by leveraging learned priors from real-world datasets. For instance, Infinicube [24] has proposed techniques for large-scale scene synthesis, but they often depend on highly annotated HD maps and 3D layout bounding boxes, which limits scalability and data efficiency. X-Scene [25] explores large-scale driving scene generation guided by textual descriptions, enabling diverse scene composition and condition control through natural language prompts. However, its control mechanism remains coarse-grained and ambiguous, making it difficult to achieve precise spatial or semantic alignment between text instructions and generated content. Some methods [3], [26], [27] have exploited block-by-block generation for semantic scene generation in outdoor environments but lack controllability and fidelity.

2.2 3D Occupancy Representation↩︎

CityDreamer [28] learns NeRF-style radiance fields or signed-distance functions to render novel views, but store geometry only implicitly; they cannot expose occupancy grids required for downstream planning or physical simulation.

Occupancy grids have emerged as a powerful and compact representation for 3D scene understanding, especially in autonomous driving and robotics due to its explicit representation for efficient computation. They provide explicit volumetric reasoning and enable consistent spatial modeling across time and views. Works such as OccNet [29], VoxelNeXt [30], and Occ3D [31] have shown promise in using learned occupancy fields for tasks like object detection and motion prediction. However, these approaches are typically limited to static or short-term settings and do not support long-range generation.

2.3 Controllable Video Generation↩︎

Video generation has recently seen significant progress with the introduction of scalable and general-purpose generative models. SVD (Stable Video Diffusion) [32] introduces a diffusion-based framework that models video as a sequence of latent frames, enabling high-quality and temporally coherent video generation. Its scalable architecture and strong pretraining enable it to generalize well across diverse prompts and temporal lengths. Building on this, CogVideoX [33] proposes a diffusion transformer video generation framework that compresses videos across spatial and temporal dimensions in a 3D Variational Autoencoder (VAE). By using hierarchical spatiotemporal transformers and a factorized design, Large-scale diffusion models like VideoComposer [34] and CogVideoX [33] condition on text, masks, or optical flow to produce plausible short clips, yet lack any notion of depth, occluded content, or kilometre-scale consistency but provide a very powerful base model.

3 Method↩︎

3.1 Overview↩︎

Our framework adopts a two-stage design for large-scale 3D scene and long-term video generation. In the first stage, we reconstruct a detailed occupancy representation from a multi-view frame and encode it into a triplane-based generator, enabling fast and flexible synthesis of coarse 3D occupancy scene under arbitrary configurations. In the second stage, we fine-tune a video diffusion model to translate the synthesized occupancy scene into high-fidelity multi-view driving videos. The generated videos are further utilized to refine the occupancy representation, forming a fine-grained updating loop. Throughout the process, we construct reference images and multiple control conditions to ensure long-term temporal and cross-modal consistency between the occupancy representation and video generation.

3.2 Image Guided Voxel Generation↩︎

We adopt XCube[20] as the backbone for our initial scene reconstruction module. XCube is a large-scale 3D generative model that represents geometry using hierarchical sparse voxel grids and synthesizes scenes via a latent diffusion process. Specifically, it learns a distribution over a latent variable \(\mathbf{X}\) encoded by a sparse structure Variational Autoencoder (VAE) defined on voxelized 3D geometry. The VAE encodes sparse voxel hierarchies into compact latent representations, and a diffusion model operates in this latent space to generate sparse voxel structures. Both the VAE encoder–decoder and the diffusion model are implemented using sparse convolutional neural networks [35], enabling efficient modeling of high-resolution geometry up to \(1024^3\).

To condition XCube on input images, we use LSS [36] unprojection pipeline to encode the image feature extracted from DINO-V2 [37] into a dense 3D voxel grid \(\mathbf{\Omega}\). The image features are processed with trainable 2D conv layers stored into feature channel \(\mathbf{C}\) \[\mathbf{F}_{j d}^i=\theta_{j d}^i \cdot \mathbf{F}_j^i, \quad \mathbf{C}_v=\sum_{(i, j, d)} \mathbf{F}_{j d}^i \in \mathcal{R}^C .\] The \(\theta_{j d}^i \in \mathbb{R}^D\) is \(D\)-dimensional Softmax-normalized vector pixel level depth distribution, the \(d \in[1, D]\) denotes index of the depth pixel where \(v\) denotes the index of a voxel. A key difference from indoor or object-centric setups is that outdoor driving scenes provide sparse and uneven camera coverage, we cannot naively broadcast per-pixel features along the entire ray, as done in prior work [38]–[40] since that the geometries is not precisely corresponding with camera frusta. Once we constructed condition grid \(\mathbf{C}\), we concatenate it with the latent \(\mathbf{X}\) and feed into diffusion network as conditioning to produce high quality voxel grid for further extrapolator.

3.3 Sketch Guided Voxel Generation↩︎

Our framework represents the environment as an explicit occupancy grid \(\mathcal{O}_t\in\{0,1\}^{H\times W\times D}\) that evolves over timesteps \(t\). Given the single-frame multi-view input, the reconstruction module (Sec. [sec:voxel32recon]) reconstructs \(\mathcal{O}_0\) with fine-grained semantics and geometry. We regard this grid as the initial state of and learn a triplane based diffusion head \(q(\mathcal{O}_{t+1}\mid\mathcal{O}_t,\mathbf{u}_t)\) that predicts the next occupancy volume conditioned on (i) the current grid \(\mathcal{O}_t\) and (ii) a compact control vector \(\mathbf{u}_t\) encoding the desired chunk Sketch trajectory. To address the challenge of maintaining geometric and semantic coherence across extended trajectories, we propose a hierarchical sketch-and-refine paradigm that decomposes long-range occupancy generation into two stages: coarse-level sketching and fine-level image-conditioned refinement method.

To achieve controllable occupancy scene out-painting and fine-level image-conditioned occupancy updating, we encode the semantic occupancy grid to triplane latent representation [26]. The triplane latent projects the 3D occupancy onto three orthogonal rigid axes, decomposing the volumetric representation into three complementary 2D feature planes: \(XY\), \(XZ\), and \(YZ\) planes. By querying features from all three planes and aggregating them through learned fusion mechanisms, we can regenerate detailed 3D semantic information while maintaining computational efficiency. In the subsequent parts of this section, all operations are performed within the triplane representation space.

Coarse-level Sketching Guided Generation.↩︎

Given an initial scene \(\mathbf{O}_0\) and a trajectory sequence \(\{T_j\}_{j=0}^N\), the sketch stage generates a coarse-level scene layout along the entire control trajectory. This stage operates at reduced spatial resolution to capture the global scene structure efficiently. The Sketch generation is conditioned on both the trajectory and the initial scene context for coherence: \[\mathbf{X}_{sketch} = D_{\theta}(\mathbf{x}_0, \{T_j\}_{j=0}^N),\] Where \(D_{\theta}\) denotes denoising loop, \(\mathbf{X}_{sketch}\) represents the generated coarse-level scene spanning the full trajectory with a volume of \(128^3\). Then we sample the occupancy alone the trajectory with a standard resolution triplane and also a sliding window denoising loop for a coarse level occupancy representation as shown here: \[\mathbf{X}_{i} = D_\theta(\mathbf{V}_{i-1}; \mathbf{C}_{context}, \mathbf{C}_{sketch}),\] where \(\mathbf{C}_{context}\) and \(\mathbf{C}_{sketch}\) contains details from the previously refined patch \(\mathbf{X}_{i-1}\), \(\mathbf{V}_{i}\) indicates the triplane spatial rotation and translation vector for triplane to sample alone the trajectory. This step ensures smooth transitions and maintains local coherence across window boundaries.

This step provides a semantically consistent global layout that serves as the foundation for subsequent refinement.

3.4 Bidirectional Closed-Loop Generation↩︎

Controllable Video Generation.↩︎

We use CogvideoX1.5-5B as our base model and LoRA-fine-tuned on our processed dataset. It is non-trivial to design a 3D representation for conditioning and generating, but we aim to efficiently store both the semantic and geometric information of any occupancy input. We use multi-plane images (MPIs) [41] as a representation to further improve video quality through acquiring 3D information of occluded objects.

3.4.0.1 ControlNet-style Injection.

As demonstrated in Fig. 3, we encode the \(D\)-plane stack acquired from camera frustum with a \(1{\times}1\) Conv \(+\) ReLU encoder \(\phi_M\) (no down-sampling) and inject it into the UNet decoder features \(\{h^\ell\}\) at matching resolutions: \[\tilde{h}^\ell \;\leftarrow\; h^\ell \;+\; \lambda_M\, \phi_M\!\big(\mathrm{MPI}^\ell\big),\] where \(\lambda_M\) scales geometry control. Using \(1{\times}1\) keeps exact spatial correspondence between multi-plane features and latent pixels, improving label alignment and compute over \(3{\times}3\) alternatives.

3.4.0.2 Multi-view/frame Consistency.

We use zero-initialized cross-view attention to exchange information with neighboring cameras and cross-frame attention to propagate temporal context: \[\begin{align} \mathrm{Attn}(Q,K,V)&=\mathrm{softmax}\!\Big(\tfrac{QK^\top}{\sqrt{d}}\Big)V,\\ h_{\text{out}}&=h_{\text{in}}+\!\!\sum_{i\in\{l,r\}}\!\!\mathrm{Attn}(Q_{\text{in}},K_i,V_i), \tag{1}\\ h_{\text{out}}&=h_{\text{in}}+\!\!\sum_{i\in\{f,h\}}\!\!\mathrm{Attn}(Q_{\text{in}},K_i,V_i), \tag{2} \end{align}\] where \(\{l,r\}\) denote left/right adjacent views and \(\{f,h\}\) future/history frames. Eqs. 1 –2 maintain instance coherence across overlapping FoVs and time. We also design a Ref-image initial conditioned injection mechanism for long-term video stability and maintain the temporal consistency. For every video generation chunk, we took last generated video frame or the initial frame if its the first chunk \({\mathrm{F}_t}\), and encoded it through vanilla VAE proposed from CogvideoX. Then we concat it through similar conditioned injection introduced in Sec 3.4.0.1 to formulated a text-pfirompted and ref-image enabled spatial temporal consistent video generation head. As illustrated in Fig. 3 we route different vae decoders for various output. Through this manner, we support 2D-3D aligned video generation with precisely occupancy-centric conditioning and outputs across semantics, depth and RGB demonstrated in Fig. 5 in nuScenes. We also present down-sampled lidar from our generated occupancy grid as lidar sensor output.

Fine-level Refinement.↩︎

The refinement stage operates on local scene patches using a sliding-window strategy to progressively enhance the coarse, sketch-based scene with fine-grained details contributed by the powerful video diffusion head. For each window position \(i\text{-th}\), we encode the corresponding camera frustum into a multi-plane image (MPI), which serves as the 3D condition for a fine-tuned video diffusion model. This process generates spatially and temporally consistent multi-view videos with high visual fidelity. With these high-quality videos, the large-scale driving scene generation task is now an image-conditioned occupancy reconstruction problem. We thus reuse the reconstruction model introduced in Sec. 3.2 to obtain detailed, high-resolution occupancy representations. This improved occupancy further enables more accurate video generation, forming a closed-loop refinement process that progressively enhances both the 3D occupancy and the visual quality of the driving videos, achieving strong multi-modal alignment.

This two-stage approach enables the generation of arbitrarily long scenes while preserving both global semantic consistency through the sketch stage and local geometric fidelity through the refinement stage. The overlapping regions between adjacent windows provide continuity constraints that prevent artifacts and maintain seamless scene transitions throughout the extended trajectory.

3.4.0.3 Text-driven Weather and Style Control.

High-level appearance is provided by a text encoder \(\tau_\theta(\cdot)\) that embeds prompts \(y\) into conditioning tokens \(g=\tau_\theta(y)\). We form \(y\) by concatenating weather/style clauses (e.g., “snowy”, “sandstorm”) with scene text; classifier-free guidance is used at sampling time. We inject \(g\) via the standard cross-attention layers of the latent diffusion UNet.

3.4.0.4 Unified Denoising Objective.

We train the video diffusion model using a standard latent denoising objective conditioned on all available control signals: \[\mathcal{L}_{\text{base}} \;=\; \mathbb{E}_{\epsilon,\,t}\!\left[ \big\|\epsilon - \epsilon_\theta\!\left(z_t,\, t,\, \phi_M(\mathrm{MPI})\right)\big\|_2^2 \odot w \right],\] where \(\epsilon_\theta\) denotes the noise predicted by the denoising network at diffusion step \(t\), \(z_t\) is the noisy latent, and \(\phi_M(\mathrm{MPI})\) encodes the multi-plane image (MPI) as the 3D control signal. The weight term \(w\) optionally applies importance weighting, such as progressive or depth-aware weighting, to stabilize training and emphasize geometrically meaningful regions.

To further enhance the generation quality of dynamic scene components, we introduce an additional region-weighted reconstruction loss that assigns higher importance to road surfaces and road actors (vehicles, pedestrians, cyclists, et al.): \[\mathcal{L}_{\text{region}} \;=\; \mathbb{E}_{\epsilon,\,t}\!\left[ \big\| \big(\epsilon - \epsilon_\theta\!\left(z_t,\, t,\, \phi_M(\mathrm{MPI})\right)\big) \odot m_{\text{road}}\big\|_2^2 \right],\] where \(m_{\text{road}}\) is a binary or soft mask highlighting road and actor regions derived from semantic category.

The final training objective combines both terms: \[\mathcal{L}_{\text{video}} \;=\; \mathcal{L}_{\text{base}} \;+\; \lambda_{\text{region}}\,\mathcal{L}_{\text{region}},\] where \(\lambda_{\text{region}}\) controls the relative importance of the region-weighted constraint.

This composite loss encourages the diffusion model to focus on both global temporal coherence and fine-grained generation of critical driving scene elements, yielding visually consistent and semantically accurate multi-view videos and also improves the fidelity of the assigned category in the bidirectional cross-Modal closed-loop generation.

3.4.0.5 Ultra-long video rollout.

Let \(\{x_{1:F}\}\) be the target frame sequence, and let \(\mathcal{C}=\{\mathrm{MPI},g\}\) be the fixed controls for the rollout. We generate with a sliding-window sampler of length \(W\) using cross-frame attention: \[\begin{align} p_\theta(x_{t}\mid x_{t-W:t-1},\mathcal{C}) \;\propto\; \prod_{k=K}^{1} p_\theta\!\big(z_{k-1}\mid z_k,\, \mathcal{C},\, x_{t-W:t-1}\big), \end{align}\] periodically re-anchoring features with \(\phi_M(\mathrm{MPI})\) (every \(R\) frames) to curb drift and preserve long-horizon geometric fidelity. The cross-frame operator in equation 2 enables temporal propagation beyond training clip lengths; combined with prompt controls, it sustains consistent appearance over arbitrarily long sequences.

4 Experiment↩︎

4.1 Dataset and Experimental Setup↩︎

4.1.0.1 Data Curation

We conduct comprehensive experiments on the nuScenes[42] and Waymo Open Dataset [23]. To support training and evaluation, we further extend the data processing pipeline provided by SCube [21] to construct a more accurate 3D occupancy dataset. To enhance the geometric fidelity of our representations, we integrate dense geometry reconstructed by the multi-view stereo pipeline of COLMAP [43], which significantly enriches the voxel representation with fine-grained structural details and effectively addresses the height limitations inherent in LiDAR sensors. We further add dynamic road actors extracted from ground-truth LiDAR frames [31], including vehicles, pedestrians, and cyclists et al. A key focus of this data curation process is maintaining precise alignment across modalities—ensuring that RGB imagery, LiDAR point clouds, and occupancy representations are geometrically and temporally consistent within a unified world coordinate frame. This alignment is crucial for learning reliable 2D–3D correspondences, enabling InfiniVerse to jointly reason about appearance, geometry, and dynamics. Each curated scene therefore contains both static environmental geometry and temporally coherent dynamic elements, providing a strong foundation for long-horizon, controllable, and multi-modal grounded scene generation. To enable precise text control for video generation, we implement a sophisticated multi-stage captioning process. First, we divide each video clip into three segments corresponding to different front-view perspectives. For each segment, we apply a multi-stage captioning process across three front-views, beginning with InternVL2-40B-AWQ [44], [45] to generate dense video captions. These captions are then refined using Meta-Llama-3-8B-Instruct [46] to improve linguistic quality and ensure better alignment with the requirements of controllable video generation. To guarantee semantic consistency between video and text, we further apply VideoCLIP-XL [47] to filter out poorly aligned video-caption pairs.

Following standard practices in the field, we remove low-quality voxel grids and sequences with predominantly static ego trajectories, resulting in a curated dataset of 618 sequences for training and 130 sequences for evaluation in Waymo Open Dataset. We also use Occ3D-nuScenes[31] to train the occupancy generation branch and nuScenes[42] to train the multi-view multi-modal video generation branch. This filtering and dataset selection ensures the quality and relevance of our experimental data.

4.2 Implementation Details↩︎

Our implementation follows a carefully designed training protocol to ensure optimal performance across all components. The image-conditioned occupancy generation stage is trained for 100 GPU days including VAE compression, with a inference time in 2 minutes and 40GB VRAM per chunk. We use four 2D convolutional layers(channel dims: [768, 256, 256, 32, 32], kernel size: 3, stride: 1) to further process the DINO-V2 output to predict the image feature. The triplane based sketch guided diffusion is trained for 20 GPU days with inference time in 30s per triplane. For the triplane autoencoder, the input scene is encoded to triplane with a spatial resolution \((X_h, Y_h, Z_h) = (128, 128, 32)\), and the feature dimension \(C_h\) is 16, the learning rate is initialized to 1e-4 and then decreases linearly. During the diffusion process, we use the default settings[54] with 100 time steps (\(T\)). The video generation component is trained 64 GPU days, with a inference time in 2.5 minutes with 55GB VRAM per chunk. We adopt CogvideoX1.5-5B as our base model and LoRA-fine-tuned on our processed dataset. We use UniPC scheduler [55] with the classifier-free guidance (CFG) [56] that is set as 7.0. During inference, we use 20 denoising steps for dataset generation. We employ a temporal compression ratio of 4 to balance computational efficiency with temporal fidelity. All experiments are conducted on NVIDIA A800 GPUs with 80GB VRAM with mixed precision training to optimize memory usage and training speed.

4.3 Metrics↩︎

We use Fréchet Inception Distance (FID)  [57] and Fréchet Video Distance (FVD)[58] to measure the perceptual quality of generated video, and use mIoU to measure the precision of occupancy prediction.

For Voxel grid reconstruction, we utilize IoU and mIoU, the final IoU of the fine-level voxel grid reached 34.31%, and the mIoU that considers the accuracy of the voxel’s semantic prediction reached 20.12%.

4.4 Results↩︎

Fig. 4 presents qualitative results of our out-painted driving world together with the corresponding RGB video generation on the Waymo Open Dataset [23]. From top to bottom, we visualize the generated future frames along the predicted trajectory. The first frame corresponds to the ground-truth observation, which provides the initial appearance conditions such as weather and scene style. The subsequent frames illustrate that our method can generate long-horizon driving videos with realistic appearance while maintaining strong controllability.

Fig. 5 further demonstrates the multi-modal consistency of our framework on the nuScenes dataset [42]. Our generated results are aligned across multiple modalities, including RGB images, semantics, depth, occupancy, and LiDAR, highlighting the effectiveness of the proposed 2D–3D coupled generation paradigm.

Fig. 6 shows that, given an arbitrary user-defined road layout sketch, our framework generates a long-range occupancy scene that follows the specified layout and guides the subsequent video generation. Tab. 1 provides a quantitative comparison with existing methods. Despite using only a single multi-view frame as input, InfiniVerse achieves state-of-the-art video generation quality. Thanks to the proposed reciprocal refinement loop, our framework is able to produce the first long-horizon driving videos that remain consistently aligned with the underlying 3D occupancy representation.

We report a horizon-dependent FID analysis on Waymo. Since Vista [9] does not support multi-view generation, we perform this comparison in the single-view setting. For each target horizon \(T\), we evaluate the generated frame at step \(T{+}1\) and plot FID as a function of rollout length up to 300 frames.

We compare InfiniVerse against Vista in this analysis. Vista is re-implemented by ourselves. As shown in Fig. 7 (a), the FID of InfiniVerse increases much more slowly than that of Vista as the rollout horizon becomes longer. As shown in Fig. 7 (b), we report FVD at rollout horizons of 12, 24, 36, 48, 96, and up to 300 frames. As the rollout horizon increases, InfiniVerse consistently maintains lower FVD values and exhibits a slower degradation trend, indicating stronger long-range temporal stability. Table 2 presents a direct FVD comparison with Vista [9] under different rollout lengths. Among existing baselines, Vista is the only method that can be both reproduced and adapted to a setting close to ours.

We further conduct extensive ablation studies to evaluate the effectiveness of each component in the proposed pipeline in Tab. ¿tbl:ablation32framework?. Using only the 3D branch (voxel-to-video) or only the image branch (image-to-video) results in noticeable performance degradation due to the lack of either temporal consistency or geometric priors. Starting from the reference image and incorporating the 3D occupancy representation, the introduction of the proposed reciprocal loop and region loss significantly improves generation quality, validating the effectiveness of our design.

5 Conclusion↩︎

In this paper, we introduced InfiniVerse, a unified framework for controllable long-horizon generation of dynamic urban scenes for autonomous driving. Starting from a single multi-view observation, our method reconstructs a 3D occupancy representation that serves as a geometric foundation for autoregressive scene expansion along arbitrary trajectories.

To bridge spatial reasoning and visual realism, we couple occupancy-guided world modeling with a video diffusion generator. Central to our framework is a sketch-and-refine paradigm that establishes a reciprocal feedback loop between 3D occupancy generation and 2D video synthesis: coarse occupancy grids provide structural guidance for video generation, while the generated frames are re-projected to refine the underlying 3D scene representation.

This bidirectional interaction enables consistent geometry–appearance alignment and significantly improves long-horizon stability while reducing hallucination artifacts. Extensive experiments demonstrate that InfiniVerse generates temporally consistent and visually realistic driving scenarios, achieving state-of-the-art performance on standard benchmarks.

Overall, InfiniVerse provides a scalable paradigm for synthesizing controllable urban environments, offering a promising direction for data generation, simulation, and large-scale evaluation in autonomous driving systems.

References↩︎