Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation
July 15, 2026
Abstract
3D simulation platforms are critical for autonomous driving because they enable end-to-end policy evaluation, thereby reducing development costs and improving safety. In recent years, neural simulation has become predominant, with methods such as NuRec [1] playing a central role; however, these methods remain relatively slow and typically require per-scene tuning. In this work, we present Instant NuRec, a feed-forward neural reconstruction model that turns a short multi-view driving log into a fully simulatable 3D Gaussian Splatting (3DGS) world in a single forward pass. The model accepts multi-view input from a calibrated camera rig and emits a layered output consisting of static and dynamic 3DGS layers, a sky cubemap, and per-camera ISP corrections, while providing native support for non-pinhole camera models via 3DGUT [2]. It reconstructs a 10–20-second multi-camera scene in roughly 1.5 seconds and achieves a PSNR on the Waymo Open Dataset that is 2.01 dB above the strongest evaluated baseline. Instant NuRec is deeply integrated into NuRec and is compatible with AlpaSim [3] for closed-loop simulation.
Figure 1: 
.
1 Introduction↩︎
Closed-loop simulation has become a central tool for developing and validating autonomous driving (AD) stacks: it allows planners and perception modules to be tested against rare, safety-critical, or otherwise hard-to-collect interactions in a controllable environment. A faithful simulator must combine three ingredients: (i) photorealistic rendering of the surrounding world, (ii) an explicit, physics-simulation-ready decomposition of the scene into a static background and individually editable dynamic agents, and (iii) the ability to freely re-pose the ego camera and re-time the scene so that counterfactual rollouts can be evaluated. Recent advances in 3D Gaussian Splatting (3DGS) [4] have made high-quality rendering computationally inexpensive, while per-scene optimization frameworks such as OmniRe [5] have shown that the resulting reconstructions can faithfully reproduce a real driving log. However, per-scene optimization is fundamentally a bottleneck to scaling: each new clip requires hours of computation, careful hyperparameter tuning, and a large set of auxiliary inputs (LiDAR sweeps, fitted poses, and semantic masks). This cost makes it impractical to ingest the millions of clips that modern AD fleets collect every day.
In this paper, we introduce Instant NuRec, a feed-forward neural reconstruction model that turns a short multi-view driving log into a fully simulatable 3DGS world in a single forward pass. The input is a sequence of images captured by a rig with 1–5 calibrated cameras. A single forward pass through Instant NuRec produces (i) a static 3DGS layer that captures roads, buildings, and other rigid background elements; (ii) a dynamic 3DGS layer with piecewise-linear trajectories that represents moving vehicles and vulnerable road users; and (iii) a cubemap that models the distant sky. The output can be exported in USDZ format and consumed directly by closed-loop simulation frameworks such as AlpaSim [3], allowing a change in clip selection to be reflected in the simulator within seconds rather than hours.
Instant NuRec couples a single shared encoder with a set of lightweight task-specific decoders. The encoder is an alternating-attention Vision Transformer [6] that interleaves intra-image and cross-image attention to fuse information across the input cameras and over time. Its shared features drive several decoders that predict dense depth, normals, semantics, a sky cubemap, and per-camera color corrections. The predicted depth lifts a set of query points into world space to anchor the Gaussians, the semantic head separates the static background from dynamic agents, and a Gaussian head supplies the remaining shape and appearance attributes needed to complete the layered output. Crucially, a motion head directly recovers piecewise-linear actor trajectories for the query points, so the dynamic layer requires no per-actor cuboid tracks at reconstruction time. We train the model in three stages that deliberately decouple geometry supervision from rendering supervision, keeping large-scale training on \(\sim\)40K internal driving clips efficient.
In short, Instant NuRec turns a multi-camera driving log into a complete 3DGS world with static, dynamic, and sky components in a single feed-forward pass, ready for direct use in closed-loop simulation. Across novel-view synthesis, geometry, and downstream simulator integration, it closes a large fraction of the quality gap to per-scene optimization while requiring \(10^3\)–\(10^4\!\times\) less computation per clip, making large-scale reconstruction of driving logs practical. Our code is available at https://github.com/nvidia/instant-nurec, with additional documentation at https://docs.nvidia.com/nurec/index.html.
2 Related Work↩︎
Per-scene neural reconstruction for driving. Neural Radiance Fields (NeRFs) [7] and 3DGS [4] have set the standard for photorealistic novel-view synthesis, and a substantial body of follow-up work has specialized them to outdoor driving scenes. Early urban radiance fields scaled reconstruction to street and city scales using posed RGB and LiDAR [8], [9], while Neural Scene Graphs [10] introduced the now-standard decomposition of a driving scene into a static background and a set of rigidly moving foreground actors. Later methods pushed this toward self-supervised dynamics and sensor-realistic simulation [11], [12] and replaced the volumetric field with explicit Gaussians for faster, editable reconstruction [13]–[15]. OmniRe [5] extended this decomposition to non-rigid pedestrians and cyclists, while 3DGRT [16] and 3DGUT [2] introduced Gaussian renderers that natively support distorted cameras and secondary rays, removing the rectified-pinhole assumption. These methods produce extremely high-quality reconstructions but require minutes to hours of per-clip optimization and a rich set of auxiliary inputs (LiDAR, multi-traversal poses, semantic masks, and cuboid tracks). Instant NuRec preserves the same output format, namely a layered 3DGS world consumable by NuRec [1], [17] and AlpaSim [3], but amortizes the optimization cost into a single feed-forward pass.
Feed-forward 3D reconstruction. Feed-forward 3D models have rapidly progressed from single-object generation [18]–[25] to scene-level prediction. At the scene level, generalizable Gaussian models predict per-pixel Gaussians from sparse posed views in a single pass using probabilistic depth [26], plane-sweep cost volumes [27], large transformer decoders [28], or monocular depth priors [29]. In parallel, another line of work regresses pointmaps or Gaussians in a pose-free manner directly from uncalibrated images [30]–[32]. This paradigm has recently been specialized to dynamic driving scenes: STORM [33] and DrivingForward [34] predict static and dynamic Gaussians from surround-view video, and a wave of contemporaneous models extends this to pose-free, tracker-free, and off-road settings [35]–[38]. A separate line of feed-forward 3DGS work decouples Gaussian prediction from input pixels: instead of emitting one Gaussian per pixel, TokenGS [39] regresses Gaussians from a fixed set of learnable tokens, and related query- or scene-token formulations [40], [41] similarly unbind the primitive count from image resolution and view count. Unlike these reconstruction-focused models, which target bare RGB Gaussians, Instant NuRec emits a complete, simulation-ready layered 3D world.
Driving world models. A complementary and rapidly growing line of work learns generative models that synthesize plausible driving observations conditioned on the past and on control signals such as ego trajectory, layout, or text. Early latent-diffusion models [42]–[45] generate controllable single- or multi-view street video. Subsequent work improves fidelity and controllability, extends the prediction horizon [46]–[50], adopts autoregressive video-GPT or diffusion backbones for long-range rollouts [51], [52], and culminates in large world foundation models trained across driving and other embodiments [53], [54]. These models are powerful content generators but, by construction, emit 2D pixels rather than an explicit 3D state: they cannot be freely re-posed by a planner and do not directly yield a simulatable scene. A second line therefore uses generative video priors to repair or densify neural reconstructions, synthesizing novel-trajectory frames offline as a data machine [55], [56] or removing ghosting and floater artifacts online as an inference-time enhancer [57]–[59]. Intermediate variants are conditioned on point clouds or pseudo-renders [60], [61]. Most recently, joint formulations such as the Xiaomi EV world model [62] tightly couple a feed-forward reconstructor with a video generator, allowing geometry to anchor generation while the generator fills unobserved regions. Instant NuRec occupies the reconstruction side of this framework: it does not invent content but emits an explicit, navigable 3D world in a single pass, making it complementary to generative world models and a natural, fast 3D backbone for the generative-repair and joint-model lines.
3 Method↩︎
At a high level, Instant NuRec takes a short multi-view driving log as input and produces a layered 3DGS world in a single feed-forward pass. We first describe the problem formulation (input/output representation) in 3.1, then the shared encoder in 3.2, and finally the decoder heads that produce the renderable representation in 3.3. A schematic of the pipeline is shown in 2.
3.1 Problem Formulation↩︎
Input. The model accepts \(V\!\times\!T\) RGB images, where \(V\) is the number of available cameras and \(T\) is the number of temporal frames sampled from the source camera videos at 2–4 Hz. Each image \({\mathbf{I}}_{v,t}\) is paired with its 6-DoF pose \(\mathbf{T}_{v,t}\!\in\!SE(3)\) and a vector of camera-intrinsic parameters \(\boldsymbol{\kappa}_{v,t}\). Optional cuboid tracks for dynamic actors can be supplied at inference time to further calibrate the trajectories of the dynamic Gaussians.
Output. Instant NuRec emits a layered representation:
Static layer \({\mathcal{G}}^s\), a set of \(N_s\) static Gaussians, each parameterized by attributes \(({\boldsymbol{\mu}}_i,{\boldsymbol{s}}_i,{\mathbf{q}}_i,\alpha_i,c_i,\boldsymbol{n}_i,\ell_i)\), with world-space position \({\boldsymbol{\mu}}\!\in\!\mathbb{R}^3\), scale \({\boldsymbol{s}}\!\in\!\mathbb{R}^3\), rotation quaternion \({\mathbf{q}}\), opacity \(\alpha\!\in\![0,1]\), color \(c\!\in\!\mathbb{R}^3\), world-space normal \(\boldsymbol{n}\!\in\!\mathbb{S}^2\), and a semantic class \(\ell\) (road/non-road) used by downstream simulators. The quaternion primarily parameterizes the Gaussian’s rotation for view rendering, whereas the normal is used primarily to extract surfaces for simulation.
Dynamic layer \({\mathcal{G}}^d\), a set of \(N_d\) dynamic Gaussians with the same per-Gaussian attributes as the static layer, except that the position \({\boldsymbol{\mu}}_i\) is replaced by a piecewise-linear trajectory \({\boldsymbol{\mu}}(t)\) defined by knots \(\{(t_k,{\boldsymbol{\mu}}_k)\}_{k=1}^{3}\). For \(t \in [t_k,t_{k+1}]\), the trajectory is evaluated as \({\boldsymbol{\mu}}(t)\!=\!(1\!-\!\lambda)\,{\boldsymbol{\mu}}_k\!+\!\lambda\,{\boldsymbol{\mu}}_{k+1}\) for \(\lambda\!=\!(t\!-\!t_k)/(t_{k+1}\!-\!t_k)\). The opacity is gated by a smooth fade \(f(x)\!=\!\exp(-x^{10})\) around \(t_1\) and \(t_3\).
Sky cubemap \({\mathbf{I}}^{\mathrm{sky}}\!\in\!\mathbb{R}^{6\times H\times W\times 3}\), which represents the distant background.
ISP affine transforms \(\{\mathbf{A}_v\}\!\in\!\mathbb{R}^{V\times 3\times 4}\), applied to rendered colors before the photometric loss to absorb residual per-camera color/exposure mismatch.
3.2 Shared Alternating-Attention Encoder↩︎
We largely follow the backbone design of Depth-Anything-3 [63], which repurposes the DINOv2 [64] architecture for multi-view inputs. Compared with other geometry backbones, it produces sharp and accurate depth maps without requiring an additional image tokenizer.
In our model, the encoder \({\boldsymbol{\phi}}\) jointly tokenizes all input images and produces a shared latent \({\mathbf{F}}\!\in\!\mathbb{R}^{N\times C}\), where \(N\) is the total number of tokens and \(C\) is the feature dimension. Each image is split into non-overlapping \(14\!\times\!14\) patches, each of which is linearly projected to a token of dimension \(C\). A per-image class token, derived from the camera’s 6-DoF pose and field of view using sinusoidal positional encoding, is prepended to provide a global summary of the view and inject pose information directly into attention. The combined sequence is processed by a stack of \(L\) alternating attention blocks that interleave intra-image self-attention and cross-image self-attention. This interleaving design enables effective information exchange between patches in different views while maintaining computational efficiency in layers that apply intra-image attention.
3.3 Decoding Heads↩︎
Several decoding heads operate on the shared latent features to produce the final output. These decoders are lightweight, so adding an output head incurs little additional cost at inference.
Context-view decoders. We use context views to denote the input images observed by the model, as opposed to held-out target views used for rendering supervision and evaluation. The context-view decoders produce dense, pixel-aligned scene attributes for these input views. We use a set of DPT [65] fusion heads to fuse latent features from multiple layers into attribute maps. Specifically, we attach three DPT heads: one each for depth, normals, and semantic logits. The depth branch follows the Depth-Anything-3 model, but we remove the confidence channel because reconstruction tasks generally require dense predictions over the full image. For the semantic logits, we predict four classes: road, movable (vehicles and pedestrians), sky, and ego-car.
3DGS decoder. Most existing feed-forward reconstruction methods emit one Gaussian at every pixel. While this usually provides strong reconstruction quality, the number of Gaussians can be large and their pixel-aligned geometry may not reflect the underlying scene structure. In our design, we decouple the Gaussians from the latent features by selecting a set of query points from the lifted depth map. Each query point in \({\mathbf{Q}}\) is a 3D position lifted from the predicted depth map and embedded by a lightweight learned projection. The query points cross-attend to keys \({\mathbf{K}}\) and values \({\mathbf{V}}\) derived from the latent feature \({\mathbf{F}}\), and the decoder outputs the per-query Gaussian attributes, including scale, rotation, and opacity. After decoding, the Gaussians from all queries are combined and can then be rendered by 3DGUT [2].
Motion decoder. The motion decoder takes the same set of query points \({\mathbf{Q}}\) as input and predicts 3D displacements to the target timestamps \(t_{\textrm{tgt}}\). For computational efficiency, we jointly predict two 3D displacements, one for each target timestamp, yielding a 6-D regression target. The motion decoder combines elements of V-DPM [66] and 4RC [67]: a shallow Transformer operates on the encoder features and is modulated by the source timestamps \(t_{\textrm{src}}\). The Transformer’s LayerNorms are AdaLN modules [68] conditioned on the target timestamps \(t_{\textrm{tgt}}\), informing the head which source-to-target motion \(t_{\textrm{src}} \rightarrow t_{\textrm{tgt}}\) to predict. For each source timestamp \(t_{\textrm{src}}\), we select the immediately preceding and following timestamps from the same camera as targets. Given displacement predictions \((\Delta_-, \Delta_+)\), the three final 3D positions of the dynamic layer are \({\boldsymbol{\mu}}_1 = {\boldsymbol{\mu}}+ \Delta_-\), \({\boldsymbol{\mu}}_2 = {\boldsymbol{\mu}}\), and \({\boldsymbol{\mu}}_3 = {\boldsymbol{\mu}}+ \Delta_+\), where \({\boldsymbol{\mu}}\) is the 3D position lifted from the depth map. Unlike STORM [33], in which all Gaussians are considered dynamic, we use the same query set throughout and subsequently apply a semantic-head mask, assigning only points classified as movable to the dynamic layer and the remainder to the static layer.
Query point selection. Query points anchor the predicted Gaussians in space, so their selection strongly affects reconstruction quality. We consider two strategies: (i) Dense, where every pixel in the context images is a query point; and (ii) Selective, which uses strided sampling and cross-attention to sparsify queries and includes a dedicated road branch for road-dominated regions. In Selective, query points, each associated with full-resolution per-pixel signals, are sampled with stride \(s\) and grouped into \(p\!\times\!p\) token clusters. Each query token cross-attends to the latent feature map (with QK normalization and layer scale for stability) and predicts a fixed cluster of \(M\!=\!p^2\) Gaussians, whose attributes are gathered from the corresponding group of signals. For example, with \(s\!=\!2\) and \(p\!=\!8\), a single \(448\!\times\!784\) context view yields \(1{,}372\) tokens and roughly \(88\)K Gaussians with the Selective strategy. This represents a \(4\times\) reduction from the \(\sim\!351\)K produced by Dense while empirically retaining most of the reconstruction quality.
Sky decoder. The sky decoder maps the shared latent \({\mathbf{F}}\) to the distant-background cubemap \({\mathbf{I}}^{\mathrm{sky}}\). For each cubemap pixel, we form a query by sinusoidally encoding its world-space ray direction. The query cross-attends to encoder features augmented with per-camera ray embeddings so that each pixel can preferentially borrow color from the input view that observed a similar direction. A DPT-style fusion head upsamples the cross-attention output to full cubemap resolution.
ISP decoder. This decoder predicts a per-camera \(3\!\times\!4\) affine RGB transform, analogous to appearance compensation in neural rendering [69], that absorbs residual differences in white balance, gamma, and vignetting between cameras observing the same scene. A learned token cross-attends to the deepest encoder features, grouped by camera index, and a final linear projection emits the affine matrix and bias. The transform is initialized to the identity and applied to rendered colors before the photometric loss is computed. At reconstruction time, downstream simulators may either apply the transform to match the training rig’s color response or omit it to render in the canonical color space.
Forward-pass summary. Putting the pieces together, the encoder \({\boldsymbol{\phi}}\) maps the input clip to a shared latent representation \({\mathbf{F}}\), from which the context DPT heads predict dense depth, normals, and semantics. Query points \({\mathbf{Q}}\) are lifted from the depth map using either the Dense or Selective strategy. The 3DGS decoder cross-attends \({\mathbf{Q}}\) to \({\mathbf{F}}\) and assembles static Gaussians \({\mathcal{G}}^s\) with scale, rotation, opacity, color, normal, and semantic labels. The motion decoder uses the same queries \({\mathbf{Q}}\) to predict displacements; only queries subsequently masked as movable are assigned to dynamic Gaussians \({\mathcal{G}}^d\) with piecewise-linear trajectories. The sky and ISP heads produce \({\mathbf{I}}^{\mathrm{sky}}\) and per-camera affine transforms \(\{\mathbf{A}_v\}\) in parallel; no iterative optimization is performed at inference.
3.4 Long-clip Processing via Chunk Merging↩︎
Because the model is trained on short temporal windows for memory and computational efficiency, reconstructing an entire driving log requires processing it in overlapping chunks and merging the per-chunk predictions into a single global field.
Per-chunk pruning. Within each chunk, we drop Gaussians whose opacity is below a small threshold (\(\alpha\!<\!10^{-2}\)) and transform the surviving primitives into a global world frame using the chunk-level pose. In practice, this can reduce the number of Gaussians while maintaining rendering quality.
Frustum-ownership pruning. A Gaussian predicted in one chunk, particularly at a great distance from the chunk’s cameras, often lands inside another chunk’s frustum, where it competes with that chunk’s own closer, better-conditioned Gaussians and creates floaters. We therefore apply a frustum-ownership test: a Gaussian \(g\) from chunk \(i\) is kept only if no other chunk \(j\) observes it from substantially closer range. Specifically, define \(D_{gi}\) as the Euclidean distance from the Gaussian’s center to the optical center of its closest camera in chunk \(i\), with \(D_{gi}\!=\!\infty\) if \(g\) lies outside chunk \(i\)’s frustum pyramid. We retain \(g\) from chunk \(i\) if and only if \[D_{gi}\;\le\;\min_{j}\,D_{gj} + \delta \label{eq:frustum-mask}\tag{1}\] where \(\delta\) is a small tolerance that keeps Gaussians on the boundary between chunks. Intuitively, each Gaussian is “owned” by the chunk that observed it at the closest range; downstream chunks see only their own near-field reconstruction and inherit the far field from their closer neighbors. Because dynamic Gaussians are active only over a short time window and rarely overlap with another chunk’s active interval, we leave the dynamic layer untouched and apply the pruning to the static layer only.
4 Training↩︎
We describe the training objective and curriculum in 4.1 and the implementation details in 4.2.
4.1 Losses and Training Curriculum↩︎
We train the model with three groups of objectives: \[\mathcal{L}= \mathcal{L}_{\text{context}} + \mathcal{L}_{\text{motion}} + \mathcal{L}_{\text{render}},\] where
\(\mathcal{L}_{\text{context}}\) directly supervises the outputs of the context decoders. Attributes such as depth, normals, and semantic logits are pixel-aligned and have corresponding ground truth. We supervise these attributes with \(L_1\), cosine similarity, and cross-entropy losses, respectively.
\(\mathcal{L}_{\text{motion}}\) includes forward and backward 3D-flow losses, supervised by 3D scene flow derived from cuboid tracks for foreground pixels and a zero-displacement target for background pixels.
\(\mathcal{L}_{\text{render}}\) comprises differentiable rendering losses on novel, held-out training views sampled uniformly within the context time window. The loss consists of an \(L_1\) term and an LPIPS term [70], computed after applying the predicted camera-ISP affine transform. This rendering loss also backpropagates to the sky cubemap decoder. We additionally use an opacity loss to encourage transparency in sky regions so that the sky is represented primarily by the cubemap rather than the 3D Gaussians.
In practice, introducing \(\mathcal{L}_{\text{render}}\) at the beginning of training slows convergence. Moreover, rendering at every iteration is too expensive because of the large number of Gaussians. We therefore divide the full training process into a three-stage curriculum:
Stage 1: Pretraining. We reproduced the Depth-Anything-3 pretraining protocol using the depth- and ray-prediction tasks from [63] and the data-curation pipeline described in [71]. This stage provides a general initialization for the full backbone.
Stage 2: Context training. We enable the sky, depth-and-context, and motion decoders and fully fine-tune the encoder with these heads on the internally curated driving corpus with \(\mathcal{L}_{\text{context}}+\mathcal{L}_{\text{motion}}\); the context heads receive dense, well-conditioned supervision from LiDAR and auto-labels.
Stage 3: GS training. The encoder is frozen, and the camera-ISP and GS decoders are enabled. Because no ground-truth Gaussian attributes are available, we train these heads only after the geometry has stabilized, allowing rendering gradients to backpropagate through a sensible 3D scaffold. The Gaussians produced by the now-frozen geometry stack are rendered through 3DGUT [2] and additionally supervised with \(\mathcal{L}_{\text{render}}\).
Freezing the encoder in Stage 3 is a deliberate choice: it prevents high-variance rendering gradients from corrupting the well-conditioned geometry features learned in Stage 2 and substantially stabilizes training while the GS attributes are still random.
4.2 Implementation Details↩︎
Data curation and auto-labeling. Instant NuRec is trained on \(\sim\)40K filtered driving clips drawn from NVIDIA’s Autonomous Vehicle (AV) data platform. Each clip contains footage from up to five cameras, with \(\sim\)300–600 frames per camera at 30 Hz, along with the corresponding camera calibrations and rig-trajectory data. Each clip also includes LiDAR point-cloud data, from which we export depth-map ground truth. In addition to the raw sensor data and calibrations, we apply the following steps to obtain auxiliary training data:
Semantic segmentation is obtained by independently applying a state-of-the-art semantic segmentation model [1] to every image from each camera.
3D bounding boxes for movable actors are provided by an internal cuboid auto-labeler and tracker using LiDAR data.
Dense depth is obtained by accumulating LiDAR points across the clip and projecting them into each image; missing pixels (sky, ego-car, and occluded surfaces) are masked out of the depth loss. Regions belonging to dynamic actors are accumulated according to their auto-labeled bounding boxes.
Training infrastructure. Instant NuRec is implemented in PyTorch, with 3DGUT [2] serving as the differentiable renderer. Training uses standard DDP across nodes with an effective per-GPU batch size of 1–2 clips. We train the model with several common camera configurations \(V \in \{1,3,5\}\) (front only; front, front-left, and front-right; or front, front-left, front-right, rear-left, and rear-right) and varying temporal frame counts \(T \in \{8,12,18\}\). The learning rate follows a cosine annealing schedule with a linear warmup to a maximum of \(10^{-4}\). The complete three-stage training process takes \(\sim\)6 days on 8 nodes.
5 Experiments↩︎
We evaluate Instant NuRec along three axes: (i) novel-view synthesis quality against recent feed-forward baselines on the public Waymo Open Dataset (5.1); (ii) reconstruction quality and closed-loop simulation viability on our large-scale internal corpus (5.2); and (iii) ablations of the main design choices (5.3).
5.1 Comparison on the Waymo Open Dataset↩︎
For reproducibility and direct comparison with recent feed-forward reconstruction methods, we evaluate on the validation split of the Waymo Open Dataset [72]. Our evaluation follows the protocol used by STORM [33]. Each evaluation sample is a 2-second window containing 20 consecutive frames at 10 Hz. The model observes four context frames spaced 0.5 s apart and is evaluated on the held-out target frames. All RGB metrics are computed after resizing the predictions and ground truth to the STORM resolution of \(240\times160\) pixels.
Baselines. We compare against representative feed-forward reconstruction methods: DepthSplat [29], STORM [33], Depth-Anything-3 [63], and DGGT [35]. We evaluate the baseline methods using their released checkpoints without further fine-tuning. Because DGGT does not take poses as input, we align its predicted trajectory with the ground-truth trajectory before computing the metrics.
Metrics. For image quality, we report PSNR and SSIM on both the full image and the dynamic region. We define the dynamic region using semantic segmentation masks that select vehicle and movable-object pixels, and we use the same mask for all methods. For geometry, we follow the sparse-LiDAR depth evaluation protocol used in StreetForward [37]. We project Waymo LiDAR points into the front camera and evaluate only pixels with valid sparse depth. Because several feed-forward baselines predict depth up to an arbitrary scale, we align each prediction with the LiDAR depth using a per-frame least-squares scale before reporting AbsRel and \(\delta_1\) for all valid LiDAR pixels and for the subset in dynamic regions.
As shown quantitatively in 1 and qualitatively in 3, our model achieves the best RGB reconstruction among the evaluated feed-forward baselines, with even larger margins in dynamic regions. Its depth quality is on par with that of the strongest baselines, if not better on some metrics. We attribute the improvements mainly to our choice of efficient backbones and more effective training strategies.
| Method | Full Image | Dynamic Region | ||||||
|---|---|---|---|---|---|---|---|---|
| 2-5 (lr)6-9 | PSNR \(\uparrow\) | SSIM \(\uparrow\) | AbsRel \(\downarrow\) | \(\delta_1 \uparrow\) | PSNR \(\uparrow\) | SSIM \(\uparrow\) | AbsRel \(\downarrow\) | \(\delta_1 \uparrow\) |
| DepthSplat [29] | 22.48 | 0.645 | 0.295 | 0.592 | 18.59 | 0.391 | 0.445 | 0.512 |
| STORM [33] | 21.88 | 0.752 | 0.123 | 0.870 | 19.89 | 0.556 | 0.178 | 0.789 |
| Depth-Anything-3 [63] | 20.30 | 0.557 | 0.434 | 0.313 | 17.59 | 0.250 | 0.467 | 0.354 |
| DGGT [35] | 26.25 | 0.805 | 0.135 | 0.841 | 21.76 | 0.652 | 0.249 | 0.689 |
| Ours | 28.26 | 0.859 | 0.076 | 0.937 | 24.93 | 0.793 | 0.085 | 0.922 |
10pt
5.2 Comparisons and Applications on Internal Data↩︎
Public driving datasets are relatively small and cover only a limited range of conditions. To demonstrate real-world applicability, we evaluate on a held-out validation split of our internal corpus and assess reconstruction quality and simulator integration in settings that public benchmarks rarely capture, including non-pinhole cameras, longer sequences (up to 30 s), and complex driving scenarios. Throughout this section, our baseline is NVIDIA NuRec [1], [17], a per-scene-optimized 3DGS reconstruction that requires roughly \(75\) min of optimization per scene and serves as our performance reference for production-quality reconstruction.
Reconstruction quality. 4 shows single-camera reconstructions in which re-posed and bird’s-eye views remain geometrically consistent and preserve thin structures across diverse scenes. Instant NuRec also reconstructs the full multi-camera surround view in a single forward pass: 5 shows the result across all five cameras (front, front-left, front-right, rear-left, rear-right), with consistent appearance and geometry across the rig in scenes ranging from tunnels to dense urban streets.
As described in 3, Instant NuRec provides two query-point strategies, Dense and Selective, that trade Gaussian budget for reconstruction quality. On our validation set, the Selective strategy cuts the Gaussian budget by roughly \(3\times\)—from \(\sim\)351K per context view for Dense to \(\sim\)120K (\(\sim\)6.3M to \(\sim\)2.2M per 18-frame scene)—with only marginal losses in image and depth accuracy, as shown in 6. 7 further compares both variants with NuRec: both reconstruct a scene in \(\sim\)1.5 s versus \(\sim\)75 min for NuRec, a speedup of more than three orders of magnitude, while trailing only modestly in appearance quality.
We further evaluate reconstruction quality by applying an in-house 3D detection pipeline to the reconstructed scenes. We render each scene from novel viewpoints and compare the resulting detection precision and recall with those obtained from ground-truth images captured by the ego cameras. This evaluation checks that reconstruction inaccuracies do not bias downstream policies. Dense remains the stronger reconstruction variant, while Selective incurs a modest reduction in appearance and detection metrics while using roughly one-third of the Gaussian budget. Selective is therefore useful when memory or rendering cost is the primary constraint, while Dense provides the highest-quality feed-forward reconstruction.
Closed-loop simulation viability. A key question for any feed-forward reconstruction method is whether the reconstruction produced by a single forward pass is sufficient for reliable closed-loop policy simulation. To test this, we compare Instant NuRec with NuRec by running the same set of policies through the same AlpaSim [3] closed-loop stack and changing only the reconstructions. Our evaluation set contains 140 scenes. Each scene is rolled out for 20 s in closed loop, with policy replanning every 500 ms and six independent trials per scene. We simulate five driving policy configurations: VaVAM [73], Alpamayo R1 [74] (A-R1), and the 1-camera, 2-camera, and 4-camera variants of Alpamayo 1.5 [74] (A-1.5). We evaluate the policies using the average of two key metrics: Collision (Front + Lateral + Rear) and Offroad. As shown in 8, the policy ranking under Instant NuRec is identical to that under NuRec, meaning that policy selection decisions made with Instant NuRec would match those made with the more expensive per-scene reconstruction. This makes the method a reliable substitute for per-scene optimization in closed-loop policy evaluation, at a fraction of the reconstruction cost.
5.3 Ablation Study↩︎
We ablate the main design choices of Instant NuRec on the validation set introduced in 5.2. We summarize the results quantitatively in 2 and qualitatively in 9; each variant removes one component to show its individual effect on reconstruction quality.
Training strategy. We implement a single-stage variant that trains Stage 2 and Stage 3 jointly rather than following our sequential schedule. This variant takes longer to train and requires more iterations to converge, possibly because of the additional complexity introduced by gradients from the renderer. Nevertheless, its final metrics remain worse (2), so we retain the multi-stage schedule.
Sky model. We replace the sky cubemap branch with an MLP sky model similar to STORM [33]. Because of the network’s low-frequency inductive bias, this variant fails to recover high-frequency sky textures such as clouds and the sun (9).
Loss terms. We ablate the two auxiliary losses in turn. Removing the LPIPS loss makes the reconstructions noticeably blurrier, while removing the depth loss causes the model to overfit to appearance and degrade geometry, as reflected in the quantitative depth metrics.
Primitive merging strategy. We replace frustum-ownership merging with a naive concatenation of Gaussians without pruning. Without this pruning, far-field Gaussians from the outer region of one chunk can appear as floaters inside neighboring chunks, where they compete with better-conditioned local primitives and degrade quality, especially for longer sequences.
| Variant | PSNR \(\uparrow\) | SSIM \(\uparrow\) | AbsRel \(\downarrow\) | \(\delta_1 \uparrow\) |
|---|---|---|---|---|
| (full) | 29.93 | 0.793 | 0.069 | 0.950 |
| single-stage training | 27.65 | 0.751 | 0.077 | 0.955 |
| sky MLP instead of cubemap | 26.73 | 0.692 | 0.068 | 0.950 |
| w/o LPIPS loss | 26.81 | 0.705 | 0.073 | 0.934 |
| w/o depth loss | 28.92 | 0.782 | 0.103 | 0.862 |
| w/o frustum-ownership merging | 26.10 | 0.648 | 0.098 | 0.891 |
24pt
5.4 Extension to LiDAR Reconstruction↩︎
Beyond image reconstruction, we examine whether Instant NuRec can be extended to the LiDAR modality with only minimal changes.
Model adaptation. We retain the pretrained Instant NuRec backbone and add a lightweight LiDAR branch that mirrors the camera path and predicts Gaussian primitives in a single forward pass. Each LiDAR sweep is represented as a range map with per-beam range and intensity; a range-map encoder produces tokens, and a decoder maps them to \(M\) Gaussian primitives for each valid input ray. These primitives are instantiated only for valid returns and are anchored near the corresponding measurements with bounded 3D offsets; the branch also predicts their scales, orientations, opacities, and intensities. To respect the sparse angular sampling of LiDAR, we impose an anisotropic scale floor inspired by Mip-Splatting [75]. Specifically, at range \(r\), each primitive’s lateral footprint is lower-bounded by the local inter-beam spacing induced by the horizontal and vertical angular resolutions. The branch is trained by rendering range and intensity for supervision frames captured from different poses, using an \(L_1\) range loss and an MSE intensity loss.
Dataset and evaluation. We evaluate on a custom split of our internal data collected with a Pandar128 LiDAR. Ground truth consists of the valid returns in the recorded LiDAR point cloud. Geometry quality is measured by the Chamfer distance between the predicted and ground-truth geometry. We also report the per-return intensity MAE on rays for which both the filtered prediction and ground truth contain a return.
| Method | CD \(\downarrow\) | Prec. \(\downarrow\) | Cov. \(\downarrow\) | Int. MAE \(\downarrow\) | Recon. time \(\downarrow\) | Speedup \(\uparrow\) |
|---|---|---|---|---|---|---|
| NuRec [1] | 0.204 | 0.080 | 0.124 | 0.028 | \(\sim\)75 min | 1\(\times\) |
| LiDAR | 0.286 | 0.113 | 0.173 | 0.027 | \(\sim\)20 s | \(\sim\)225\(\times\) |
| w/o scale floor | 0.936 | 0.195 | 0.741 | 0.036 | – | – |
15pt
3 compares Instant NuRec LiDAR with the per-scene-optimized NuRec. Compared with NuRec, our method has higher CD, mainly because of coverage rather than precision, indicating well-localized but less complete geometry. Intensity prediction remains comparable. The ablation without the anisotropic scale floor confirms its importance for LiDAR reconstruction; otherwise, Gaussian primitives collapse around observed rays, producing fragmented and incomplete geometry.
Despite the quantitative gap, Instant NuRec LiDAR preserves the dominant scene geometry (10): it reconstructs the main road layout, nearby structures, and scene boundaries without evident distortion. To obtain the best quality, we increase the number of context LiDAR frames, forming roughly 20 windows per clip. Although this increases the reconstruction time from \(1\) s to \(\sim\)20 s, the method remains orders of magnitude faster than NuRec. Overall, we conclude that Instant NuRec can reconstruct LiDAR geometry with qualitatively comparable results despite incomplete coverage, while remaining orders of magnitude faster.
6 Discussion and Conclusion↩︎
Limitations. Instant NuRec faces an inherent tension between the number of Gaussians and reconstruction quality: matching the quality of per-scene optimization can require many primitives, while a smaller budget under-samples thin structures. Test-time training that refines the predicted Gaussians on the input clip is a promising way to close this gap. Generalization is also limited by the training corpus, so rigs that differ substantially from those seen in training (e.g., low-mounted or fisheye-only setups) currently require fine-tuning. Finally, the three-keyframe piecewise-linear actor trajectories cannot capture sub-second non-rigid motion, such as pedestrian articulation. Denser keyframes could address this limitation. Separately, a streaming formulation in which past reconstructions condition the current forward pass is a promising direction for scaling to long logs.
Conclusion. Instant NuRec demonstrates that the cost of high-quality driving-scene reconstruction can be amortized into a single feed-forward pass. It turns a short multi-view log into a layered, fully simulatable 3DGS world consisting of static and dynamic Gaussians, a sky cubemap, and per-camera ISP corrections, directly consumable by NuRec and AlpaSim. By approaching per-scene quality at a cost that is orders of magnitude lower, we hope that Instant NuRec will lower the barrier to using neural reconstruction at the fleet scale that AD development now demands.
7 Contributors↩︎
Research. Jiahui Huang, Jiawei Ren, Michal Tyszkiewicz, Xin Kang, Seung Wook Kim, Shengyu Huang, Laura Leal-Taixe\(^\ddagger\), Zan Gojcic\(^\ddagger\), Sanja Fidler\(^\ddagger\).
Engineering. Bjoern Haefner, Michael Shelley, Ning Xu, Qi Wu, Janick Martinez Esturo, Nick Schneider\(^\ddagger\).
8 Acknowledgments↩︎
We greatly appreciate the contributions of the following individuals:
Alessandro Burzio, Alex Perec, Bingxin Ke, Daniel Dworakowski, Despoina Paschalidou, Emmanuel Attia, Jun Gao, Katarina Tothova, Lei Zhang, Lucrezia Shen, Murat Arar, Naveen Kumar Rai, Nicolas Moenne-Loccoz, Rodolfo Lima, Sangeetha Grama Srinivasan, Sean Pieper, Sergio Agostinho, Sherwin Bahmani, Shikhar Solanki, Sipeng Zhang, Tianchang Shen, Tobias Fischer, Weihua Zhang, Xuanchi Ren, Yixin Cao.