1 Introduction↩︎

Embodied AI agents and autonomous systems increasingly require accurate and detailed 3D understanding of their surroundings [3], which forms the foundation for reasoning, planning, and interaction in complex environments. Recent advances in vision-based perception have significantly improved 3D scene understanding by exploiting rich semantic and geometric cues [4]–[8]. Among various scene representations, 3D occupancy [1], [2] has emerged as a powerful paradigm by jointly modeling foreground objects, background structures, and free space within a unified volumetric representation, making it a fundamental component for downstream tasks such as robotic navigation [9], manipulation [10], and autonomous driving [11].

Although vision-based occupancy estimation has been extensively investigated in autonomous driving [12]–[18], fine-grained occupancy prediction in indoor environments remains relatively underexplored. Compared with outdoor driving scenes, indoor environments exhibit highly cluttered spatial layouts, substantial appearance variability, and a broader diversity of object categories, posing additional challenges for accurate volumetric reasoning. Recent approaches such as ISO [1] and EmbodiedOcc [2] have demonstrated promising results by leveraging depth priors [19], [20]. Specifically, ISO lifts image features into dense volumetric representations through estimated depth distributions and performs occupancy reasoning using a 3D U-Net. EmbodiedOcc, in contrast, initializes Gaussian primitives and iteratively refines them through cross-attention between projected 3D anchors and image features before splatting the refined Gaussians into occupancy space. Despite their effectiveness, these methods exploit only limited geometric cues [21] and often devote substantial representational capacity to large empty regions, leading to reduced efficiency and constrained generalization.

Meanwhile, recent progress in visual geometry priors has introduced powerful geometric representations, ranging from monocular depth foundation models such as the DepthAnything family [19], [20] to multi-view Visual Geometry Models (VGMs) [22]–[27]. These models provide rich geometric information, including depth maps, point maps, and camera parameters, enabling high-quality 3D reconstruction across diverse environments. However, such representations are inherently surface-centric: each image pixel typically corresponds to a single visible surface point, whereas occupancy prediction requires reasoning over occupied and free space throughout a 3D volume, including occluded and unobserved regions. This discrepancy raises a fundamental question: how can surface-centric visual geometry priors be transformed into volumetric scene representations suitable for occupancy reasoning?

To address this challenge, we proposed GPOcc, a framework that combines generalizable Geometry Priors (GPs) with sparse Gaussian representations for fine-grained 3D occupancy prediction [28]. Sparse Gaussians provide continuous spatial support around geometry-prior points while maintaining a compact representation, avoiding the unnecessary computation introduced by densely modeling empty space. Their probabilistic formulation also enables differentiable aggregation and naturally accommodates uncertainty in the inferred volumetric support. Our approach builds on four key components. (1) To overcome the surface-only nature of geometry priors, a ray-based volumetric sampling strategy extends geometry-prior points along camera rays to generate volumetric anchors, as illustrated in 1 (c). Each anchor predicts a Gaussian primitive that models its local spatial neighborhood. (2) An opacity-based pruning mechanism removes low-contribution Gaussians, substantially reducing redundancy while preserving informative volumetric support. (3) Occupancy is estimated from the remaining sparse Gaussian set through a probabilistic formulation following [18], transforming continuous Gaussian support into discrete occupancy predictions. (4) For embodied applications with streaming observations, a training-free incremental update strategy progressively fuses per-frame Gaussian representations into a global scene representation. Together, these components establish a principled transformation from surface-centric geometry priors to volumetric representations.

While GPOcc demonstrates the effectiveness of geometry priors for occupancy estimation, several limitations remain. First, it processes geometry priors largely in a frame-wise manner and therefore cannot explicitly exploit complementary temporal or multi-view observations to resolve ambiguities caused by occlusion, missing evidence, and limited fields of view. Second, its fixed ray-sampled anchors provide limited flexibility for modeling irregular shapes, thin structures, object boundaries, and local geometry errors. Third, GPOcc has mainly been validated in indoor embodied environments, leaving its generalization to large-scale outdoor autonomous-driving scenarios insufficiently explored.

Motivated by these observations, we further propose GPOcc++, a substantially enhanced framework that extends GPOcc in both capability and applicability. GPOcc++ introduces two key technical advances while broadening geometry-prior-based occupancy prediction from indoor embodied environments to large-scale autonomous-driving scenarios. First, we propose Ray-Conditioned Multi-Image Fusion, which performs geometry-aware fusion across observations acquired from different viewpoints and timestamps within a unified representation space. By explicitly modeling camera-ray geometry, the proposed module exploits complementary observations to reduce ambiguities in single-view geometry priors while naturally supporting single-frame, temporal, and multi-view inputs. Second, we introduce Offset-Guided Ray Anchoring, which predicts explicit 3D offsets for ray-sampled anchors, allowing Gaussian primitives to adaptively align with complex scene structures while preserving the geometric inductive bias of camera rays. Together, these advances enable more accurate and flexible volumetric scene representations across diverse environments. Extensive experiments on indoor embodied-scene and outdoor autonomous-driving benchmarks demonstrate that GPOcc++ consistently achieves state-of-the-art or highly competitive performance under single-frame, temporal, and multi-view settings while maintaining favorable computational efficiency. These results demonstrate the effectiveness and generality of the proposed geometry-prior-based surface-to-volume transformation across diverse environments.

Compared with GPOcc, the additional contributions of this work are summarized as follows:

1. We propose GPOcc++, a geometry-prior-based framework for 3D occupancy prediction that extends surface-to-volume transformation from frame-wise processing to unified temporal and multi-view reasoning.

2. We introduce Ray-Conditioned Multi-Image Fusion, which explicitly models cross-view and cross-temporal geometric relationships and exploits complementary observations to alleviate ambiguities in single-image geometry priors.

3. We introduce Offset-Guided Ray Anchoring, which predicts explicit 3D residuals for ray-sampled anchors, enabling Gaussian centers to better align with complex scene structures.

4. We extend geometry-prior-based occupancy prediction from embodied environments to autonomous-driving scenarios and conduct extensive experiments demonstrating state-of-the-art or highly competitive performance with favorable efficiency.

This article builds upon our previous conference paper [28] published at CVPR 2026. Compared with the conference version, the present work introduces Ray-Conditioned Multi-Image Fusion for unified multi-view and multi-frame reasoning, Offset-Guided Ray Anchoring for more flexible geometric modeling, and comprehensive validation across both indoor and outdoor benchmarks, accompanied by substantially expanded experimental analyses.

2 Related Work↩︎

2.1 Visual Geometry Foundation Models↩︎

Earlier geometry learning methods have explored generalizable monocular depth [29] estimation and ray-based [30] multi-view reconstruction. Despite their strong geometric reasoning capabilities, these methods primarily recover depth or visible surface geometry and do not directly construct volumetric representations required by occupancy prediction. More recently, visual geometry foundation models have learned more unified geometric representations from large-scale data. DUSt3R [23] and MASt3R [24] predict coupled scene representations, including camera poses and geometry parameterized by pointmaps, from image pairs. However, they still require expensive post-processing or symmetric inference for unconstrained multi-view SfM. Subsequent works including Spann3R [31], CUT3R [32], and MUSt3R [33] reduce the reliance on classical optimization by introducing latent-state memory in transformers, enabling more end-to-end multi-view reconstruction. Fast3R [25] further scales this paradigm to efficiently handle more than 1000 input images. Building on this line, VGGT [22] jointly predicts pointmaps, depth, camera poses, and tracking features with minimal hand-crafted 3D inductive biases. StreamVGGT [34] reformulates VGGT with a causal transformer for efficient long-sequence processing. Several variants have also been developed on top of VGGT: \(\pi^3\) [26] removes the dependence on the first input frame as the reference coordinate system, while Dens3R [35] enriches geometric predictions with surface normals. These models provide increasingly strong priors for downstream 3D understanding. Nevertheless, their outputs remain largely surface-centric, typically describing one visible surface point per image pixel. In contrast, our work studies how such general visual geometry priors can be transformed into sparse volumetric representations that model both visible surfaces and latent occupancy support.

2.2 Occupancy Prediction↩︎

MonoScene [36] extends semantic scene completion [36]–[38] to monocular 3D occupancy prediction. Earlier semantic scene completion methods have explored anisotropic 3D convolutions [39], local implicit functions [40], and semantic reconstruction from posed RGB images [41]. These approaches demonstrate the importance of jointly recovering scene geometry and semantics, but typically rely on task-specific voxel, LiDAR, or implicit representations rather than general visual geometry priors. Occupancy prediction has subsequently been extensively studied in autonomous-driving scenarios [12], [17], [18], [42], [43], with multi-camera BEV perception providing an important foundation for spatial reasoning [44]. Recent studies further extend occupancy beyond a single perception task: OccScene [45] uses semantic occupancy to facilitate cross-task scene generation, while SPOT [46] employs occupancy prediction as a scalable pre-training objective for transferable 3D representations. Large-scale datasets such as OmniHD-Scenes [47] further support multimodal autonomous-driving perception, and occupancy-like scene predictions have also been incorporated into integrated driving planning systems [48]. In comparison, research on fine-grained indoor occupancy remains relatively limited [1], [2], [49]–[51]. Existing occupancy methods adopt different scene representations and lifting strategies. Grid-based approaches lift 2D image features into dense volumetric spaces using depth distributions or ray-based transformations, followed by 3D convolutions or volumetric decoders [1], [36], [37], [52], [53]. Transformer-based architectures have also been introduced for volumetric representation learning [15]. Other methods inject depth or signed-distance cues into dense 3D grids [38], while tri-plane representations reduce the computational cost of full voxel volumes [14]. Beyond fixed grids, point- and Gaussian-based formulations initialize 3D anchors or primitives and refine them through iterative feature aggregation [2], [16], [17]. Sparsity has also been exploited by pruning dense voxel features and subsequently processing the retained regions using sparse convolutions [43] or transformers [12], [54]. However, these methods generally construct their 3D representations from task-specific image features, predefined anchors, or dense intermediate volumes.

In contrast, our framework directly transforms surface-centric geometry priors into sparse Gaussian volumetric representations. It explicitly models cross-view and cross-temporal relationships via ray-conditioned feature fusion to exploit complementary observations and resolve ambiguities caused by occlusion and limited visibility. Furthermore, learnable 3D offsets enable ray-initialized Gaussian anchors to adapt to complex scene structures, providing a unified formulation for both indoor temporal and outdoor multi-camera inputs.

3 Preliminaries↩︎

Occupancy Prediction. Given one or more RGB images \(\{\mathbf{I}_t\}_{t=1}^{T}\) with corresponding camera parameters, the goal is to predict a voxel-wise semantic occupancy map \(\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c}\), where \(X\), \(Y\), and \(Z\) denote the spatial resolution of the scene and \(N_c\) is the number of semantic classes. The case \(T=1\) corresponds to monocular occupancy prediction, while \(T>1\) represents temporal or multi-view inputs.

Gaussian Representation. A 3D scene can be represented by a set of semantic Gaussian primitives \(\mathbf{G}=\{\mathcal{G}_i\}_{i=1}^{P}\). Each primitive \(\mathcal{G}_i\) models a local spatial region centered at \(\mu_i\) and is parameterized by scale \(s_i\), rotation \(r_i\), opacity \(a_i\), and semantic feature \(c_i\). This representation provides continuous spatial support while compactly encoding geometry and semantics.

Gaussian-to-Occupancy. Occupancy is obtained by aggregating the contributions of nearby Gaussian primitives for each voxel \(p\) [2], [17], [18]: \[\label{eq:gs95occ} \hat{o}(p;\mathbf{G}) = \sum_{i\in\mathcal{N}(p)} g_i(p;\mu_i,s_i,r_i,a_i,c_i),\tag{1}\] where \(\mathcal{N}(p)\) denotes the set of primitives influencing voxel \(p\).

4 GPOcc↩︎

4.1 Overview↩︎

As illustrated in 2, GPOcc uses a visual geometry prior to extract 3D-aware image features and surface-level geometric predictions. Ray-based volumetric sampling extends these visible surface points into volumetric anchors, from which semantic Gaussian primitives are predicted. Low-contribution primitives are removed through opacity pruning, and the remaining sparse Gaussians are probabilistically mapped to occupancy. For streaming observations, per-frame Gaussian predictions are further integrated into a global memory through a training-free incremental update strategy.

4.2 Ray-based Volumetric Sampling↩︎

Visual geometry models [20], [22], [25], [27] provide strong priors by predicting depth or point maps. However, these predictions primarily describe visible surfaces, whereas occupancy reasoning also requires volumetric interiors and object thickness. We therefore extend surface predictions along their corresponding camera rays to construct volumetric anchors.

Given an RGB image \(\mathbf{I}\in\mathbb{R}^{H\times W\times3}\), a geometry-prior model extracts an image feature map \(\mathbf{F}\in\mathbb{R}^{H\times W\times C}\). To reduce computation, we downsample the intermediate features and regress a lower-resolution depth map: \[\mathbf{F}=\operatorname{GP}(\mathbf{I}), \mathbf{F}^{1/4}=\operatorname{DownSample}(\mathbf{F}), \mathbf{d}=\operatorname{MLP}_{\mathrm{depth}}(\mathbf{F}^{1/4})\] where \(\mathbf{F}^{1/4} \in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C}\) and \(\mathbf{d}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}}\).

For a pixel \((u,v)\), its normalized camera-ray direction is computed from the camera intrinsics: \[x=\frac{u-c_x}{f_x}, \qquad y=\frac{v-c_y}{f_y}, \qquad \mathbf{r}_{(u,v)} = \frac{[x,y,1]^\top}{\sqrt{x^2+y^2+1}},\] where \((c_x,c_y)\) and \((f_x,f_y)\) denote the principal point and focal lengths, respectively. The surface point predicted at \((u,v)\) is \[\mathbf{x}^{\mathrm{surf}}_{(u,v)} = \mathbf{d}_{(u,v)}\mathbf{r}_{(u,v)}.\] To approximate volumetric support beyond the visible surface, we sample \(K\) anchors along the same camera ray: \[\mathbf{x}_{(u,v,k)} = \bigl(\mathbf{d}_{(u,v)}+\delta_{(u,v,k)}\bigr) \mathbf{r}_{(u,v)}, \qquad k=1,\ldots,K,\] where the ray offsets are defined as \[\label{eq:learnable95scale} \{\delta_{(u,v,k)}\}_{k=1}^{K} = \operatorname{linspace}(0,1,K) \cdot \operatorname{scale} \bigl(\mathbf{F}^{1/4}_{(u,v)}\bigr).\tag{2}\] Here, \(\operatorname{scale}(\cdot)\) is a lightweight predictor that adapts the ray extent to the local image and geometry features.

To distinguish the \(K\) anchors associated with the same image location, we introduce a learnable embedding matrix \(\mathbf{E}\in\mathbb{R}^{K\times C}\). The anchor-wise features are constructed through broadcast addition: \[\hat{\mathbf{F}}^{1/4} = \mathbf{F}^{1/4}\oplus\mathbf{E}, \qquad \hat{\mathbf{F}}^{1/4} \in \mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times K\times C},\] where \(\oplus\) denotes broadcast addition. Gaussian attributes are then predicted for each anchor: \[\{s_i,r_i,a_i,c_i\} = \operatorname{MLP}_{\mathrm{gs}} \bigl(\hat{\mathbf{F}}^{1/4}\bigr),\] where \(i=1,\ldots,\frac{H}{4}\times\frac{W}{4}\times K\), and the Gaussian center is initialized as the corresponding ray-sampled anchor \(\mathbf{x}_{(u,v,k)}\).

By extending surface geometry along camera rays, GPOcc constructs volumetric support without introducing dense 3D anchors [2] or lifting all image features into a dense voxel volume [1].

4.3 From Sparse Gaussians to Occupancy↩︎

Prior Gaussian occupancy methods [2] initialize a dense set of predefined 3D anchors and classify them as occupied or empty. Since most of the scene volume is unoccupied, many primitives are consequently assigned to empty regions, as shown in 3 (a). In contrast, ray-based volumetric sampling places Gaussian primitives near the predicted surface and interior regions, yielding a sparse distribution shown in 3 (b). We adopt the probabilistic Gaussian superposition formulation of GaussianFormer-2 [18]. For a query location \(p\), the spatial contribution of Gaussian \(\mathcal{G}_i\) is defined by \[o(p;\mathcal{G}_i) = \exp\left( -\frac{1}{2} (p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i) \right),\] where \(\Sigma_i\) is constructed from the predicted scale \(s_i\) and rotation \(r_i\). Contributions from neighboring primitives are combined according to 1 to obtain voxel-wise semantic occupancy. The Gaussian kernels define a continuous volumetric field whose spatial support is controlled by the predicted covariance. Under voxel-wise occupancy supervision, the Gaussian scales are optimized to match the corresponding object extent while leaving unsupported regions empty. To further reduce redundancy, we discard primitives whose opacity satisfies \(a_i<\tau\). We use \(\tau=0.01\) by default, retaining only informative Gaussians for occupancy aggregation.

4.4 Incremental Sparse Gaussian Update↩︎

For embodied agents receiving streaming observations, GPOcc maintains a global Gaussian memory that progressively integrates per-frame predictions. Unlike methods that repeatedly update a dense predefined anchor set [2], our strategy directly accumulates sparse and adaptively generated primitives without additional training.

Let \(\mathcal{M}\) denote the global Gaussian memory and \(\mathbf{G}^{t}=\{\mathcal{G}^{t}_j\}_{j=1}^{P_t}\) the Gaussian set predicted from frame \(\mathbf{I}_t\). Using the known camera pose, all Gaussians in \(\mathbf{G}^{t}\) are first transformed from the camera coordinate system into the global coordinate system.

For each memory Gaussian \(\mathcal{G}_i\in\mathcal{M}\), we identify its newly observed neighbors \(\mathcal{N}(\mathcal{G}_i)\subseteq\mathbf{G}^{t}\) within a spatial radius \(\epsilon\). If neighboring primitives exist, their attributes are fused through confidence-weighted averaging: \[\theta_i \leftarrow \frac{ \gamma p_i\theta_i +(1-\gamma) \sum_{\mathcal{G}^{t}_j\in\mathcal{N}(\mathcal{G}_i)} p_j\theta_j }{ \gamma p_i +(1-\gamma) \sum_{\mathcal{G}^{t}_j\in\mathcal{N}(\mathcal{G}_i)} p_j },\] where \(\theta\in\{\mu,\Sigma,a,c\}\) denotes the Gaussian mean, covariance, opacity, or semantic feature, and \(p\) is the top-1 semantic confidence. The parameter \(\gamma\in(0,1)\) balances historical and newly observed information, and we set \(\gamma<0.5\) to favor recent observations. After the update, new Gaussians that have not been assigned to any memory neighborhood are directly inserted into \(\mathcal{M}\). This strategy provides training-free, confidence-aware accumulation of sparse Gaussian representations over streaming observations.

4.5 Training Objectives↩︎

We optimize both GPOcc and GPOcc++ using a composite objective that combines semantic occupancy and geometric supervision. Following EmbodiedOcc [2], we use focal loss \(L_{\mathrm{focal}}\), Lovász-Softmax loss \(L_{\mathrm{lov}}\), and geometric and semantic scene-class affinity losses \(L^{\mathrm{geo}}_{\mathrm{scal}}\) and \(L^{\mathrm{sem}}_{\mathrm{scal}}\). We additionally apply a Huber loss \(L_{\mathrm{depth}}\) to the predicted depth, enabling end-to-end optimization of the geometry-prior adaptation and occupancy prediction modules: \[\label{eq:loss} \begin{align} \mathcal{L} ={}& L_{\mathrm{focal}} \bigl( Y_{\mathrm{pred}}^{\mathrm{fov}}, Y_{\mathrm{gt}}^{\mathrm{fov}} \bigr) + L_{\mathrm{lov}} \bigl( Y_{\mathrm{pred}}^{\mathrm{fov}}, Y_{\mathrm{gt}}^{\mathrm{fov}} \bigr) \\ &+ L^{\mathrm{geo}}_{\mathrm{scal}} \bigl( Y_{\mathrm{pred}}^{\mathrm{fov}}, Y_{\mathrm{gt}}^{\mathrm{fov}} \bigr) + L^{\mathrm{sem}}_{\mathrm{scal}} \bigl( Y_{\mathrm{pred}}^{\mathrm{fov}}, Y_{\mathrm{gt}}^{\mathrm{fov}} \bigr) + L_{\mathrm{depth}}. \end{align}\tag{3}\]

5 GPOcc++↩︎

5.1 Overview↩︎

Although GPOcc can use geometry priors produced from multiple images, it processes their features largely in a frame-wise manner and constrains Gaussian centers to their initial ray-sampled locations. As illustrated in 4, GPOcc++ addresses these limitations with two extensions. Ray-Conditioned Multi-Image Fusion explicitly models cross-view and cross-temporal feature relationships using camera-ray geometry, while Offset-Guided Ray Anchoring refines the initial volumetric anchors with learnable 3D residuals. The subsequent Gaussian prediction, opacity pruning, occupancy aggregation, and incremental update follow the same sparse formulation as GPOcc.

5.2 Ray-Conditioned Multi-Image Fusion↩︎

Multiple observations provide complementary geometric evidence in both embodied perception and autonomous driving. Temporal frames reveal scene regions over continuous exploration, while synchronized cameras cover different spatial viewpoints [55], [56]. However, independently processing these images cannot explicitly model their geometric relationships. Let the input image set be \(\{\mathbf{I}_t\}_{t=1}^{T}\), where \(T=1\) denotes the single-image case, and \(T>1\) represents either temporally adjacent frames or synchronized multi-view images. For each image, the geometry-prior model extracts a feature map \(\mathbf{F}_t\in\mathbb{R}^{H\times W\times C}\) together with its surface geometry prediction.

We introduce a lightweight interaction module that alternates between local and global attention [22]. Local attention first models intra-image spatial relationships: \[\bar{\mathbf{F}}_t = \operatorname{LocalAttn}(\mathbf{F}_t).\] The resulting tokens from all observations are then jointly processed by global attention: \[\tilde{\mathbf{F}} = \operatorname{GlobalAttn} \left( [ \bar{\mathbf{F}}_1+\mathbf{E}^{\mathrm{cam}}_1; \ldots; \bar{\mathbf{F}}_T+\mathbf{E}^{\mathrm{cam}}_T ] \right),\] where \([\cdot;\cdot]\) denotes token concatenation and \(\mathbf{E}^{\mathrm{cam}}_t\) encodes the camera-ray geometry of image \(t\).

For temporal observations, global attention uses a causal mask [34], allowing each frame to attend only to its current and preceding observations. During inference, historical tokens are retained in a cache [57], so only newly observed frames need to be processed. To explicitly encode camera geometry, we represent each image ray using Plücker coordinates [58], [59]. For pixel \((u,v)\) in image \(t\), let \(\mathbf{o}_t\) be the camera center and \(\mathbf{r}_{t,(u,v)}\) the normalized ray direction in the global coordinate system. The ray representation is \[\mathbf{p}_{t,(u,v)} = \left[ \mathbf{r}_{t,(u,v)}, \mathbf{o}_t\times\mathbf{r}_{t,(u,v)} \right],\] and is projected into the image feature space as \[\mathbf{E}^{\mathrm{cam}}_{t,(u,v)} = \operatorname{MLP}_{\mathrm{cam}} \left(\mathbf{p}_{t,(u,v)}\right).\] Adding these embeddings before global attention enables the model to jointly reason over appearance, camera configuration, and scene geometry. The module outputs fused feature maps \(\{\tilde{\mathbf{F}}_t\}_{t=1}^{T}\) for subsequent anchor refinement and Gaussian prediction, while explicitly modeling cross-view and cross-temporal relationships.

5.3 Offset-Guided Ray Anchoring↩︎

The ray-based anchors in GPOcc are constrained to fixed trajectories determined by the predicted depth and radial sampling offsets. Although this provides a strong geometric initialization, the optimal volumetric support may deviate from these initial positions for thin structures, irregular boundaries, partially occluded regions, or locally inaccurate geometry predictions. We therefore treat the initial ray samples as coarse anchors and predict a 3D residual for each anchor. For image \(t\), the initial anchor associated with pixel \((u,v)\) and sample index \(k\) is \[\mathbf{x}^{0}_{t,u,v,k} = \mathbf{o}_t + \bigl( \mathbf{d}_{t,u,v} + \delta_{t,u,v,k} \bigr) \mathbf{r}_{t,u,v}.\]

The anchor query combines the corresponding fused image feature with the learnable sampling-index embedding: \[\mathbf{q}_{t,u,v,k} = \tilde{\mathbf{F}}^{1/4}_{t,u,v} + \mathbf{E}_k.\] For a single-image input, \(\tilde{\mathbf{F}}_t\) is replaced by the original geometry feature \(\mathbf{F}_t\). A 3D residual offset is predicted as \[\Delta\mathbf{x}_{t,u,v,k} = \operatorname{MLP}_{\mathrm{off}} \left(\mathbf{q}_{t,u,v,k}\right),\] and the refined Gaussian center becomes \[\mathbf{x}_{t,u,v,k} = \mathbf{x}^{0}_{t,u,v,k} + \Delta\mathbf{x}_{t,u,v,k}.\] The remaining Gaussian attributes are predicted from the same anchor query: \[\{s_i,r_i,a_i,c_i\} = \operatorname{MLP}_{\mathrm{gs}} \left(\mathbf{q}_{t,u,v,k}\right).\]

This design preserves the geometry-guided ray initialization while allowing Gaussian centers to correct local depth errors and adapt to complex scene structures. With multiple observations, the offsets are conditioned on fused cross-image features; with a single observation, the module performs monocular anchor refinement.

6 Experiments↩︎

0.008

0.007

0.005

In this section, we comprehensively evaluate GPOcc and its extended version, GPOcc++, across indoor monocular occupancy prediction, embodied scene-level occupancy prediction, and outdoor multi-view occupancy prediction. We compare them with representative state-of-the-art approaches on Occ-ScanNet, EmbodiedOcc-ScanNet, and nuScenes, and conduct ablation studies to validate the key designs of GPOcc and the additional components in GPOcc++. We further present model profiling and qualitative analyses to examine accuracy–efficiency trade-offs and reconstruction quality.

6.1 Datasets and Metrics↩︎

Occ-ScanNet [1] is a large-scale benchmark for monocular indoor occupancy prediction, containing 45,755 training samples and 19,764 testing samples. The dataset covers diverse scenes and viewpoints, and provides voxelized frames in \(60 \times 60 \times 36\) grids, corresponding to a \(4.8 \text{m} \times 4.8 \text{m} \times 2.88 \text{m}\) volume in front of the camera. Each voxel is annotated with 12 semantic classes, including 11 valid categories (ceiling, floor, wall, window, chair, bed, sofa, table, TV, furniture, objects) and one class for empty space.

EmbodiedOcc-ScanNet [2] is a reorganized version of Occ-ScanNet, consisting of 537 training scenes and 137 validation scenes. Each scene contains 30 posed frames, and the global occupancy resolution of a scene is defined as \(\frac{l_x \times l_y \times l_z}{(0.08\text{m})^3}\), where \(l_x \times l_y \times l_z\) denotes the spatial range of the scene in world coordinates.

nuScenes [63] is a large-scale benchmark for autonomous driving, containing 1000 surround-view driving scenes collected in Boston and Singapore. Following the official split, the dataset is divided into 700, 150, and 150 scenes for training, validation, and testing, respectively. Each scene is 20 seconds long and annotated at 2Hz, with synchronized sensor data from 6 cameras, 1 LiDAR, 5 radars, and 1 IMU. Following prior works [13], [18], [62], we use the 3D semantic occupancy annotations provided by SurroundOcc [13] for supervision and evaluation. The annotated occupancy space covers the range of \([-50, 50]\) m, \([-50, 50]\) m, and \([-5, 3]\) m along the \(x\), \(y\), and \(z\) axes, respectively. Each voxel has a side length of 0.5 m and is assigned one of 18 classes, including 16 semantic classes, one empty class, and one noise class.

Evaluation Metrics. We follow prior work and adopt mIoU and IoU as evaluation metrics. Specifically, for Occ-ScanNet [1], [2], we compute IoU between predictions and ground truth within the camera frustum of each frame. For EmbodiedOcc-ScanNet [2], we evaluate global occupancy by computing IoU at the scene level, where the entire reconstructed scene is considered. For nuScenes [63], we follow the standard protocol in prior occupancy prediction works [13], [18], [62] and report mIoU over semantic occupancy predictions in the predefined 3D voxel space.

6.2 Implementation Details↩︎

Indoor benchmarks. For the Occ-ScanNet benchmarks, we adopt a unified training strategy for all models. We use the AdamW optimizer [64] with a weight decay of 0.01. The learning rate is linearly warmed up during the first 1000 iterations to a maximum value of \(2\times10^{-4}\) and then decayed with a cosine schedule. The models are trained for 10 epochs with a total batch size of 8 on 4 NVIDIA A800 GPUs. Input images are resized such that the longer side is 518 pixels, following VGGT [22]. We apply gradient clipping with a maximum norm of 1.0. Unless otherwise specified, we set \(K=16\) for ray-based volumetric sampling, and \(\tau=0.01\) for opacity-based pruning.

Temporal training. For temporal indoor experiments, we adopt a two-stage training strategy. We first train a monocular GPOcc++ model equipped with the proposed temporal modules, where only the current frame is used as input. This stage provides a strong initialization while keeping the optimization close to the single-frame setting. We then use the resulting model as the pretrained model and continue training with temporal inputs. During temporal training, frames are processed in chronological order, and features from previous frames are cached and fed to the current frame, enabling causal multi-frame reasoning without accessing future observations.

Outdoor benchmark. For nuScenes, we adopt a separate training setup tailored to outdoor autonomous driving scenes following previous works [18], [62]. We use DepthAnything-V2-Base [20] as the backbone, since its latency is close to current state-of-the-art occupancy methods. We use the AdamW optimizer with a base learning rate of \(4\times10^{-4}\) and a weight decay of 0.01. The model is trained for 20 epochs with a total batch size of 8, and gradient clipping with a maximum norm of 1.0 is also applied. We use six surround-view camera images as input, and follow the standard nuScenes occupancy setting with a 3D range of \([-50,50]\times[-50,50]\times[-5,3]\) meters and a voxel size of 0.5 meters. Input images are resized such that the longer side is 518 pixels.

6.3 Occupancy Prediction Results↩︎

Results on Occ-ScanNet. As shown in ¿tbl:tab:main95mono?, bold numbers denote the best result and underlined numbers denote the second best. Overall, GPOcc++ (VGGT) achieves the best result with 63.72 IoU and 57.41 mIoU, surpassing EmbodiedOcc and EmbodiedOcc++ by +12.26 and +11.21 mIoU, respectively. Using DepthAnything as the geometry prior, same as [1], [2], Ours-DPT model already delivers consistent improvements across classes over previous methods and improves mIoU from 46.20 (EmbodiedOcc++) to 51.88. The GPOcc++ extension further raises DPT-based mIoU to 53.88. Furthermore, replacing DepthAnything with the stronger VGGT prior yields additional gains, showing that our framework generalizes well across different geometry priors and can fully benefit from more powerful foundation models.

Results on EmbodiedOcc-ScanNet. ¿tbl:tab:main95online? reports the embodied prediction performance on the EmbodiedOcc-ScanNet benchmark. Our base GPOcc already achieves strong performance in the embodied setting. With DepthAnything as the geometry prior, GPOcc (DPT) attains 56.39 IoU and 51.22 mIoU, outperforming EmbodiedOcc++ [49] by +4.19 IoU and +7.62 mIoU. Replacing DepthAnything with the stronger VGGT prior further improves the result to 61.41 IoU and 55.39 mIoU, showing that GPOcc can effectively exploit stronger geometry priors. With the proposed GPOcc++ extensions, the semantic occupancy quality is further improved. GPOcc++ (DPT) reaches 60.89 IoU and 56.41 mIoU, improving over GPOcc (DPT) by +4.50 IoU and +5.19 mIoU. GPOcc++ (VGGT) obtains the best results with 62.50 IoU and 57.41 mIoU, which is +10.30 IoU and +13.81 mIoU higher than EmbodiedOcc++ and +1.09 IoU and +2.02 mIoU higher than GPOcc (VGGT). These gains indicate that Ray-Conditioned Multi-Image Fusion and Offset-Guided Ray Anchoring improve both geometric completeness and semantic consistency in the global embodied scene. These results show that the proposed extensions are particularly beneficial for embodied occupancy prediction, where sequential observations must be fused into a coherent scene-level representation.

Results on nuScenes. 1 reports the semantic occupancy prediction results on nuScenes. Different from indoor embodied benchmarks, nuScenes focuses on large-scale outdoor driving scenes with surround-view camera inputs, long-range spatial layouts, dynamic traffic participants, and complex road structures. This setting introduces a distinct type of challenge, requiring the model to recover both global scene layout and fine-grained semantic geometry from multi-view observations. GPOcc achieves competitive performance with 31.92 IoU and 19.83 mIoU. GPOcc++ further improves the performance to 33.28 IoU and 20.78 mIoU, bringing gains of +1.36 IoU and +0.95 mIoU over GPOcc. Compared with prior Gaussian-based methods, GPOcc++ achieves stronger overall accuracy while preserving the efficiency and compactness of the sparse representation. The improvements are also reflected in several structure-sensitive categories, such as driveable surface, other flat, sidewalk, indicating that GPOcc++ recovers more complete scene layouts and richer semantic details in outdoor driving environments. These results show that our method can effectively extend from indoor monocular occupancy prediction to multi-camera semantic occupancy reconstruction in large-scale autonomous driving scenes.

6.4 GPOcc ablations↩︎

We first analyze the key design choices in GPOcc, including ray sampling and opacity pruning on Occ-ScanNet. Detailed sensitivity analyses of the sampling number \(K\) and opacity threshold \(\tau\), together with ablations on the incremental update strategy, are provided in the Appendix.

Effect of volumetric ray sampling and opacity pruning. As shown in 2, restricting each camera ray to a single surface sample substantially reduces the mIoU from 56.19 to 47.88. This result confirms that extending surface-centric geometry priors along camera rays is essential for recovering volumetric object support beyond the visible surfaces. Removing opacity pruning yields only modest improvements of 0.62 mIoU and 0.39 IoU, while retaining all low-opacity Gaussian primitives and consequently increasing representation redundancy. The full model therefore adopts opacity pruning to achieve a more favorable trade-off between occupancy accuracy and representation compactness.

6.5 GPOcc++ ablations↩︎

We evaluate the two key components of GPOcc++, namely Ray-Conditioned Multi-Image Fusion and Offset-Guided Ray Anchoring, on both indoor embodied benchmarks and outdoor autonomous driving datasets.

Effect of GPOcc++ components on Occ-ScanNet. 3 compares GPOcc and GPOcc++ under monocular and embodied settings. On Occ-ScanNet, Ray-Conditioned Multi-Image Fusion consistently improves performance, increasing mIoU from 51.88 to 52.26 with DPT and from 56.19 to 56.73 with VGGT. Incorporating Offset-Guided Ray Anchoring further improves performance to 53.88 and 57.41 mIoU, respectively. These results indicate that both components benefit even single-image occupancy prediction by enhancing cross-token interaction and improving geometric alignment.

Effect of GPOcc++ components on EmbodiedOcc-ScanNet. The improvements are more pronounced in embodied settings as shown in 3. With DPT, Ray-Conditioned Multi-Image Fusion improves IoU/mIoU from 56.39/51.22 to 58.72/54.48, and the full GPOcc++ further boosts performance to 60.89/56.41. With VGGT, the corresponding results improve from 61.41/55.39 to 61.98/56.43 and then to 62.50/57.41. This demonstrates that explicit modeling of inter-image relationships is particularly beneficial for embodied perception, where sequential observations must be aggregated into a coherent scene representation.

Effect of GPOcc++ components on nuScenes. 4 shows the corresponding GPOcc++ component ablation on nuScenes. Ray-Conditioned Multi-Image Fusion improves the baseline from 31.92 IoU and 19.83 mIoU to 32.04 IoU and 20.03 mIoU. The full GPOcc++ with both Ray-Conditioned Multi-Image Fusion and Offset-Guided Ray Anchoring further improves the result to 33.28 IoU and 20.78 mIoU, showing that the two modules provide complementary benefits in large-scale outdoor scenes.

Model profile. [tab:profile_occ_scannet,tab:profile_embodiedocc_scannet,tab:profile_nuscenes] summarize the accuracy, inference speed, and model size across indoor, embodied, and outdoor settings. DPT and VGGT refer to [20] and [22], respectively. FPS values are measured on the same NVIDIA A800 GPU and averaged over 1000 runs after 100 warm-up iterations. On Occ-ScanNet, GPOcc (DPT, Ours) achieves strong efficiency, running at 28.22 FPS with only 97.95M parameters, which is nearly \(3\times\) faster and less than half the size of EmbodiedOcc. GPOcc++ (DPT, Ours) still maintains 23.85 FPS with moderate parameter overhead, while VGGT variants further improve accuracy with a larger backbone.

On EmbodiedOcc-ScanNet, DPT-based variants remain compact and efficient, while VGGT-based variants provide higher accuracy. Compared with EmbodiedOcc, GPOcc (DPT, Ours) improves mIoU by 8.69 points and runs \(2.63\times\) faster with less than half of the parameters. GPOcc++ further improves both backbones, showing that its causal temporal modeling brings consistent gains with limited additional cost.

On nuScenes, GPOcc++ (Ours) runs at 8.02 FPS, faster than GaussianFormer-2 and QuadricFormer, while achieving the best overall accuracy. These results demonstrate a favorable trade-off between accuracy and efficiency across both indoor embodied perception and outdoor autonomous driving scenarios. For FPS interpretation on nuScenes, we note that our method uses images resized to a maximum side length of 518, while the compared methods follow the official \(900 \times 1600\) input resolution.

6.6 Qualitative Results↩︎

6.6.0.1 Indoor scenes

5 shows qualitative comparisons on indoor monocular occupancy prediction. GPOcc++ produces more complete and geometrically consistent occupancy predictions than EmbodiedOcc [2] and GPOcc, especially for large structures such as walls, floors, furniture, and shelves. The predicted Gaussian primitives provide an interpretable intermediate geometry, showing that the model learns compact 3D scene structure before decoding it into voxel occupancy. 6 presents qualitative results on streaming indoor inputs. With sequential observations, our incremental update strategy progressively integrates newly observed regions and refines incomplete predictions, leading to more complete and consistent occupancy reconstruction.

6.6.0.2 Outdoor scenes

7 shows qualitative results on outdoor driving scenes from nuScenes. Given surround-view camera images, GPOcc++ predicts semantic occupancy that is well aligned with the ground truth and preserves more complete road layouts, object-level structures, and fine-grained scene details than QuadricFormer. These results demonstrate that GPOcc++ can recover coherent semantic geometry in complex driving scenarios with long-range spatial structures.

More visualizations are provided in the Appendix.

7 Limitations↩︎

Although GPOcc++ shows strong generality across indoor and outdoor benchmarks, several limitations remain. First, the method can still be less accurate on large textureless or flat regions such as floors and roads, where geometry priors may provide weak local evidence and the sparse Gaussian representation may under-cover broad surfaces. Second, the incremental update strategy used for streaming embodied inputs continuously inserts new Gaussian primitives, which may increase memory and computation for long sequences. Third, the current formulation mainly focuses on static scenes; explicitly modeling dynamic objects and maintaining long-term maps remain important directions for future work.

8 Conclusion↩︎

We presented GPOcc++, a general and flexible framework for geometry-prior-based occupancy prediction across both embodied and autonomous driving scenarios. Building upon GPOcc, the proposed framework introduces Ray-Conditioned Multi-Image Fusion to explicitly model relations across temporal and multi-view observations with camera-ray geometry. In addition, Offset-Guided Ray Anchoring replaces the original fixed ray sampling scheme by directly predicting 3D residuals for sampled anchors, better aligning Gaussian centers with complex structures. Together with sparse Gaussian representation and incremental update for streaming inputs, our approach provides an effective and scalable solution for fine-grained 3D occupancy prediction. Extensive experiments on Occ-ScanNet, EmbodiedOcc-ScanNet, and nuScenes demonstrate that GPOcc++ achieves strong performance across diverse indoor and outdoor benchmarks, while retaining favorable efficiency and generality across different geometry priors. We hope this work can serve as a step toward more unified 3D scene understanding systems that effectively integrate strong visual geometry priors for a broad range of embodied and autonomous perception tasks.

Acknowledgment↩︎

This work was supported in part by the National Natural Science Foundation of China under Grant 62573370 and in part by the Education Department of Guangdong Province under Grant 2025ZDZX3051.

References↩︎