1 Introduction↩︎

Accurate perception of the surrounding environment is essential for autonomous driving systems [6]–[8]. In recent years, vision-based 3D perception algorithms have gained significant attention and advanced rapidly due to their cost-effectiveness. Typically, these algorithms first use a 2D encoder to extract latent features from images and then apply a view transformation method, such as lift-splat-shoot (LSS) [9], to lift the 2D features into a 3D voxel space using predicted depth information. This 3D scene representation pipeline serves as the basis for extracting geometric and semantic information that describes the driving environment, supporting various 3D perception tasks, such as 3D object detection [1], [10]–[17] and semantic segmentation [18]–[23]. To comprehensively understand dynamic scenes, 3D occupancy prediction [2], [5], [24]–[34] has emerged as a pivotal task. Typically, it comprises two objectives: scene completion to recover the underlying geometry of occluded regions, and semantic classification to assign categorical labels to every voxel in 3D space.

Fig. 1 (a) compares the existing representations for occupancy prediction. The dense representation [2] is the most straightforward approach, storing features in continuous memory to facilitate dense 3D operations like convolutions. However, this approach is computationally redundant, as approximately 67% of the 3D space is empty³. Alternatively, the Bird’s-Eye View (BEV) [3], [4], [35] representation projects 3D space onto a 2D plane, reducing costs by leveraging efficient 2D encoders. However, this projection incurs a loss of geometric information, limiting the ability to capture fine-grained 3D structure. While the Tri-Perspective View (TPV) [5], [36] attempts to mitigate this loss, it still suffers from compromised perceptual accuracy. Consequently, there is a pressing need for a latent representation that encodes 3D scene structure losslessly with minimal computational overhead. Motivated by the inherent sparsity of 3D space, we investigate the feasibility of fully sparse representation, a widely used practice in point cloud processing [37], for describing voxels in a 3D latent space.

In our preliminary conference work [27], we introduced SparseOcc, a fully sparse architecture operating exclusively on non-empty voxels. SparseOcc comprises three core components: 1. A sparse latent propagator that performs feature propagation and contextual aggregation to complete partially observed geometries while preserving sparsity. 2. A sparse feature pyramid that employs sparse downsampling and interpolation to expand receptive fields across scales, reducing excessive feature propagation. 3. A sparse transformer head that formulates semantic occupancy prediction as a sparse mask estimation problem, confining transformer computation to sparse voxels. This approach demonstrates that a fully sparse representation can effectively balance computational efficiency with structural fidelity.

Despite these advantages, current sparse representation methods, including our SparseOcc, rely on an entangled geometric-semantic representation. They typically perform scene completion by propagating high-dimensional semantic features from visible regions into occluded areas (e.g., via the sparse latent propagator in SparseOcc), followed by voxel-wise classification to determine occupancy status and semantic labels simultaneously. To ensure structural completeness, this paradigm necessitates processing numerous candidate voxels that are ultimately empty (Fig. 1 (b), left), leading to significant computational redundancy in empty spaces and geometric ambiguity.

To fully leverage sparsity in 3D space, we introduce SparseOcc++, a novel framework that explicitly disentangles scene completion from semantic classification via an innovative geometry-aware sparse representation (Fig. 1 (b), right). Inspired by anchor-based object detection methods like Faster R-CNN [38], which predict the geometry of objects via bounding-box regression rather than pixel-wise entangled semantic classification (Fig. 1 (c), left), we reformulate 3D occupancy prediction in two stages: compact scene completion followed by geometry-constrained semantic prediction. Our fundamental insight is that scene completion is better formulated as a geometric regression problem (i.e., determining the spatial extent of an object) rather than as a classification problem (i.e., determining the occupancy status of each voxel).

To mathematically formalize this regression target, we leverage the principles of the signed distance field (SDF, Fig. 1 (c), middle) [39], a representation that inherently encodes geometric proximity, and introduce the scene completion field (SCF, Fig. 1 (c), right). Unlike standard SDFs, which typically model an isotropic scalar distance to the nearest surface, our SCF is a vector-valued field tailored for outdoor scene completion using a set of sparse anchor voxels (similar to anchor boxes in detection). Specifically, SparseOcc++ dynamically initializes sparse anchor voxels from images in a coarse-to-fine manner. Then, it learns an SCF that predicts the signed distances to scene boundaries for each anchor voxel. To effectively model the strong geometric anisotropy of outdoor scenes (e.g., flat roads vs. thin poles), we propose an orthogonal decomposition strategy that instantiates the SCF along planar and vertical axes, complemented by a discretized learning scheme for robust optimization. Based on the learned SCF, we employ an efficient geometry-guided propagation mechanism to achieve compact scene completion. This ensures that computationally intensive semantic segmentation is strictly confined to geometrically verified occupied regions, thereby minimizing computational waste.

The preliminary results of this work have been published in [27]. In this paper, we significantly extend our previous method, SparseOcc, with three solid contributions:

• We introduce a novel geometry-aware sparse representation that disentangles scene completion from semantic classification. By reformulating scene completion as a scene completion field (SCF) learning task on sparse anchor voxels, we significantly reduce computational redundancy in empty regions.

• We propose an orthogonal decomposition strategy and a discretized learning scheme for robust anisotropic SCF learning, as well as a geometry-guided propagation mechanism that compactly converts the learned SCF into complete scenes.

• The proposed SparseOcc++ establishes a new state-of-the-art performance while maintaining real-time inference speed. It improves IoU by 2.3% and runs \(3.9\times\) faster than SparseOcc on nuScenes [2], and \(5.9\times\) faster than the dense counterpart, OccFormer [40], on SemanticKITTI [41].

2 Related Work↩︎

In this section, we briefly review the recent advances in occupancy prediction for autonomous driving and 3D scene representation.

2.1 Occupancy Prediction for Autonomous Driving↩︎

Vision-centric occupancy prediction aims to perceive the voxel-wise 3D geometric and semantic states around the ego vehicle from camera inputs, providing a fine-grained spatial understanding that is critical for downstream planning and control [7]. Early work such as MonoScene [42] demonstrated the feasibility of monocular occupancy prediction. Subsequently, large-scale multi-view benchmarks with dense occupancy annotations, including OpenOccupancy [2], Occ3D [34], OccNet [33], and SurroundOcc [24], were constructed on autonomous driving datasets [43], [44], significantly accelerating progress in this field.

Building upon these benchmarks, numerous methods have been proposed. To mitigate the computational burden of dense grids, a series of efficiency-oriented works [25], [28], [29], [45], [46] and compact representations like TPV [5], [47] and Gaussian Splatting [30], [48]–[51] have been proposed. To boost recognition capability, efforts have been directed toward architectural refinements, including pretraining [52], stronger backbones [53], and robust 2D-to-3D lifting [32], [54], [55], as well as expanding into open-vocabulary [56]–[58] and panoptic [59] domains. Furthermore, to handle data scarcity and complexity, the community is actively exploring self-supervised learning [60]–[63], multi-modal fusion [64]–[67], and generative approaches [68]–[70].

Despite these multi-faceted improvements, effectively modeling 3D geometry from 2D images remains a fundamental bottleneck. Since sensors only capture visible surfaces, a substantial portion of the scene remains occluded. To achieve scene completion, many approaches rely on dense feature propagation, spreading information into zero-initialized voxels via heavy 3D operators [71], [72]. While effective, this results in significant computational redundancy. Alternatively, recent works resort to neural rendering strategies. Methods like UniOcc [73], RenderOcc [74], and OccNeRF [75] impose NeRF-style 2D supervision to implicitly learn 3D geometry, while GaussianFormer [30] utilizes object-centric 3D Gaussians [76]. However, these methods often entangle geometry with appearance or require complex rendering pipelines, limiting their direct applicability to efficient geometry reasoning.

In contrast to these dense hallucination or implicit rendering approaches, we advocate for a fully sparse representation. In our preliminary work [27], we introduced a sparse latent propagator to perform scene completion exclusively within non-empty regions. Extending this, this work further disentangles geometry from semantics by introducing anchor voxels. Inspired by anchor-based object detection [38] and signed distance fields [39], we formulate geometry reconstruction as a scene completion field learning problem on sparse anchor voxels. This allows us to efficiently recover precise volumetric structures without the overhead of dense computation or the complexity of implicit rendering.

2.2 3D Scene Representation↩︎

Representing the surrounding environment with spatial latent information is an indispensable procedure for autonomous driving perception algorithms. A straightforward solution is to split the scene into voxels and describe it with a 3D dense representation to which 3D operators are applied. VoxelNet [77] is a pioneering work that first explores this strategy for object detection. Following this, occupancy networks also use 3D dense voxels to describe the scene [2], [24], [40], [42], [78], [79]. For example, C-CONet [2] uses ResNet3D [80] and FPN3D [81] to propagate non-empty features to adjacent zero-initialized areas. SurroundOcc [24] uses successive deformable cross attention layers [82] to transform the multi-scale image features into a multi-scale 3D dense volume. However, these 3D operators often have cubic time and spatial complexity, which is not feasible in practice.

To alleviate the heavy computational burden of dense volumes, the community shifted towards Bird’s-Eye View (BEV) representations [3], [10], [35], [83]–[87]. These methods flatten the 3D space into a 2D plane by compressing the height dimension, allowing for the use of efficient 2D backbones. While effective for tasks like object detection where height is less critical, this aggressive dimensionality reduction causes severe information loss. The collapsed height dimension makes it inherently difficult to distinguish overlapping structures (e.g., bridges, signs) or capture fine-grained 3D occupancy details.

To bridge the gap between dense fidelity and BEV efficiency, the Tri-Perspective View (TPV) representation [5], [36] was proposed. Instead of compressing the scene into a single BEV plane, TPV factorizes the 3D space into three orthogonal 2D planes (top, front, and side views). Voxels in 3D space are reconstructed by summing the projected features from these planes. Although TPV mitigates the geometric loss of BEV, the projection process inevitably introduces feature ambiguity, as a single pixel on a plane corresponds to a line in 3D space, limiting its ability to accurately model complex driving scenes.

Given that outdoor 3D scenes are inherently sparse (dominated by empty space), sparse representations offer a promising direction to bypass the redundancy of dense grids. The core idea is to allocate computation exclusively to non-empty regions. In object detection, the FSD series [88]–[90] pioneered fully sparse architectures to eliminate dense feature maps. In occupancy prediction, SparseOcc [27], [28] and OctreeOcc [25] utilize sparse queries or octree structures to model valid voxels. More recently, implicit sparse representations like GaussianFormer [30], [31] and set-based prediction methods like OPUS [26] have emerged. These approaches successfully reduce memory usage, demonstrating the effectiveness of sparse representation.

However, we find that the entanglement of geometry and semantics in existing sparse methods introduces a new form of redundancy, i.e., processing candidate regions that turn out to be empty. To address this, this paper presents a novel geometry-aware sparse representation that explicitly disentangles these two processes, attaining state-of-the-art accuracy while running at real-time inference speed.

3 SparseOcc↩︎

This section reintroduces SparseOcc, a fully sparse architecture for efficient 3D semantic occupancy prediction. We begin by defining the sparse representation that encodes only occupied voxels (Sec. 3.1). Subsequently, we introduce the core components: the sparse latent propagator for scene completion (Sec. 3.2), the multi-scale sparse feature pyramid for contextual reasoning (Sec. 3.3), and the sparse transformer head for mask-based occupancy prediction (Sec. 3.4), followed by the training objective (Sec. 3.5). Finally, we analyze the limitations of its entangled geometric-semantic representation (Sec. 3.6), which sheds light on the extension method in the next section.

3.1 Sparse Representation↩︎

Given a monocular image or a set of images \(\mathbf{I}=\{I_i\}_{i=1}^{N_\text{view}}\) captured by surrounding cameras, the goal of vision-based 3D semantic occupancy prediction is to predict a semantic label for every voxel in a predefined 3D volume \(\mathbf{Y}=\{0,1,2,...,N_\text{cls}\}^{H\times W \times D}\), where \(\{H,W,D\}\) denotes the spatial resolution, the class label 0 indicates an empty voxel, and \(N_\text{cls}\) is the number of semantic classes for non-empty voxels.

In the initial stage of our model, we follow the Lift-Splat-Shoot (LSS) [9] framework. The images are first passed through an image encoder, such as ResNet [80] augmented with FPN [81]. This encoder generates latent features on the 2D perspective plane. Subsequently, the 2D latent features are lifted to 3D space using predicted depth maps, resulting in dense cubic features denoted as \(\mathbf{V}\in \mathbb{R}^{H\times W \times D \times C}\). Notably, we observe that approximately 80% of the voxels in LSS lifted \(\mathbf{V}\) are empty. This sparsity arises due to the nature of LSS, where 2D features are cast through ray casting and become sparser in distant regions.

We then convert the dense feature \(\mathbf{V}\) to a sparse representation by gathering non-empty voxels. The sparse tensor is stored efficiently in the commonly used coordinate (COO) format:

\[\label{eq:sparse95representation} \mathbb{V} = \{ (\mathbf{p}_i=[x_i, y_i, z_i] \in \mathbb{Z}^3, \mathbf{f}_i \in \mathbb{R}^C ) \vert i=1,2,...N \}.\tag{1}\] In the above equation, \(N\) represents the number of non-empty voxels, while \(\mathbf{p}_i\) and \(\mathbf{f}_i\) denote the coordinates and features of the \(i\)-th voxel, respectively. All subsequent operations are performed on this sparse representation, thereby eliminating redundant computations on empty voxels. The following sections elaborate on the proposed sparse building blocks, namely the sparse latent propagator, feature pyramid, and transformer head. An overview is shown in Fig. 2.

3.2 Sparse Latent Propagator↩︎

The sparse representation \(\mathbb{V}\) is derived via ray casting, yielding a predominantly sparse representation that is limited to the initial intersection surfaces between rays and objects. Consequently, most observations are inherently incomplete. In contrast, the objective of an occupancy network is to predict complete occupancy rather than solely the visible parts. Traditional approaches address this by incorporating 3D dense convolutions (e.g., 3D ResNet and 3D FPN) or attention layers (e.g., deformable self-attention) to propagate non-empty features to adjacent empty regions, thereby completing the scene. In this work, we aim to design a sparse variant of the latent propagator. However, a notable challenge arises: the propagator’s objective appears to conflict with the sparse design. By stacking additional completion blocks, the scene is better completed; however, spatial sparsity also decreases, thereby reducing efficiency. To strike a balance between scene completion and sparsity, we build our sparse latent propagator with two key components: A sparse completion block, which executes only necessary latent propagation; and a contextual aggregation block, which aggregates valid features without engaging in completion.

Sparse Completion Block. We opt for the 3D sparse convolution implemented by [37] to build the sparse completion block. A sparse convolution performs the computation in a local window where at least one non-empty voxel resides, allowing for the propagation of features from non-empty voxels to their neighbors. The range of propagation can be expanded by stacking multiple layers of sparse convolution. To maintain spatial sparsity, we employ only one 3D convolutional layer in a sparse completion block.

Contextual Aggregation Block. After completion, we introduce the contextual aggregation block to effectively utilize geometric and semantic features from the local context. For constructing this block, we choose sparse submanifold convolution [37] over regular sparse convolution. Submanifold convolution ensures that an output location is active only if the corresponding input location is active, thereby maintaining sparsity even when stacking multiple layers.

Kernel Decomposition. Foreground objects and background elements in driving scenes often exhibit specific shape distributions. For instance, roadways and sidewalks typically have a thin, flat shape located at the bottom of the 3D volume, making them amenable to completion through convolutions in the horizontal direction. Conversely, structures such as buildings and car-like objects are rectangular, necessitating feature propagation in the vertical direction. To fully leverage these distinct shape distributions, we decompose a conventional \(k\times k \times k\) kernel into orthogonal kernels [91]. Specifically, for the sparse completion block, we replace the sparse convolution with three consecutive layers with \(k\times k\times 1\), \(k\times 1 \times k\), and \(1 \times k \times k\) kernels, respectively. For the contextual aggregation block, we follow Cylinder3D [91] and replace a \(k\times k \times k\) submanifold convolution with two parallel but asymmetrical branches of decomposed layers. One branch consists of two consecutive layers with \(1 \times k \times k\) and \(k \times 1 \times k\) kernels, and the other branch with \(k \times 1 \times k\) and \(1 \times k \times k\) kernels. Note that the complexity is reduced from \(\mathcal{O}(k^3)\) to \(\mathcal{O}(3k^2)\) or \(\mathcal{O}(4k^2)\) after the decomposition. Although the actual cost is not reduced when using a small kernel with \(k=3\), the expressive capacity of decomposed kernels outperforms that of a single full kernel; thus, we can still achieve efficiency improvements by stacking fewer layers. The final sparse latent propagator is shown in Fig. 3.

3.3 Sparse Feature Pyramid↩︎

A straightforward approach to completing the scene is to stack the proposed sparse propagator multiple times. However, this necessitates a substantial number of sparse propagators to ensure an adequately large receptive field, which is particularly important for recognizing large objects like ‘truck’ or static elements such as ‘road’. The computational cost is obviously high. To address this issue, we note that down-sample layers, implemented with sparse convolution with a stride greater than one, not only reduce spatial resolution but also increase relative sparsity in the new scale. By building a multi-scale sparse feature pyramid with down-sample layers, we can readily obtain a coarse-to-fine representation of the scene. This ensures that querying any spatial location can be addressed by at least one feature scale, simultaneously reducing computational costs. Formally, we stack the sparse propagator for \(L\) times, each one followed by a down-sample layer, and the feature pyramid is collected as \(\{\mathbb{V}_l\}_{l=1}^L\). Additionally, the spatial size along the height dimension of the last two scales is too small, so we simply omit the \(D\) dimension. For the completion blocks, the 3D convolution is replaced with a 2D version with \(3 \times 3\) kernel. For the contextual aggregation block, we replace the asymmetrical branches with two parallel 2D submanifold convolutions with \(5\times5\) kernels.

Sparse Voxel Decoder. Former methods [40], [92] use the multi-scale deformable attention transformer (MSDeformAttn) [93] in the pixel/voxel decoder, which is responsible for aggregating intra-scale and inter-scale features. Aiming to save GPU memory and time, we simplify this process by using interpolation to fuse multi-scale sparse features output by the 3D sparse encoder. Specifically, for a given scale \(\mathbb{V}_l\) from the feature pyramid, we augment it by fusing all the other scales, \[\hat{\mathbb{V}}_l = \sum_{j\ne l} W_j \cdot \mathrm{Interp}(\mathbb{V}_j, \mathbb{V}_l)\] where \(W_j\) is a learned weight for the \(j\)-th scale, and \(\mathrm{Interp}(\mathbb{X},\mathbb{Y})\) indicates linear interpolation of features from the sparse tensor \(\mathbb{X}\) to the coordinates of \(\mathbb{Y}\).

By leveraging this lightweight feature-fusion approach, the feature pyramid is enriched with semantic information across different scales. Moreover, the high-resolution features benefit from additional completion provided by the low-resolution features, as the denser low-resolution features resist dilution by interpolation operators.

3.4 Sparse Transformer Head↩︎

We frame the semantic occupancy prediction as a 3D segmentation problem and employ a transformer head, inspired by the design of Mask2Former [92]. This head iteratively updates a set of learnable queries through masked attention and subsequently decodes these queries into 3D masks for all semantic classes. However, a challenge arises due to the need for dense binary masks to represent semantic classes. Additionally, as we perform predictions in a 3D space [40], the computational and memory costs associated with applying successive transformer decoder layers to dense masks become impractical.

To overcome this challenge, we adapt the dense transformer head to a sparse variant. Specifically, we assert that accurate segmentation is crucial for occupied voxels, while non-occupied voxels need not be considered during this phase. In the ensuing discussion, we outline several steps to delineate the sparse transformer decoder.

Preprocessing and Notations. Given the sparse feature pyramid \(\hat{\mathbb{V}}_l\), we first employ a linear binary classifier for coarse segmentation. The binary classifier is trained to label a voxel as empty if the semantic ground truth is 0 and as non-empty otherwise. We only preserve voxels that are classified as non-empty, resulting in a filtered, sparser tensor \(\hat{\mathbb{V}}_l = \{ (\mathbf{p}_i, \mathbf{f}_i) \vert i=1,...,N_l\}\), where we use \(N_l\) to denote the number of remaining voxels for the \(l\)-th scale. Storing variant features for the empty voxels is unnecessary; instead, we find it sufficient to use only a single learnable token \(\mathbf{p}_{\phi}\) to represent all the empty voxels.

Query Decoding. The former method [40] performs an outer product between queries \(Q \in \mathbb{R}^{N_q \times C}\) and 3D dense features \(F \in \mathbb{R}^{C\times H\times W\times D}\) to decode the queries into a 3D mask with a shape \((N_q, H, W, D)\), which has a time complexity of \(\mathcal{O}(N_q H W D C)\). To facilitate inference speed, we only perform an outer product between \(Q\) and \(\mathbf{p}_{\phi} \cup \{\mathbf{f}_i \vert (\mathbf{p}_i, \mathbf{f}_i) \in \hat{\mathbb{V}}_l \}\) to obtain a series of occupied masks \(M^\mathrm{occ}\in\mathbb{R}^{N_q\times N_l}\) and a single mask \(M^{\phi}\in\mathbb{R}^{N_q\times 1}\) that represents empty voxels. With the saved coordinates \(\mathbf{p}\) of occupied voxels, we can easily reconstruct the dense 3D mask \(M\in\mathbb{R}^{Q\times H\times W\times Z}\) from the predicted sparse tensor using a scatter operation that does not break the gradient flow. In this way, the complexity is reduced to \(\mathcal{O}(N_l N_q C+HWZ)\) for mask prediction and reconstruction. Note that \[\begin{align} \label{Eq:ema} & \mathcal{O}(N_l N_q C+HWZ) \nonumber \\ & = \mathcal{O}(HWZN_q C(\frac{N_l}{HWZ}+\frac{1}{N_qC})) \\ & < \mathcal{O}(H W Z N_q C), \nonumber \end{align}\tag{2}\] because \({N_l}/{HWZ}\) is much smaller than \(1\) in our sparse case. Moreover, the \(N_q\) queries are also input to a \(C\)-way linear classifier to classify the corresponding 3D mask into predefined \(C\) semantic categories.

Query Updating. Given the input \(L\) layers of multi-scale sparse features, we iteratively alternate between query decoding and query updating in each transformer layer. With the predicted 3D masks \(M_{l-1}\) in the \((l-1)\)-th transformer layer, we update the queries via \[Q_l = \mathrm{softmax}\left[\mathcal{M}_{l-1}+W_q Q_{l-1}(W_k\hat{\mathbf{V}}_l)^T\right]W_v \hat{\mathbf{V}}_l+Q_{l-1},\] where \(W_q, W_k, W_v\) are linear layers, \(\hat{\mathbf{V}}_l\) is the dense version reconstructed from \(\hat{\mathbb{V}}_l\), and the attention mask \(\mathcal{M}_{l-1}\) at location \((x, y, z)\) is obtained by \[\begin{align} &\mathcal{M}_{l-1}(x, y,z)=\left\{ \begin{array}{ll} 0 & \text{if } \sigma \left(M_{l-1}^{'}(x, y, z)\right) \geq 0.5\\ -\infty & \text{otherwise} \end{array} \right. \end{align}\] where \(\sigma\) is the sigmoid function, \(M_{l-1}^{'}=\mathrm{maxpooling}(M_{l-1})\) which resizes the 3D mask to the same resolution as \(\hat{\mathbf{V}}_l\), similar to the implementation in [40].

3.5 Objective Function↩︎

Considering that the sparse transformer head formulates semantic occupancy as a mask set prediction task [40], [92], bipartite matching with the Hungarian solver is used to assign binary mask labels and corresponding semantic class labels to the predicted masks. Based on the assignment, we calculate the mask loss \(\mathcal{L}_\mathrm{mask}\) and the classification loss \(\mathcal{L}_\mathrm{cls}\). Additionally, \(\mathcal{L}_\mathrm{depth}\) is calculated between the predicted depth map and the ground truth projected by point clouds for the supervision of the LSS component. Moreover, the coarse binary classification on non-empty voxels is also supervised by a segmentation loss \(\mathcal{L}_\mathrm{seg}\). Finally, the overall objective function is a simple summation of these loss terms \[\mathcal{L} = \mathcal{L}_\mathrm{mask}+\mathcal{L}_\mathrm{cls}+\mathcal{L}_\mathrm{depth}+\mathcal{L}_\mathrm{seg}.\]

3.6 Entangled Geometric-Semantic Sparse Representation↩︎

While the sparse representation defined in Eq. 1 significantly reduces memory usage compared to dense counterparts, it operates on an entangled geometric-semantic sparse representation, where each voxel \((\mathbf{p}_i, \mathbf{f}_i)\) encodes both the geometric occupancy and the semantic information of objects within a single high-dimensional feature vector \(\mathbf{f}_i\).

To complete the scene from the initially sparse latent features, SparseOcc resorts to the Sparse Latent Propagator (refer to Sec. 3.2) to indiscriminately propagate these high-dimensional entangled latent features from observed regions into the empty spaces surrounding the object. Then, it performs voxel-wise classification to determine both occupancy status (‘occupied’ or ‘empty’) and semantic labels (‘car’ or ‘bicycle’), thereby achieving scene completion and semantic prediction simultaneously. This entanglement leads to two critical limitations: (1) Computational redundancy: To ensure the geometric completeness of the scene, SparseOcc is compelled to reconstruct features for a vast number of candidate voxels surrounding the objects. As shown on the left in Fig. 1 (b), this results in significant wasted computation, as it requires substantial resources calculating high-dimensional embeddings for voxels that are ultimately classified as empty. (2) Geometric ambiguity: Since the Sparse Latent Propagator indiscriminately propagates high-dimensional semantic features from observed regions into the surroundings, it inevitably contaminates adjacent empty voxels with strong feature signals. Consequently, the occupancy classifier easily misclassifies empty regions as occupied, introducing geometric ambiguity.

4 SparseOcc++↩︎

To further exploit the potential of sparse representation, we present SparseOcc++, which features a novel geometry-aware sparse representation, as defined in Sec. 4.1. Fig. 4 presents an overview of the proposed SparseOcc++. It first extracts 2D features from images and dynamically initializes a set of sparse anchor voxels (Sec. 4.2). To reconstruct the 3D geometry of occupied regions, these sparse anchor voxels function as queries to index corresponding image features via deformable cross attention (DCA), which are then used to complete the scene via scene completion field (SCF) learning (Sec. 4.3). Then, adapting to the predicted occupied regions, we use a Sparse U-net [94] to extract sparse voxel features and predict the final occupancy results (Sec. 4.4).

4.1 Geometry-Aware Sparse Representation↩︎

In prior approaches, scene completion is inherently entangled with high-dimensional semantic reasoning, leading to computational redundancy and geometric ambiguity (as discussed in Sec. 3.6). To address this limitation, we propose a novel geometry-aware sparse representation. Our core strategy is to explicitly disentangle scene completion from semantic classification. Instead of achieving scene completion via indiscriminate semantic feature propagation and voxel-wise classification, we reformulate the problem as a regression task, allowing us to first focus on recovering the scene’s structural extent before attending to semantics.

Sparse Anchor Voxel. To implement this regression-based scene completion, we draw inspiration from anchor-based 2D object detection methods such as Faster R-CNN [38], which efficiently estimates object geometry via bounding-box regression from anchor boxes rather than dense pixel-wise classification. Analogously, we aim to recover the 3D scene geometry by regressing 3D geometric properties from sparse anchor voxels \(\mathbb{A}\). Unlike the sparse representation in Eq. 1 , which contains only coordinates and semantic embeddings, we augment each anchor with an explicit geometric attribute: \[\mathbb{A} = \{ (\mathbf{p}_i, \mathbf{f}_i, \mathbf{s}_i) \vert i=1,2,...N \},\] where \(\mathbf{p}_i \in \mathbb{Z}^3\) denotes the coordinate, \(\mathbf{f}_i \in \mathbb{R}^C\) is the learnable feature embedding, and \(\mathbf{s}_i\) is the newly introduced geometric attribute designed to encode the scene structure.

Scene Completion Field. However, unlike object detection, which merely approximates an object’s coarse envelope via bounding boxes, occupancy prediction demands the recovery of precise geometric boundaries. To define a geometric attribute \(\mathbf{s}\) that captures this fine-grained structural detail, we leverage the theory of Signed Distance Fields (SDF), which is widely used in computer graphics. An SDF represents geometry as a continuous field, where the value indicates the distance to the nearest surface. Building on this theoretical foundation, we define a specialized scene completion field (SCF). Different from standard SDFs that map coordinates to a single isotropic scalar representing the radius to the nearest surface point, our SCF is tailored to complete the complex, anisotropic shapes of outdoor objects. Therefore, we formulate the SCF as a vector-valued function \(\Phi\) conditioned on both spatial location \(\mathbf{p}_i\) and semantic context \(\mathbf{f}_i\): \[\label{equ:scf} \mathbf{s}_i = \Phi(\mathbf{p}_i, \mathbf{f}i) \in \mathbb{R}^K.\tag{3}\] Here, the geometric attribute \(\mathbf{s}_i \in \mathbb{R}^K\) is a vector where each component \(s\) represents the signed distance along a specific direction (e.g., orthogonal axes): \[\label{equ:sdf} {s} = \begin{cases} +\min_{\mathbf{v} \in \partial \Omega} \| \mathbf{p}_i - \mathbf{v} \|_2 & \text{if } \mathbb{A}_i \in \Omega_\text{empty} \\ 0 & \text{if } \mathbb{A}_i \in \partial \Omega \\ -\min_{\mathbf{v} \in \partial \Omega} \| \mathbf{p}_i - \mathbf{v} \|_2 & \text{if } \mathbb{A}_i \in \Omega_\text{occupied} \end{cases}\tag{4}\] where \(\Omega \subset \mathbb{Z}^3\) denotes the voxel space and \(\partial \Omega\) represents the geometric boundary along the corresponding direction of \(s\). Specifically, the sign of these distance components serves as a decisive indicator of occupancy status: positive values indicate empty space, while negative values signify the interiors of occupied regions.

By explicitly incorporating the SCF distance \(\mathbf{s}_i\) into the sparse representation, we fundamentally shift the scene completion paradigm from exhaustive voxel-wise classification to spatial proximity learning. Consequently, we can efficiently determine the spatial extent of the scene, ensuring that subsequent fine-grained semantic predictions are strictly confined to the geometrically verified occupied voxels, thereby significantly reducing computational redundancy.

Remarks: Connection and Distinction with Standard SDF. While our SCF is theoretically rooted in the Signed Distance Field, it differs fundamentally in three aspects tailored for occupancy prediction:

• Conditioning Context: Standard SDFs typically map spatial coordinates \(\mathbf{p}\) to distances for a static shape. In contrast, our SCF is a conditional field \(\Phi(\mathbf{p}, \mathbf{f})\) inferred from visual observations. It relies on semantic image features \(\mathbf{f}\) to predict the missing geometries of occluded regions rather than merely fitting the observed surface voxels.

• Anisotropy: A standard SDF is a scalar field representing the isotropic distance to the nearest surface. Our SCF is formulated as a vector-valued field in Eq. 3 . This vector formulation enables the encoding of anisotropic geometric information (e.g., decomposing into orthogonal components in Sec. 4.3), which is critical for completing the complex aspect ratios of outdoor objects.

• Task Objective: SDFs are often used for surface reconstruction or rendering. Our SCF is specifically designed for scene completion, where the zero-level set serves as a geometric hull to constrain and propagate semantic information.

4.2 Sparse Anchor Voxel Generation↩︎

To efficiently recover the volumetric structure using the proposed SCF without exhaustively processing numerous empty candidate voxels, we introduce the concept of sparse anchor voxels. Similar to anchor boxes in object detection [38], [95], these anchor voxels serve as initial geometric proposals from which our SCF is sampled and learned.

A straightforward implementation of sparse anchor voxel generation is to sample coordinates uniformly across the entire space. However, given the inherent sparsity of 3D scenes, in which occupied regions constitute only a small fraction of the total volume, uniform sampling incurs significant computational costs in empty spaces. To mitigate this, following [25], [26], [28], we adopt a coarse-to-fine strategy to progressively generate anchor voxels strictly within geometrically important regions.

Contextualization via Deformable Cross Attention. To determine the importance of each voxel, we must first endow it with image context. Given a set of sparse anchor voxel candidates \(\mathbb{\tilde{A}}\), we utilize Deformable Cross Attention (DCA) [82] to aggregate multi-view image features. Specifically, for each anchor candidate \((\mathbf{p}, \mathbf{f}) \in \mathbb{\tilde{A}}\), we project its 3D center \(\mathbf{p}\) into the 2D image planes via the LiDAR-to-image transformation matrices \(\mathcal{T}\), obtaining reference points \(\pi(\mathbf{p}, \mathcal{T})\). These points guide the sampling of image features \(\mathbf{F}^\text{2D}\). The updated feature is computed as: \[\text{DCA}(\mathbb{\tilde{A}}, \mathbf{F}^\text{2D}) = \frac{1}{N_\text{view}} \sum_{v=1}^{N_\text{view}} \sum_{j=1}^{N_s} w_{vj} \cdot \mathbf{F}^\text{2D}_v(\pi(\mathbf{p}, \mathcal{T}_v) + \Delta \mathbf{p}_{vj}),\] where \(N_s\) is the number of sampling points, \(\Delta \mathbf{p}_{vj}\) is the learned sampling offset, and \(w_{vj}\) is the attention weight. This mechanism aligns the 3D sparse anchors with their 2D visual evidence, thereby facilitating the effective selection of sparse anchor voxels.

Pyramidal Generation Strategy. We instantiate the anchor generation as a multi-stage process across resolution levels \(r \in \{0, 1, 2\}\), corresponding to \(\{1/8, 1/4, 1/2\}\) of the original resolution. Starting from an initial set \(\mathbb{\tilde{A}}^{0}\) comprising dense voxels at the coarsest level (initialized with random embeddings), we iteratively refine the set to obtain \(\mathbb{\tilde{A}}^{r}\): \[\begin{align} &\mathbb{S}^{r} = \text{Split}(\mathbb{\tilde{A}}^{r-1}), \\ &\mathbb{\tilde{A}}^r = \text{Top-k} \left( \mathbb{S}^{r}, \mathcal{M}(\text{DCA}(\mathbb{S}^{r})), N^{r} \right). \end{align}\] Here, \(\text{Split}(\cdot)\) denotes the geometric subdivision, where each parent voxel at level \(r-1\) is split into 8 sub-voxels at level \(r\). \(\mathcal{M}(\cdot)\) is a lightweight MLP that predicts an occupancy score from DCA-aggregated features. \(\text{Top-k}(\cdot)\) selects the top \(N^r\) voxels with the highest scores to form the sparse anchor set for the current level. In our implementation, we retain the top \(\{1/5, 1/10\}\) candidates for levels 1 and 2, respectively. Crucially, this cascade ensures that computation is concentrated solely on regions with high geometric saliency. The final set \(\mathbb{\tilde{A}}^{2}\) is then utilized as the sparse anchor voxels \(\mathbb{A}\) for the SCF-based scene completion described in Sec. 4.3.

4.3 Compact Scene Completion↩︎

Building on the generated sparse anchors, this section details the learning of the scene completion field (SCF) \(\Phi\). To accurately model complex outdoor scenes, we propose a specific orthogonal instantiation of the geometric vector \(\mathbf{s}\), trained via a robust discretized learning strategy.

Orthogonal Decomposition Strategy. While the SCF definition in Eq. 3 allows for arbitrary directional distances, a naive isotropic design (i.e., treating \(\mathbf{s}\) as a single scalar radius) is suboptimal for outdoor driving scenes due to strong geometric anisotropy. For instance, ‘road’ surfaces exhibit extensive horizontal spans but are extremely thin vertically, whereas ‘poles’ are the inverse. As a result, a single scalar cannot simultaneously capture these conflicting spatial scales. As shown in Fig. 5 (b), the SCF distance \(\mathbf{{s}}\) is too small for sparse anchor voxels to model the scene.

To address this, we leverage the vector nature of our SCF and design \(\mathbf{s}\) to explicitly decouple the scene along orthogonal axes. Specifically, we instantiate the SCF vector with two orthogonal components: the planar SCF distance \({s}^{\text{X-Y}}\) and the vertical SCF distance \({s}^{\text{Z}}\), i.e., \(\mathbf{s} = ({s}^{\text{X-Y}}, {s}^{\text{Z}})\). Specifically, for the anchor voxel at \(\mathbf{p}_i=(x_i, y_i, z_i)\). The planar component \({s}^{\text{X-Y}}\) measures the distance to the boundary within the horizontal cross-section plane \(\mathcal{P}_i = \{ (x, y, z) \mid z = z_i \}\), capturing the horizontal spread. Conversely, the vertical component \({s}^{\text{Z}}\) is constrained to the vertical axis \(\mathcal{L}_i = \{ (x, y, z) \mid x = x_i, y = y_i \}\), capturing height information. As shown in Fig. 5 (c-d), this orthogonal decomposition enables the anchors to flexibly adapt to irregular shapes, significantly improving reconstruction accuracy.

Discretized Learning. Direct regression of continuous SCF values often suffers from convergence instability due to the infinite solution space and high variance inherent in outdoor environments. Inspired by recent advances in depth estimation [10], we reformulate this regression problem as a classification task. Taking the planar SCF as an instance, we define a range \([-{s}^{\text{X-Y}}_{\text{max}}, {s}^{\text{X-Y}}_{\text{max}}]\) and discretize it into a set of bins \(\mathbb{D} = \{d_1, d_2, \dots, d_{|\mathbb{D}|}\}\), where \({s}^{\text{X-Y}}_{\text{max}}\) is the maximum SCF planar distance. Consequently, the regression task is transformed into a \(|\mathbb{D}|\)-way classification problem, where \(|\mathbb{D}| = 2{s}^{\text{X-Y}}_{\text{max}} + 1\). For each anchor voxel, we employ an MLP head to predict the probability distribution \(\mathbf{P}^{\text{X-Y}}\) over \(\mathbb{D}\). The final distance is determined by the class index with the maximum likelihood: \[{\hat{s}}^{\text{X-Y}}_i = d_{k} \quad \text{where} \quad k = \operatorname*{argmax}_{k \in {1, \dots, \|\mathbb{D}\|}} \mathbf{P}^{\text{X-Y}}_i(k).\]

Similarly, the vertical component \({s}^{\text{Z}}\) is learned using the same formulation. This discretized geometry learning strategy stabilizes training convergence and improves robustness to sensor noise.

Geometry-Guided Propagation.

Once the sparse anchor voxels with their predicted SCF distances are obtained, we proceed to complete the scene. The predicted SCF distance \({\hat{s}}\) serves as a strong geometric prior, indicating the spatial support of the object centered at each anchor. To complete the scene while maintaining efficiency, we introduce a Geometry-Guided Propagation strategy (Algorithm 6). Instead of indiscriminately densifying the space using the sparse latent propagator, we propagate the occupancy status from each sparse anchor to its local neighborhood, strictly guided by the predicted distance \({\hat{s}}\). Specifically, we treat \({\hat{s}}\) as a propagation radius. By iterating through quantized distance values \({\hat{s}}_j\), we apply a 3D Max-Pooling operation with a kernel size of \(2{\hat{s}}_j + 1\) on the corresponding anchors. This operation effectively expands the single occupied anchor voxel into a local block with a radius of \({\hat{s}}_j\), completing the missing voxels inside the scene boundary. This strategy ensures that scene completion is geometry-aware: new voxels are generated only where the SCF predicts the existence of an object, thereby avoiding computational waste from processing empty regions.

SCF Ground Truth Generation.

To supervise SCF learning, we efficiently generate ground truth (GT) from semantic occupancy labels. The process is outlined in Algorithm 7. First, we simplify the semantic labels into a binary geometric occupancy grid \(\mathbf{G}\) (1 for occupied, 0 for empty). Instead of using an expensive Euclidean Distance Transform (EDT), we employ an iterative 3D average-pooling strategy. For a candidate distance \({s}_j\), we apply 3D average pooling with a kernel size \(K = 2{s}_j+1\) on \(\mathbf{G}\). If a voxel’s value remains unchanged after pooling, it indicates that its neighborhood within a radius \({s}_j\) is homogeneous (purely occupied or empty), confirming that its distance is at least \({s}_j\). By iterating \({s}_j\) from 1 to \({s}_{\text{max}}\), we resolve the distance values for all voxels. Consistent with our decomposition strategy, we generate GT separately for planar distance \({s}^{\text{X-Y}}\) and vertical distance \({s}^{\text{Z}}\) by adjusting the kernel dimensions: \(K^{\text{X-Y}}=[2{s}^{\text{X-Y}}+1, 2{s}^{\text{X-Y}}+1, 1]\) for the horizontal plane and \(K^{\text{Z}}=[1, 1, 2{s}^{\text{Z}}+1]\) for the vertical axis.

4.4 Geometry-Constrained Semantic Prediction↩︎

Upon scene completion, we obtain a refined voxel set \(\mathbb{V}_{\text{occ}}\). However, a feature gap exists: while the original anchor voxels possess learned embeddings, the newly completed voxels are initially featureless (zero-initialized). To bridge this gap, we reapply Deformable Cross Attention (DCA) to the complete voxel set \(\mathbb{V}_{\text{occ}}\). This enables newly generated voxels to query their projected locations in multi-view images and to aggregate discriminative visual features. Finally, these geometrically refined and semantically enriched sparse voxels are processed by a lightweight Sparse U-Net [94], which facilitates local context exchange among neighboring voxels, yielding the final semantic occupancy results \(\mathbb{\hat{Y}}\).

4.5 Objective Function↩︎

Following the previous occupancy prediction method [2], [42], we employ cross-entropy loss \(\mathcal{L}_\mathrm{ce}^\mathrm{occ}\), Lovasz loss \(\mathcal{L}_\mathrm{lovasz}\), affinity loss \(\mathcal{L}_\mathrm{scale}^\mathrm{geo}\), and \(\mathcal{L}_\mathrm{scale}^\mathrm{sem}\) to supervise the final occupancy \(\mathbb{Y}\):

\[\label{equ:loss95occ} \mathcal{L}_\mathrm{occ} = \mathcal{L}_\mathrm{ce}^\mathrm{occ}+\mathcal{L}_\mathrm{lovasz}+\mathcal{L}_\mathrm{scale}^\mathrm{geo}+\mathcal{L}_\mathrm{scale}^\mathrm{sem}.\tag{5}\]

We also apply another loss \(\mathcal{L}_\mathrm{anchor}\), which is the same occupancy loss in Eq. 5 , to supervise the multi-level predictions on \(\mathbb{\tilde{A}}\) and guide the sparse anchor voxel generation in Sec. 4.2. Besides, the learning of SCF in Sec. 4.3 is also supervised by a cross-entropy loss \(\mathcal{L}_\mathrm{scf} = \mathcal{L}_\mathrm{ce}^\mathrm{scf}\) as a classification task.

Hence, the overall objective function is formulated as: \[\mathcal{L} = \mathcal{L}_\mathrm{occ}+\mathcal{L}_\mathrm{anchor}+\mathcal{L}_\mathrm{dist}.\]

5 Experiments↩︎

This section provides a comprehensive evaluation of SparseOcc and SparseOcc++.

0.0035

5.1 Datasets↩︎

We evaluate our proposed sparse representations on two popular benchmarks: SemanticKITTI [41] and nuScenes-Occupancy [2].

SemanticKITTI [41] consists of 22 sequences comprising monocular images, LiDAR point clouds, LiDAR segmentation labels, and semantic scene completion labels. The dataset split is defined as follows: sequence 08 is reserved for validation, sequences 00-10 (excluding 08) are used for training, and sequences 11-21 are set aside for testing. Each 3D voxel is annotated as either empty or one of 19 semantic classes.

nuScenes-Occupancy [2] extends the widely adopted nuScenes [43] dataset by providing dense 3D semantic occupancy annotations for keyframes. These annotations, which include one empty class and 16 semantic classes, were generated using the Augmenting and Purifying pipeline. The dataset comprises 700 training scenes and 150 validation scenes from nuScenes. The original nuScenes dataset provides the corresponding multi-view images and LiDAR point clouds.

5.2 Evaluation Metrics↩︎

Following standard practice [30], [40], [42], we employ Intersection-over-Union (IoU) to evaluate class-agnostic geometry completion and mean IoU (mIoU) across all semantic classes as the semantic metric: \[\text{IoU} = \frac{\text{TP}_{\neq 0}}{\text{TP}_{\neq 0} + \text{FP}_{\neq 0} + \text{FN}_{\neq 0}},\] \[\text{mIoU} = \frac{1}{N_\text{cls}} \sum_{i=1}^{N_\text{cls}} \frac{\text{TP}_i}{\text{TP}_i + \text{FP}_i + \text{FN}_i}, \quad\] where \(N_\text{cls}\) denotes the largest index of non-empty classes, 0 represents the empty class, and \(\text{TP}\), \(\text{FP}\), and \(\text{FN}\) denote the counts of true positives, false positives, and false negatives, respectively.

5.3 Implementation Details↩︎

Input & Backbone. We set the input image resolutions to \((384, 1280)\) for SemanticKITTI and \((256, 704)\) for nuScenes-Occupancy [2]. For the image encoder, to ensure a fair comparison with prior work [2], [42], [72], we primarily use ResNet-50 [80] in most experiments. Specifically for SemanticKITTI, we follow [40] by employing EfficientNetB7 [102] with SecondFPN. Additionally, to demonstrate the scalability and robustness of our method, we introduce a variant named \(\text{SparseOcc++}_{\text{swin}}\) equipped with a stronger Swin Transformer backbone [103].

Network Architecture Details. For SparseOcc, the 2D-to-3D view transformation is implemented following [2], [40]. It generates 3D feature volumes of size 128\(\times\)​128\(\times\)​16 and 128\(\times\)​128\(\times\)​10, with 128 channels for SemanticKITTI and nuScenes-Occupancy, respectively, before converting them into a sparse representation.. We stack the sparse propagator for \(L=4\) times, with each layer followed by a sparse convolution (stride 2) for downsampling. The kernel size \(k\) for the orthogonal convolutions in the sparse completion and contextual aggregation blocks is set to 3. The sparse voxel decoder projects the multi-scale features to 192 channels, enhancing them via interpolation and summation. The sparse transformer head consists of 9 layers for query updating and decoding, utilizing \(N_q=100\) queries. For SparseOcc++, during compact scene completion (Sec. 4.3), we set the maximum planar SCF distance to \(s_\text{max}^{\text{X-Y}}=10\) and the maximum vertical SCF distance to \(s_\text{max}^{\text{Z}}=5\). This completion step is applied once to the final output of the sparse anchor voxel generation module. The lightweight sparse U-Net [94] comprises four layers, each consisting of one sparse residual block [104].

Training Strategy. The learning rate undergoes a warmup phase for the first 100 iterations on SemanticKITTI and 500 iterations on nuScenes-Occupancy, followed by a cosine decay schedule. All models are trained for 30 epochs with a batch size of 8. During training, we apply standard data augmentations, including random flipping, rotation, and resizing, to both input images and the 3D occupancy GT. Following [40], [92], we sample 50,176 points based on class frequency for supervision to accelerate the training of SparseOcc.

Inference & Efficiency. During inference, the predicted masks are upsampled \(2\times\) and \(4\times\) via trilinear interpolation to match the ground-truth sizes of SemanticKITTI and nuScenes-Occupancy, respectively. For efficiency analysis, we report Frames Per Second (FPS) evaluated on a single NVIDIA A100 GPU with a batch size of 1 in FP32 precision. To ensure a fair comparison, no custom kernels are utilized beyond the standard deformable CUDA operator.

3pt

5.4 Benchmark Results↩︎

SemanticKITTI. In Tab. 1, we present a quantitative comparison of our SparseOcc and SparseOcc++ against existing monocular approaches on the SemanticKITTI validation set. Overall, our methods deliver comparable or superior performance. Notably, compared with the dense-occupancy baseline, we achieve higher IoU and mIoU. In addition, VoxFormer [72] and Symphonies [101] adopt a two-stage occupancy framework: they first use pretrained models to estimate occupancy or depth maps, followed by a dense occupancy network to generate final predictions. While this yields strong performance, the FPS reported for these methods accounts only for the second-stage inference. In contrast, our improved variant, SparseOcc++, is fully end-to-end and considerably faster. Furthermore, we observe that SparseOcc++ provides notably better results for small object categories such as ‘person’, ‘bicyclist’, and ‘motorcycle’. This indicates that our approach adaptively propagates features according to the relative positions of voxels with respect to the geometric surface. Crucially, our sparse representation offers distinct efficiency gains: SparseOcc++ operates at a real-time rate of 24.7 FPS, making it \(5.9\times\) faster than OccFormer [40].

nuScenes-Occupancy. Tab. 2 quantitatively compares the proposed sparse representations with several previous works. Specifically, our SparseOcc outperforms the 3D dense representation based C-CONet [2] by 1.7% in geometry IoU and 1.3% in semantic mIoU. This result underscores the effectiveness of SparseOcc. Besides, we observe that the model’s FLOPs are reduced by 74.9% relative to the baseline, further justifying the efficiency of our SparseOcc. Furthermore, benefiting from the proposed geometry-aware sparse representation, our SparseOcc++ achieves the highest geometric IoU, surpassing the dense occupancy network C-OccGen [69] by 0.7% and the entangled geometry-semantic sparse method SparseOcc by 2.3%. Note that the spatial resolution of nuScenes-OpenOccupancy is \(512 \times 512 \times 32\), with each frame containing 6 surrounding images. Consequently, while the inference speed decreases to 19.7 FPS, it remains significantly faster than previous methods [2], [5].

5.5 Qualitative Evaluation↩︎

Fig. 8 compares our methods with the dense OccFormer [40] on SemanticKITTI. Our sparse representation minimizes artifacts, effectively eliminating trailing noise from dynamic vehicles (row 1) and reducing hallucinations around parked cars (row 2). Furthermore, SparseOcc++ excels at fine-grained perception, accurately completing small objects such as a bicyclist (row 3) and a crossing pedestrian (row 4) that OccFormer misses or misclassifies.

Fig. [fig:vis95nusc] extends this comparison to nuScenes-Occupancy against the dense C-CONet [2]. Under challenging conditions, including rainy (row 1), night driving (row 2), and crowded environments (row 3), our methods robustly complete both large planar structures (e.g., roads) and dynamic agents. In summary, these results confirm that our geometry-aware sparse representations achieve precise scene completion with superior efficiency compared to dense baselines.

5.6 Ablation Studies↩︎

Sparse Completion Block. As detailed in Sec. 3.2, the sparse completion block spatially decomposes a 3D sparse convolution kernel into a combination of three orthogonal convolution kernels. Tab. 3 ablates the type and number of sparse kernels. Interestingly, we find that SparseOcc performs satisfactorily even without applying any sparse completion block. We believe this is because the decoder, which fuses the sparse feature pyramid, can also partially complete the scene. Compared with the regular 3\(\times\)​3\(\times\)​3 kernel, the decomposed orthogonal kernels achieve superior scene completion and semantic segmentation results. Additionally, stacking more convolutional blocks does not yield performance improvements. Hence, we build our sparse completion block with only a group of decomposed orthogonal kernels.

Sparse Feature Pyramid.

As shown in Tab. 4, when SparseOcc is equipped with 6 layers of multi-scale deformable attention (MSDeformAttn), it outperforms the proposed sparse decoder. However, as the number of layers in MSDeformAttn decreases, both IoU and mIoU decline, falling behind our simple sparse decoder. Moreover, the training GPU memory and FLOPs of the MSDeformAttn3D are much higher than those of the proposed interpolation and summation method. Compared with FPN3D [81], our sparse voxel decoder yields a 2.1% increase in IoU and a 3.3% increase in mIoU, with only a marginal increase in memory usage.

Sparse Transformer Head. We compare several prediction heads, including a simple linear head, the transformer head proposed in [40], and our sparse transformer head in Tab. 5. As shown, the linear head achieves the highest geometry IoU of 36.8. We hypothesize that this occurs because the linear head is supervised by an explicit geometry loss, i.e., a cross-entropy loss between occupied and non-occupied voxels. Conversely, the transformer decoder formulates occupancy prediction as mask generation for semantic classes only, imposing no explicit supervision on occupied voxels. SparseOcc employs a linear layer for coarse segmentation to filter out non-occupied voxels, followed by a sparse transformer head for mask prediction. Therefore, it strikes an effective balance between these two approaches, achieving satisfactory performance with low training memory requirements.

3D Resolution. SparseOcc utilizes LSS [9] to lift 2D image features into a 3D volume, from which the 3D sparse representation is extracted. The resolution of the LSS output volume may influence performance. To investigate this efficiently, we use SparseOcc with a linear segmentation head. As shown in Tab. 6, scaling up the LSS output resolution by 2\(\times\) boosts mIoU from 11.8 to 12.3 but decreases geometry IoU by 0.4. We attribute this IoU drop to the insufficient number of sparse completion blocks to cover the entire scene at a higher resolution. This issue can be resolved by stacking an additional sparse completion block at the final layer of the 3D sparse encoder.

Geometry-Aware Sparse Representation. We explore the performance and speed gains contributed by each module of our geometry-aware sparse representation in Tab. 7. When all modules are disabled, the model degrades to a dense version, achieving competitive mIoU but at a reduced speed. When utilizing only sparse anchor voxel generation without scene completion, performance drops significantly due to compromised scene integrity. However, adding compact scene completion increases IoU, albeit with a speed reduction due to the processing of newly introduced voxels. By constraining semantic prediction to the completed scene, we achieve the optimal mIoU-FPS trade-off.

Sparse Anchor Voxel Generation. We compare the coarse-to-fine strategy with uniform sampling in Tab. 8. We find that the coarse-to-fine approach yields better performance, as it consistently focuses on the most promising occupancy priors.

Scene Completion Field Learning Strategies. We validate the design choices for SCF learning strategies in Tab. ¿tbl:tab:abl95dist95pred?. Given that outdoor scenes exhibit strong geometric anisotropy, the baseline model fails to accurately complete the scene with a single scalar SCF, achieving an mIoU of only 10.31. Introducing orthogonal decomposition significantly improves performance by 2.05 mIoU. This suggests that decomposing the SCF into independent orthogonal components simplifies the learning task. Furthermore, incorporating discretized learning yields an additional 0.87 mIoU gain. Moreover, replacing the sparse latent propagator, which employs indiscriminate feature propagation, with our geometry-guided propagation further increases mIoU to 13.23 and improves inference speed from 15.7 to 24.7 FPS, demonstrating the efficiency of our proposed SCF learning strategies.

Compact Scene Completion. The scene completion operation can be applied iteratively, where occupied voxels generated in one step serve as sparse anchors for the next, thereby progressively increasing the density of the predicted space. Additionally, the maximum SCF vector \(({s}_{\text{max}}^{\text{X-Y}},{s}_{\text{max}}^{\text{Z}})\) directly governs the range of voxel propagation. To investigate the impact of the iteration count and the maximum SCF distance, we conduct an ablation study across different settings, as shown in Tab. 9. When the iteration count is zero (i.e., no completion), the network degrades to a naive dense-to-sparse pruning baseline, destroying structural integrity and yielding suboptimal accuracy. Increasing the number of iterations initially improves IoU by recovering more complete geometry. However, we observe that applying the operation twice (2 iterations) with a large range of \((10, 3)\) reduces the mIoU to 12.08, likely due to error accumulation or geometric ambiguity arising from over-propagation. Consequently, to balance accuracy and efficiency, we adopt a single-iteration setting with the maximum SCF distance set to \((10, 3)\).

Image Backbone and Input Size. We study the influence of different image backbones and input sizes in Tab. 10. SparseOcc outperforms C-CONet on the nuScenes-Occupancy dataset even when trained with a smaller input size (\(704\times256\)) and a lighter backbone (ResNet-50), demonstrating the effectiveness of the architecture. Additionally, using a larger input size (\(1600\times900\)) improves mIoU for both C-CONet and SparseOcc due to the increased density of semantic features. However, the IoU of SparseOcc decreases at higher image resolutions. We attribute this to the erroneous hallucination of spatially empty voxels caused by over-dense 3D sparse features. Moreover, our SparseOcc++ achieves the best IoU by a significant margin (24.1 vs. 20.2 on nuScenes-Occupancy), even with a smaller image backbone and input size, demonstrating its superior design. Furthermore, a stronger image encoder, such as Swin Transformer [103], could further improve network performance.

Efficiency Analysis. The complexity of our sparse representations is expected to be approximately linear with respect to the number of non-empty voxels, as operations are restricted to feature-occupied regions. We compare the inference GPU memory and the number of non-empty voxels in Fig. 9. As observed, when scaling up the 3D resolution from \(2\times\) to \(1\times\), the dense representation-based method OccFormer [40] suffers from a steep increase in GPU memory usage. In contrast, SparseOcc and SparseOcc++ exhibit a linear increase relative to the number of non-empty voxels, confirming the superior efficiency of the 3D sparse representation.

6 Limitations↩︎

Despite its efficiency, SparseOcc++ has two primary limitations: (1) Geometric simplification: The orthogonal directional distance model favors locally convex or structurally simple geometries. For non-convex objects or intricate structures (e.g., irregular ‘vegetation’ in Fig. 8), our regression-based reconstruction may yield over-simplified results. (2) Long-range ambiguity: As a vision-based approach, accurately regressing distances for far-range objects remains ill-posed without direct depth measurements (e.g., long-range ‘drivable surface’ in Fig. [fig:vis95nusc]). Consequently, geometric precision degrades at a distance compared to multi-modal solutions, i.e., 24.1 (SparseOcc++) vs. 30.3 (OccGen [69]). Future research will focus on three directions: (1) incorporating temporal cues to resolve long-range depth ambiguity and ensure cross-frame consistency; (2) exploring adaptive anchor sampling to better capture complex, non-convex geometries; and (3) extending the framework to multi-modal fusion to leverage the geometric robustness of LiDAR while maintaining the efficiency of sparse representations.

7 Conclusion↩︎

In this paper, we present SparseOcc++, a comprehensive extension of our preliminary work, SparseOcc, designed to address efficiency bottlenecks in vision-based 3D occupancy prediction. We identify that existing sparse methods suffer from an entanglement of scene completion and semantic classification, which leads to computational redundancy in empty regions and boundary ambiguity. To address this limitation, we propose a novel geometry-aware sparse representation that explicitly disentangles these two processes. By reformulating scene completion as a scene completion field (SCF) learning task defined on sparse anchor voxels, we successfully reduce unnecessary computation while enhancing structural fidelity. To implement the SCF effectively, we introduce an orthogonal decomposition strategy and a discretized learning scheme for robust anisotropic modeling, followed by a geometry-guided voxel propagation mechanism that efficiently reconstructs the explicit volumetric structure. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that SparseOcc++ achieves state-of-the-art performance, showing superior IoU and faster inference speeds compared to both its predecessor and dense counterparts. We believe that this fully sparse geometry-aware representation offers a scalable and efficient direction for future 3D perception in autonomous driving.

References↩︎

---

1. Pin Tang, Guoqing Wang, Xiangxuan Ren, and Chao Ma are with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China. E-mail: {pin.tang, guoqing.wang, bunny_renxiangxuan, chaoma}sjtu.edu.cn?↩︎

2. Zhongdao Wang is with Huawei Noah’s Ark Lab, China. E-mail: wangzhongdao@huawei.com↩︎

3. Statistics derived from ground-truth occupancy labels in the first 10 sequences of the SemanticKITTI dataset.↩︎