1 Introduction↩︎

Comprehensive 3D scene understanding is an important problem in multimedia perception, requiring models to recover visual semantics, geometry, and physical space occupancy. Recent studies have advanced this goal through camera-based 3D semantic scene completion [1], [2], geometry-aware image-based 3D perception [3], [4], multimodal 3D scene reasoning [5], and open-vocabulary visual understanding [6]–[9]. This paper focuses on a more unconstrained setting: robust 3D scene understanding from sparse, unposed images. For robotics, embodied agents, and augmented reality, a reconstructed scene should support novel-view synthesis, open-vocabulary semantic querying, and explicit reasoning about which regions of 3D space are free or occupied. Recent advances in Neural Radiance Fields (NeRF) [10] and 3D Gaussian Splatting (3DGS) [11] have greatly improved renderable scene representations, yet many reconstruction pipelines still assume a known calibrated camera or camera poses estimated by Structure-from-Motion (SfM) [12]. Moreover, rendering-oriented representations alone do not directly provide a physically grounded volumetric description of the scene.

Recent feed-forward reconstruction methods have made important progress toward scalable 3D reconstruction from sparse or uncalibrated images. Generalizable gaussian methods such as pixelSplat [13], MVSplat [14], and VolSplat [15] learn image-to-gaussian priors for efficient sparse-view reconstruction, while unposed reconstruction models [16]–[21] reduce or remove the need for external camera calibration. Building on this trend, pose-free Gaussian methods such as LSM [22], AnySplat [23], UniForward [24], and Uni3R [25] predict renderable Gaussian representations from unposed multi-view inputs, and several of them further attach semantic features for open-vocabulary scene understanding. Concurrently, open-vocabulary 2D perception models such as LSeg [26], together with foundation models such as CLIP [27] and SAM [28], have fostered a new wave of semantic 3D methods [22], [29]–[33]. These approaches attempt to lift 2D semantic features into 3D space to achieve open-vocabulary understanding without extensive annotation. In particular, LSM [22] and Uni3R [25] demonstrate that pose-free semantic gaussians can support novel-view synthesis, open-vocabulary segmentation, and depth prediction in a single feed-forward framework. These advances establish semantic Gaussian splatting as a strong representation for scalable 3D scene reconstruction.

6pt

Despite this progress, existing pose-free semantic Gaussian representations remain largely surface-centric. Their Gaussian opacity and semantic features are optimized via image-space rendering losses, but the learned primitives are not explicitly constrained to represent free and occupied 3D space. Although recent works improve Gaussian reconstruction efficiency or compactness [33], [34], they do not enforce a physically plausible occupancy distribution during training. As summarized in Tab. 1, a straightforward post-hoc voxelization of predicted Gaussians can expose a volumetric field during evaluation, but it cannot correct the underlying Gaussian representation without volumetric feedback during training. As a result, a model can produce plausible RGB or semantic renderings while still placing ambiguous density in unobserved regions, producing floaters, hollow structures, or physically implausible layouts.

To address this limitation, we propose COVScene, an occupancy-grounded semantic Gaussian framework for pose-free 3D scene understanding. Rather than treating occupancy as an auxiliary branch or an offline conversion, COVScene differentiably lifts predicted semantic gaussians into a dense semantic occupancy field inside the training computation graph. The resulting volumetric field provides explicit free/occupied-space regularization, and its gradients directly update the same Gaussian opacity, geometry, and semantic features used for novel-view synthesis and open-vocabulary rendering. This closed-loop Gaussian-volume coupling makes the renderable representation more physically plausible while preserving the efficiency and flexibility of feed-forward semantic Gaussian reconstruction.

We evaluate COVScene on novel-view synthesis, open-vocabulary segmentation, and semantic occupancy prediction. Beyond standard comparisons, we include controlled baselines based on semantic Gaussians, post-processing Gaussian voxelization, and ablated COVScene variants. These experiments show that simply converting semantic Gaussians to occupancy at evaluation is insufficient, whereas training-time volumetric grounding substantially improves semantic occupancy and reduces geometric artifacts, achieving competitive rendering and segmentation performance.

Our contributions are summarized as follows:

• We propose COVScene, an occupancy-grounded semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. The lifted field is used during training, rather than as detached post-processing, so volumetric losses directly regularize Gaussian opacity, geometry, and semantic features.

• We introduce an occupancy regularization objective that encourages physically plausible free/occupied structure in weakly observed regions while preserving open-vocabulary rendering from the same Gaussian representation.

• A multi-task adaptive decoder is designed to fuse depth features with probabilistic cost-volume cues. Crucially, an occupancy-entropy regularization strategy is introduced to enforce bimodal physical priors, thereby resolving geometric inconsistencies and artifacts inherent in previous unposed 3DGS.

• Extensive evaluations on ScanNet [35] and ScanNet++ [36] demonstrate that COVScene achieves state-of-the-art (SOTA) performance across versatile tasks, including novel view synthesis, open-vocabulary semantic segmentation, and zero-shot 3D occupancy prediction.

2 Related Work↩︎

2.1 Generalizable 3D Reconstruction Methods↩︎

NeRF [10] and 3DGS [11] have substantially advanced image-based 3D reconstruction and novel view synthesis [16], [18], [37], [38]. However, many early radiance-field and Gaussian-based pipelines depend on per-scene optimization, which limits their efficiency and scalability. Subsequent acceleration techniques [39], [40] reduce the optimization cost, but the reconstruction process still often requires iterative fitting for each scene. To improve generalization across scenes, feed-forward methods [13], [14], [37] learn reconstruction priors from large-scale data and predict scene representations in a single network pass. In particular, PixelSplat [13] and MVSplat [14] use epipolar geometry [41] and cost volumes [42] to infer depth and Gaussian attributes from sparse image pairs. Later variants, including MVSplat360 [43], DrivingForward [44], and YoNoSplat [45], extend feed-forward Gaussian reconstruction to broader camera distributions and more complex scenes. Another line of work reduces the dependence on known camera parameters. The DUST3R series [19], [46] estimates dense correspondences and camera geometry from image collections, and recent pose-free methods [16], [16]–[18], [47], [48] further couple camera estimation with feed-forward 3D reconstruction. These methods make reconstruction from unposed images increasingly practical, but most of them remain focused on geometry and rendering rather than dense semantic occupancy. COVScene follows the pose-free feed-forward setting, while introducing volumetric semantic occupancy as an explicit training-time constraint on the Gaussian representation.

2.2 Open-Vocabulary 3D Scene Understanding↩︎

Beyond geometric reconstruction, recent work has increasingly studied the semantic and structural understanding of 3D scenes [49]–[63]. Early open-vocabulary 3D methods incorporate features from CLIP [27] into NeRF fields, as demonstrated by LERF [49]. This idea has been extended to 3DGS to support efficient rendering and queryable semantic fields. For example, GARField [51] distills masks from SAM [28] into Gaussian primitives, LangSplat [53] constructs a scene-wise language field through feature auto-encoding, and Gaussian Grouping [64] attaches identity codes for instance-level grouping. Further extensions, including OpenSplat3D [56] and 4D LangSplat [60], support promptable segmentation and temporally coherent language fields. OV-NeRF [65] improves cross-view semantic consistency through semantic field distillation, while MaskField [66] decomposes SAM mask features in Gaussian Splatting for efficient 3D semantic segmentation. These methods demonstrate the value of lifting 2D semantic priors into 3D, but most of them assume calibrated inputs, rely on per-scene optimization, or remain centered on surface-level semantic fields.

Recent feed-forward and pose-free methods move closer to scalable semantic 3D scene understanding. LSM [22] predicts semantic 3D representations from unposed images, while AnySplat [23] learns feed-forward Gaussian reconstruction from unconstrained views. UniForward [24], FLEG [67], and Uni3R [25] further extend this direction by coupling feed-forward Gaussian reconstruction with semantic or language-aligned scene fields. In parallel, semantic Gaussian methods [29]–[33] attach distilled visual-language features, learned feature fields, or language embeddings to 3D Gaussian primitives for open-vocabulary querying and segmentation. These studies are highly relevant to our setting because they reduce the dependence on calibrated cameras or enable semantic reasoning over Gaussian representations. However, they still primarily supervise renderable surfaces and semantic fields through image-space objectives, and they do not explicitly couple the Gaussian representation with a dense semantic occupancy field during training. In contrast, COVScene derives semantic occupancy from pose-free Gaussian primitives through differentiable volumetric lifting, so that rendering, semantic querying, and physical space reasoning are optimized within a single coupled representation.

2.3 Semantic Occupancy Prediction↩︎

Semantic occupancy prediction provides a dense representation for reasoning about both scene geometry and semantic layout. Large-scale benchmarks, including MonoScene [68], SurroundOcc [69], Occ3D [70], OpenOccupancy [71], SemanticKITTI [72] and Nuplan-Occ [73], have promoted the development of robust occupancy models through standardized data and evaluation protocols. Fully supervised outdoor methods define several influential design choices [50], [74], [75]. Transformer-based models such as TPVFormer [74], HTCL [76], VoxFormer [77], and OccFormer [75] lift image features into dense or semi-dense 3D representations, including voxel grids and tri-plane features. To reduce the redundancy of dense voxels, GaussianFormer [78] and GaussianFormer-2 [79] represent occupancy with sparse 3D Gaussians. CausalOcc [80] further improves semantic consistency by constraining gradient flow in modular lifting pipelines. Because exhaustive 3D labels are expensive, self-supervised and rendering-based alternatives such as SelfOcc [81], GaussianOcc [82], SimpleOcc [83], and OccNeRF [84] reduce the reliance on dense annotations. Indoor semantic occupancy remains less explored. MonoScene [68] and ISO [85] infer dense geometry and semantics from limited monocular observations, while EmbodiedOcc [86], EmbodiedOcc\(++\) [87], and SliceOcc [88] address occlusion, active perception, and efficient volumetric processing. However, these methods are usually built on voxel-centric formulations and often require stronger supervision or posed inputs. COVScene differs by deriving semantic occupancy directly from pose-free Gaussian primitives through differentiable volumetric lifting, which enables volumetric regularization without introducing a detached occupancy branch.

3 Methodology↩︎

3.1 Problem Setup↩︎

Given a set of \(N\) uncalibrated and unposed RGB images \(\mathcal{I} = \{I_i\}_{i=1}^N\), where each \(I_i \in \mathbb{R}^{H \times W \times 3}\) captures a sparse observation of a 3D scene, our goal is to recover a globally consistent 3D representation for novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction without relying on external Structure-from-Motion (SfM) pipelines. Recent feed-forward 3D Gaussian Splatting (3DGS) methods, such as AnySplat [23], LSM [22], Uni3R [25] and MVSplat [14], have shown that multi-view image features can be efficiently converted into renderable Gaussian primitives. However, these methods mainly operate on discrete, surface-level primitives and supervise them through image-space rendering or semantic losses. Even when sparse 2D geometric priors are used, the representation lacks explicit continuous 3D volumetric constraints for weakly observed regions. As a result, sparse and unposed inputs can still lead to semi-transparent “floaters”, hollow structures, or physically implausible occlusion geometry.

To address this limitation, we propose a Volume-Regularized 3DGS framework that explicitly couples renderable Gaussian primitives with a dense semantic occupancy field during training. The key design is to treat occupancy not as an auxiliary prediction head or a detached post-processing result, but as a differentiable volume lifted from the predicted Gaussian representation. This closed-loop Gaussian-volume coupling allows volumetric losses to update the same opacity, geometry, covariance, and semantic features that are used for novel view synthesis and open-vocabulary segmentation. Therefore, the lifted occupancy field acts as a dense physical prior over free and occupied space, while the underlying Gaussian representation retains the rendering efficiency and semantic flexibility of 3DGS. Formally, as shown in Fig. 2, COVScene first predicts semantic Gaussian primitives from unposed input images and then derives semantic occupancy through differentiable volumetric lifting: \[\mathcal{G} = F_\theta(\mathcal{I}), \quad \mathcal{O} = \mathcal{L}_{\mathrm{lift}}(\mathcal{G}).\] Here, \(\mathcal{G} = { (\mu_g, \sigma_g, r_g, s_g, c_g, f_g) }{g=1}^G\) denotes the predicted 3D Gaussian set; \(\mu_g\), \(\sigma_g\), \(r_g\), \(s_g\), and \(c_g\) represent the center, opacity, rotation, scale, and color of primitive \(g\), respectively. \(f_g \in \mathbb{R}^d\) encodes open-vocabulary semantic features aligned with 2D vision-language models. The dense semantic occupancy field \(\mathcal{O}\) is derived from \(\mathcal{G}\) alone through \(\mathcal{L}_{\mathrm{lift}}\); thus, volumetric regularization on \(\mathcal{O}\) propagates gradients back to \(\mathcal{G}\) rather than being confined to a separate occupancy branch. This makes the lifted occupancy a native training-time representation of the unified model and distinguishes COVScene from post-processed Gaussian voxelization.

3.2 Geometry Encoding and Multi-Task Decoding↩︎

Recovering a consistent Gaussian representation from uncalibrated views requires the model to infer camera geometry and scene structure directly from images. To avoid dependence on external Structure-from-Motion pipelines, COVScene adopts a semantic-aware Geometry Transformer as the shared feature encoder. Its geometry-oriented attention blocks are initialized from the pretrained VGGT model [20], while additional semantic fusion layers and task-specific prediction heads are introduced to encode language-aligned features and Gaussian attributes. The encoder is designed to capture both local visual evidence within each view and cross-view correspondences across the unposed image set, so that downstream prediction heads can estimate camera parameters, depth, and semantic Gaussian attributes in a unified feed-forward pass.

Given the unposed input images \(\mathcal{I} = \{I_i\}_{i=1}^{N}\), the Geometry Transformer maps them into multi-view latent tokens: \[\{Z_i\}_{i=1}^{N} = \Phi_{\mathrm{geo}}(\mathcal{I}),\] where each \(Z_i\) contains dense image tokens together with global tokens for camera and scene-level reasoning. The encoder first embeds image patches and then alternates between intra-view self-attention and inter-view attention. Intra-view attention models local image context, while inter-view attention establishes geometric correspondences across different views. The resulting 3D-aware tokens provide the geometric prior needed for reconstructing scenes from uncalibrated images, without assuming known camera poses or an external SfM reconstruction.

These latent tokens are decoded by a Dense Prediction Transformer (DPT)-based decoder with three specialized prediction heads. The Camera Head decodes the global camera tokens to predict relative camera extrinsics and intrinsics. The Depth Head estimates per-pixel depth maps from the dense image tokens; combining the predicted depths with the inferred camera parameters gives the 3D centers \(\{\mu_g\}\) of the Gaussian primitives through unprojection. The Semantic Gaussian Head fuses Transformer features with high-resolution features from a pretrained 2D semantic encoder to predict opacity \(\sigma_g\), rotation \(r_g\), scale \(s_g\), color \(c_g\), and the open-vocabulary semantic feature \(f_g\) for each primitive. Because \(f_g\) is attached to the same 3D Gaussian feature field used for rendering, source and target semantic maps are rendered from a shared representation, which improves cross-view semantic consistency and supports open-vocabulary querying.

3.3 Differentiable Volumetric Lifting for Semantic Occupancy↩︎

The differentiable volumetric lifting module converts the predicted Gaussian set into a dense semantic occupancy field inside the training computation graph. Standard 3DGS represents a scene using discrete surface primitives, which are efficient to render but do not explicitly model empty space. This limitation is problematic for sparse and unposed inputs because ambiguous density can be assigned to weakly observed regions while still satisfying image-space supervision. To introduce volumetric reasoning, we define a voxelized 3D domain \(V \subset \Omega \subset \mathbb{R}^{3}\) and evaluate, at each voxel center \(\mathbf{x} \in V\), an occupancy probability \(O(\mathbf{x}) \in [0,1]\) together with a semantic feature \(F(\mathbf{x}) \in \mathbb{R}^{d}\).

Given the Gaussian set \(\mathcal{G}\), each primitive contributes a local density to spatial position \(\mathbf{x}\). For a Gaussian primitive \(g\), this density is computed as \[\tau_g(\mathbf{x}) = \sigma_g \cdot \exp\Big(-\frac{1}{2} (\mathbf{x} - \mu_g)^\top \Sigma_g^{-1} (\mathbf{x} - \mu_g)\Big),\] where \(\Sigma_g\) is the covariance matrix parameterized by scale \(s_g\) and rotation \(r_g\). We define \(\mathcal{N}(\mathbf{x})\) as the local support of \(\mathbf{x}\), containing the Gaussians that fall within a truncation radius of the voxel center, with a top-\(k\) cap when more candidates are present. Gaussians outside this local support have negligible density and are ignored for efficiency. The voxel-size factor and density scale are absorbed into \(\tau_g(\mathbf{x})\). The occupancy probability is then obtained by accumulating the densities from \(\mathcal{N}(\mathbf{x})\) using the standard volumetric transmittance form:

\[O(\mathbf{x}) = 1 - \exp\Big(-\sum_{g \in \mathcal{N}(\mathbf{x})} \tau_g(\mathbf{x})\Big).\] The corresponding voxel-wise semantic feature is computed by density-weighted aggregation of the Gaussian semantic features: \[F(\mathbf{x}) = \frac{\sum_{g \in \mathcal{N}(\mathbf{x})} \tau_g(\mathbf{x}) f_g}{\sum_{g \in \mathcal{N}(\mathbf{x})} \tau_g(\mathbf{x}) + \epsilon},\] where \(\epsilon\) is a small constant for numerical stability. This construction maps \(\mathcal{G}\) to a dense field \((O,F)\) without introducing an auxiliary occupancy head. At inference time, semantic occupancy is decoded from the same voxel feature field. Given a text or class vocabulary \(\mathcal{C}\) with normalized embeddings \(\{e_c\}_{c \in \mathcal{C}}\) in the same semantic space, each occupied voxel is assigned an open-vocabulary semantic label by \[\hat{s}(\mathbf{x}) = \arg\max_{c \in \mathcal{C}} \cos\big(F(\mathbf{x}), e_c\big), \quad \mathrm{for}\;O(\mathbf{x}) > \eta,\] where \(\eta\) is the occupancy threshold, and voxels with \(O(\mathbf{x}) \leq \eta\) are treated as free space.

Because the lifting operation is fully differentiable, any volumetric regularization applied to the dense field propagates gradients back to the same Gaussian opacity, centers, covariance, and semantic features used for rendering and open-vocabulary segmentation. This property distinguishes COVScene from evaluation-only post-voxelization, in which trained Gaussians are converted to voxels after optimization, and no volumetric gradient is used during training. In COVScene, the lifted occupancy field is therefore a training-time geometric constraint on the Gaussian representation, while voxel semantics are obtained from the language-aligned feature field at inference time rather than from direct voxel-level semantic supervision.

3.4 Physics-Informed Optimization via Entropy Regularization↩︎

Optimizing COVScene requires supervision that can constrain unposed 3D reconstruction without explicit 3D voxel labels. Image-space rendering losses provide strong appearance supervision, but they do not sufficiently constrain geometry in occluded or weakly observed regions. We therefore combine 2D rendering supervision, feature-level semantic alignment, geometric foundation distillation, and occupancy entropy regularization into a single end-to-end objective: \[\begin{align} \mathcal{L}_{\mathrm{total}} = &\lambda_{\mathrm{photo}} \mathcal{L}_{\mathrm{photo}} + \lambda_{\mathrm{sem}} \mathcal{L}_{\mathrm{sem}} \\ &+ \lambda_{\mathrm{distill}} \mathcal{L}_{\mathrm{distill}} + \lambda_{\mathrm{occ\_reg}} \mathcal{L}_{\mathrm{occ\_reg}}. \end{align}\] where the balancing weights are empirically set to \(\lambda_{\mathrm{photo}} = 1.0\), \(\lambda_{\mathrm{sem}} = 0.3\), \(\lambda_{\mathrm{distill}} = 0.7\), and \(\lambda_{\mathrm{occ\_reg}} = 0.25\).

Geometric Foundation Distillation (\(\mathcal{L}_{\mathrm{distill}}\)) To bootstrap geometry in the strictly unposed setting, we distill camera and spatial cues from a frozen geometry foundation model, such as VGGT [20]. The frozen teacher is used only to generate pseudo targets, while the trainable COVScene encoder includes additional semantic fusion layers and task-specific heads. Let \(\Pi\), \(D\), and \(P\) denote the predicted camera parameters, depth maps, and 3D point maps, and let \(\tilde{\Pi}\), \(\tilde{D}\), and \(\tilde{P}\) denote the corresponding pseudo targets from the frozen teacher. The distillation loss is defined as \[\mathcal{L}_{\mathrm{distill}} = \gamma_{\mathrm{pose}} \|\Pi - \tilde{\Pi}\|_1 + \gamma_{\mathrm{depth}} \|D - \tilde{D}\|_1 + \gamma_{\mathrm{point}} \|P - \tilde{P}\|_1,\] where \(\gamma_{\mathrm{pose}}\), \(\gamma_{\mathrm{depth}}\), and \(\gamma_{\mathrm{point}}\) balance the pose, depth, and point-map terms. This guidance provides an initial multi-view geometric scaffold, accelerates convergence, and reduces scale ambiguity without relying on SfM point clouds.

Photometric and Semantic Supervision (\(\mathcal{L}_{\mathrm{photo}}\), \(\mathcal{L}_{\mathrm{sem}}\)) We differentially splat the predicted Gaussians onto target views. The photometric loss \(\mathcal{L}_{\mathrm{photo}}\) combines the \(\mathcal{L}_1\) and LPIPS distances between rendered and ground-truth RGB images. For semantic supervision, we render the Gaussian semantic features \(f_g\) into 2D feature maps and align them with pseudo target features from a frozen vision-language model, such as CLIP [27], using a cosine similarity loss \(\mathcal{L}_{\mathrm{sem}}\). This feature-level supervision avoids fixed-category pseudo-labels and preserves open-vocabulary querying via text-feature matching.

Occupancy Entropy Regularization (\(\mathcal{L}_{\mathrm{occ\_reg}}\)) Although rendering supervision and foundation distillation provide the main geometric constraints, regions with limited observations can still yield ambiguous occupancy values in the lifted field. The entropy term is therefore not used as direct voxel supervision; instead, it regularizes the occupancy field induced by the learned Gaussian geometry. We impose a bimodal prior on the lifted field so that each spatial location tends to be either free, with \(O(\mathbf{x}) \to 0\), or occupied, with \(O(\mathbf{x}) \to 1\). We implement this prior by minimizing the entropy of the predicted occupancy field: \[\begin{align} \mathcal{L}_{\mathrm{occ\_reg}} = - \frac{1}{|V|} \sum_{\mathbf{x} \in V} \big[ O(\mathbf{x}) \log \big(O(\mathbf{x}) + \epsilon\big) \\ {}+ \big(1 - O(\mathbf{x})\big) \log \big(1 - O(\mathbf{x}) + \epsilon\big) \big], \end{align}\] where \(\epsilon\) denotes a small constant for numerical stability. The entropy reaches its maximum when \(O(\mathbf{x}) \approx 0.5\) and decreases as the prediction approaches either 0 or 1. Therefore, conditioned on the Gaussian field learned from rendering, semantic alignment, and geometric distillation, \(\mathcal{L}_{\mathrm{occ\_reg}}\) penalizes residual uncertain density and sharpens the free/occupied decision, as shown in Fig. 3. Since \(O(\mathbf{x})\) is lifted from the Gaussian representation, the entropy gradients are propagated through \(\mathcal{L}_{\mathrm{lift}}\) to the corresponding Gaussian opacity and geometry parameters. This training-time volumetric feedback reduces ambiguous floating density and differs from post-voxelization baselines, where occupancy is computed only after Gaussian optimization.

4 Experiments↩︎

Implementation Details We evaluate COVScene on three downstream tasks: novel view synthesis (NVS), 3D open-vocabulary segmentation (OVS), and semantic occupancy prediction. Following the data filtering protocol of LSM [22], we train the model on ScanNet [35] and ScanNet++ [36], yielding approximately 1,500 valid training scenes, and evaluate it on 40 unseen ScanNet scenes with additional validation on ScanNet++. Source views are used as context inputs, while target views are disjoint held-out views used only for evaluation and are not used for training, pseudo-label generation, or model input. For occupancy evaluation, semantic occupancy annotations are constructed following the ISO protocol [85] when native voxel annotations are unavailable, and dataset categories are used only for evaluation rather than as category-level training labels.

For NVS and OVS, we compare COVScene with representative feed-forward and semantic Gaussian baselines, including AnySplat [23], LSM [22], Feature-3DGS [30], and the 2D open-vocabulary model LSeg [26]. Since LSM is originally designed for two-view inputs based on DUSt3R [19], we extend it to multi-view settings using the global alignment strategy of DUSt3R for fair comparison across different input view counts. For semantic occupancy, we compare with fully supervised methods, including MonoScene [68], ISO [85], and EmbodiedOcc [86], as well as the self-supervised baseline SelfOcc [81]. NVS is evaluated by PSNR, SSIM, and LPIPS, OVS by mIoU and mAcc, and semantic occupancy by IoU and mIoU.

The semantic-aware Geometry Transformer uses the same layer count as VGGT [20]. Geometry-oriented blocks are initialized from VGGT and fine-tuned during training, while the newly introduced semantic fusion layers and task-specific heads are initialized randomly. We train all models with the objective in Sec. 3. Additional reproducibility details, including volumetric lifting hyperparameters, grid resolutions, loss weights, and optimization settings, are provided in the supplementary material.

4.1 Multi-Task Comparison↩︎

Open-Vocabulary Segmentation As shown in Fig. 4 and Tab. 2, COVScene achieves strong open-vocabulary segmentation performance across both source and target views. Compared with LSeg, which performs 2D open-vocabulary segmentation independently for each view, COVScene produces competitive or better mIoU while maintaining a single 3D semantic Gaussian field. The advantage becomes more visible as the number of input views increases: at \(32\) views, COVScene reaches \(55.90\) target-view mIoU and \(79.40\) target-view accuracy. In contrast, LSM does not benefit consistently from additional views, with source-view mIoU decreasing from \(51.40\) under \(2\) views to \(49.50\) under \(32\) views. These results suggest that the shared 3D semantic representation improves cross-view semantic consistency under multi-view inputs.

Novel View Synthesis As shown in Fig. 5 and Tab. 2, COVScene preserves competitive rendering quality while adding semantic and occupancy reasoning to the Gaussian representation. Under the \(2\)-view setting, COVScene obtains the best PSNR and SSIM among the compared methods, with slightly higher LPIPS than LSM. Under \(8\) and \(16\) views, AnySplat remains stronger in PSNR and SSIM because it is optimized specifically for NVS, whereas COVScene obtains the best LPIPS. With \(32\) input views, COVScene achieves the best PSNR, SSIM, and LPIPS among all baselines. These results indicate that the proposed representation does not trade off rendering quality for semantic and occupancy prediction, and that the unified model can effectively use additional views.

Semantic Occupancy Prediction Tab. 3 compares COVScene with supervised occupancy methods and self-supervised baselines. A clear gap remains between methods trained with explicit 3D labels and those trained without direct voxel supervision, which reflects the difficulty of semantic occupancy prediction under weak supervision. Within the no-3D-label setting, COVScene improves over SelfOcc from \(7.68\) to \(18.32\) IoU and from \(6.93\) to \(17.78\) mIoU. The gains appear across most semantic classes, including structural regions such as walls and windows as well as object categories such as chairs, beds, and sofas. Qualitative zero-shot results in Fig. 6 further illustrate that the learned representation can infer plausible occupancy layouts for images without available 3D ground-truth annotations.

4.2 Ablation Study↩︎

4pt

5pt

We perform ablation studies to validate the role of training-time volumetric coupling and the proposed components. Tab. 4 compares COVScene with evaluation-only post-voxelization baselines under the 2-view setting. Directly converting trained semantic Gaussians to occupancy improves over methods without occupancy output, but remains clearly weaker than COVScene. Compared with AnySplat+sem.+post-voxel. and Uni3R+post-voxel., COVScene improves Occ. mIoU by \(9.89\) and \(4.58\), respectively, while avoiding the extra post-voxelization time. This result supports that the occupancy gain comes from training-time Gaussian-volume coupling rather than a detached conversion step.

We further ablate key components of COVScene under the 8-view setting in Tab. 5.

Impact of Pose Distillation Removing pose distillation tests its role in unposed reconstruction. Without this guidance, the network lacks geometry constraints to establish initial multi-view correspondences. Consequently, scene geometry collapses, severely degrading novel view synthesis (PSNR decreases to 12.50) and semantic segmentation (mIoU drops to 0.224). This indicates that distilling pose priors from geometric foundation models is essential for optimizing uncalibrated scenes.

Effectiveness of Differentiable Lifting This module bridges discrete Gaussians and the continuous occupancy field. Removing it restricts supervision strictly to 2D projections without 3D volumetric constraints. This degrades 3D scene understanding (mIoU drops from 0.552 to 0.376) and heavily impacts rendering quality (PSNR drops from 23.15 to 20.34). This dual degradation proves that mapping Gaussians to a continuous volume allows spatial constraints to back-propagate and regularize the discrete primitives, maintaining a unified architecture.

Occupancy Entropy Regularization Finally, we evaluate the bimodal physical prior (\(\mathcal{L}_{\mathrm{occ\_reg}}\)). Removing this loss fills unobserved free space with artifacts, as the lack of penalty allows the network to satisfy 2D photometric losses using imprecise geometries. Omitting this regularization decreases rendering PSNR by 0.64 and segmentation mIoU by \(3.4\%\). This confirms that enforcing structural certainty successfully suppresses floaters and sharpens semantic boundaries.

5 Conclusion and Future Work↩︎

In this paper, we presented COVScene, a pose-free feed-forward framework for comprehensive 3D scene understanding from sparse, unposed images. The central idea is to couple renderable semantic Gaussians with a dense occupancy field through differentiable volumetric lifting, so that volumetric regularization provides training-time feedback to Gaussian opacity, geometry, and semantic features rather than acting as evaluation-only voxelization. This coupled representation supports novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction without direct voxel-level supervision. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger no-3D-label semantic occupancy prediction than the self-supervised baseline. The ablation results further indicate that geometric distillation and differentiable volumetric lifting are important for stable unposed reconstruction and occupancy-aware scene modeling.

Limitations and Future Work. The current framework is evaluated mainly on static indoor scenes, and jointly optimizing rendering, semantic alignment, geometric distillation, and volumetric regularization remains computationally demanding. Future work will focus on improving inference and training efficiency, extending the volumetric coupling to dynamic 4D scenes, and studying online deployment in embodied perception systems.

References↩︎