1 Introduction↩︎

Privacy-preserving perception has received growing attention as intelligent systems are increasingly deployed in real-world environments [1]. This requirement is particularly important for indoor 3D scene understanding, where embodied systems and robots are expected to operate in homes, offices, hospitals, and other privacy-sensitive spaces. However, the RGB observations commonly used by these systems may capture sensitive visual information such as human faces, computer screens, documents, and personal belongings, raising privacy concerns [2].

Open-vocabulary 3D semantic segmentation is a key capability for general-purpose 3D scene understanding, enabling dense semantic parsing of unseen environments beyond a fixed label set. This flexibility is important for downstream applications such as robotic navigation, embodied agents, and human-scene interaction [3], [4], where the set of relevant categories may not be known in advance. However, privacy-preserving perception remains underexplored in this setting. Existing methods typically obtain rich semantic cues from RGB information [5]–[7]. Although such cues have significantly advanced open-vocabulary 3D recognition, they make inference dependent on appearance observations from the test scene, which may conflict with privacy-sensitive applications.

Depth-only 3D geometry offers a natural sensing modality for privacy-preserving 3D perception [8], [9]. In this setting, the perception system is restricted to depth observations and depth-derived geometry, without access to real RGB images, point-level colors, textures, or image features from the test scene. Such geometry-only input preserves the spatial structure needed for 3D scene understanding while reducing usage to privacy-sensitive appearance information. However, this setting also deprives open-vocabulary models of the texture and color cues that often provide critical semantic evidence, making depth-only open-vocabulary predictions more uncertain and less reliable.

The key challenge is that uncertainty under RGB-free inference is not uniformly distributed across the scene. Some regions remain stable because their geometry provides sufficient semantic evidence, while others become ambiguous due to missing appearance cues. This structured uncertainty provides a useful signal for inference-time refinement: reliable predictions should be preserved, whereas uncertain predictions should be allowed to change under geometric and semantic regularization. Since the privacy-preserving setting does not provide RGB observations or additional labeled data, such refinement should be performed at test time without updating the underlying model.

We therefore propose UTTO, a model-agnostic uncertainty-guided test-time optimization framework for depth-only open-vocabulary 3D semantic segmentation. Starting from a frozen open-vocabulary 3D model, UTTO optimizes only the per-point class distributions at inference time, making it applicable to different 3D backbones without retraining. It first constructs multiple test-time predictions through label-preserving augmentations and estimates per-point uncertainty from their agreement. The resulting uncertainty map allows the optimization to preserve stable predictions while selectively refining uncertain ones. To further improve open-vocabulary recognition, we introduce two complementary foundation-model priors: text-side debiasing to reduce category-prototype bias, and feature-space semantic consistency to connect semantically related points beyond local geometry. The whole framework is used only at inference time, without model training, ground-truth labels, or real RGB observations.

Fig. 1 illustrates this privacy-preserving setting from an embodied application perspective and shows how UTTO supports downstream semantic goal navigation from geometry-only input. Our contributions are as follows:

• We formulate privacy-preserving open-vocabulary 3D semantic segmentation under a depth-only sensing constraint, where the perception system has access only to depth-derived geometry and no RGB observations from the test scene.

• We propose UTTO, a model-agnostic uncertainty-guided test-time optimization framework that refines predictions from frozen open-vocabulary 3D backbones without additional training, ground-truth labels, or real RGB input.

• We identify structured test-time uncertainty as an effective reliability signal for this constrained regime, where open-vocabulary inference must operate without RGB appearance cues or closed-set supervised training. This insight enables UTTO to preserve stable predictions while selectively refining unreliable regions with geometric and foundation-model priors.

Experiments on ScanNet20, ScanNet40, and ScanNet200 validate the effectiveness of UTTO across different backbones and representative RGB-free baselines, and a privacy-preserving semantic goal grounding case study further illustrates its downstream applicability.

2 Related Work↩︎

2.1 Open-Vocabulary 3D Semantic Segmentation↩︎

Open-vocabulary 3D semantic segmentation is an important step toward general-purpose 3D scene understanding, as it enables models to parse 3D environments beyond a fixed set of predefined categories [10]. Such an ability is crucial for downstream applications including robotic navigation, AR/VR, and human-scene interaction.

RGB-Dependent Open-Vocabulary 3D Segmentation. Existing open-vocabulary 3D semantic segmentation methods commonly obtain semantic cues from appearance observations available at inference time. One representative strategy is to extract dense features from posed RGB or RGB-D images using 2D open-vocabulary models and then lift these features to 3D points through camera projection and multi-view fusion. OpenScene [8] is a representative example: its 2D fusion and 2D-3D ensemble settings back-project per-pixel open-vocabulary features to 3D points and combine them with 3D representations for semantic prediction. Recent works further improve RGB-to-3D transfer using structured 2D priors. Diff2Scene [6] leverages text-to-image diffusion representations and 2D mask embeddings as semantic queries for open-vocabulary 3D semantic segmentation. XMask3D [11] introduces cross-modal mask reasoning to align 3D representations with a 2D-text embedding space at the mask level, improving fine-grained segmentation boundaries.

Another line of work directly operates on colored 3D inputs. Instead of explicitly lifting dense image features at inference time, these methods consume point clouds whose points are associated with RGB values. For example, Mosaic3D [5] builds a foundation model for open-vocabulary 3D semantic and instance segmentation by exploiting colored 3D inputs. Similarly, language-driven 3D scene understanding frameworks such as PLA [12] are applied to colored point-cloud inputs, where RGB channels provide additional appearance cues for point-level recognition. Although these RGB-dependent methods have significantly advanced open-vocabulary 3D semantic segmentation, their inference pipelines require real RGB images, point-level colors, or image-derived features from the test scene. This dependence may conflict with privacy-sensitive applications where RGB observations can expose faces, screens, documents, or personal belongings.

RGB-Free Open-Vocabulary 3D Segmentation. RGB-free inference provides a more privacy-compatible alternative by avoiding real RGB observations from the test scene and relying on depth-derived geometry or pure 3D structure. Several recent works have explored pure-geometry inference for open-vocabulary 3D scene understanding. OpenScene3D [8] introduces a 3D-distilled setting, where knowledge from 2D open-vocabulary models is distilled into a 3D network that predicts dense point features from 3D point positions at inference time. Building on this direction, GGSD [13] improves distilled 3D representations by incorporating geometric priors and self-distillation to better exploit the representational advantages of 3D point clouds. OpenUrban3D [14] studies annotation-free open-vocabulary semantic segmentation for large-scale urban point clouds and generates semantic features directly from raw point clouds without requiring aligned multi-view images at inference time. These works show that open-vocabulary 3D segmentation can be performed without online RGB feature fusion, making them more compatible with privacy-preserving deployment.

However, RGB-free open-vocabulary 3D segmentation remains harder because geometry alone often lacks the texture and color cues needed to distinguish geometrically similar categories. As a result, predictions under depth-only inference can become spatially inconsistent and unreliable. In contrast to existing RGB-free methods that directly use the frozen model predictions, we treat this reliability variation as an optimization signal and refine depth-only open-vocabulary predictions at test time without using real RGB observations, additional training, or ground-truth labels.

2.2 Uncertainty-Guided Refinement and Test-Time Optimization↩︎

Uncertainty-Guided Refinement. Uncertainty estimation is important for 3D semantic segmentation, where point-level predictions can be unreliable due to sparse observations, occlusion, and geometric ambiguity. Existing works have exploited uncertainty in different ways. In closed-set LiDAR or point-cloud segmentation, uncertainty is often used to assess prediction reliability and improve robustness or calibration [15]. In label-efficient 3D segmentation, it is commonly used for pseudo-label generation, pseudo-label filtering, or active sample selection [16], [17]. Recent open-world and open-vocabulary 3D methods further use uncertainty to identify unknown regions or to balance heterogeneous semantic cues from multiple foundation models [18], [19]. However, uncertainty remains largely unexplored as an inference-time reliability signal for depth-only open-vocabulary 3D semantic segmentation. This setting is particularly challenging because RGB appearance cues are unavailable by design, making semantic predictions more ambiguous and spatially uneven in reliability.

Test-Time Optimization. Test-time optimization and inference-time refinement aim to improve scene-level predictions without collecting labeled target data. In open-vocabulary 3D segmentation, existing methods often refine 3D masks by leveraging class-agnostic 3D proposals, multi-view vision-language features, or mask-level feature aggregation [20], [21]. Other training-free methods use foundation models to refine 3D prompts, merge superpoints, or aggregate open-world tags for open-vocabulary 3D scene understanding [22], [23]. These methods demonstrate the value of inference-time refinement, but they are mostly built on RGB-derived observations, image-level foundation-model features, or mask proposals that depend on appearance-rich inputs. In contrast, UTTO performs uncertainty-guided test-time optimization directly on depth-only geometry-based predictions. It optimizes per-point class distributions from frozen open-vocabulary 3D backbones, preserving reliable regions while refining uncertain ones under geometric and foundation-model priors, without using real RGB observations from the test scene.

3 Our Method↩︎

We propose an uncertainty-guided test-time optimization framework for depth-only open-vocabulary 3D semantic segmentation. Given a geometry-only scene, we keep all backbone and foundation encoders frozen and optimize only the per-vertex class distributions at inference time.

As illustrated in Fig. 2, our framework consists of three components. First, label-preserving test-time perturbations induce prediction variation, from which we estimate vertex-level uncertainty and derive reliability weights for the data objective. Second, label-free prompt-ensemble text prototypes reduce text-side bias in the unary predictions. Third, coordinate-space and feature-space consistency regularization refines the optimized distributions by enforcing local spatial smoothness and semantic agreement. The entire procedure is performed per scene without model training, RGB input, or ground-truth labels.

3.1 Uncertainty-Guided Inference Optimization↩︎

Under the depth-only privacy-constrained setting, we leverage prediction variation induced by label-preserving test-time perturbations as a label-free uncertainty signal, and convert this uncertainty into point-wise reliability weights for inference-time optimization.

To obtain multiple predictions without supervision or model training, we perform test-time augmentation on the input scene. Let \(\mathcal{A}=\{a_m\}_{m=1}^{M}\) denote the set of semantics-preserving augmentations, such as rotations and voxel-grid phase shifts. These augmentations preserve scene semantics while inducing prediction variation from the frozen model.

Given a geometry-only scene \(\mathcal{S}\) with \(V\) vertices and \(C\) semantic categories, the frozen open-vocabulary 3D model produces a per-vertex class distribution for each augmented input: \[\small P_{(m)} = f_{\theta}(a_m(\mathcal{S}); T) \in \mathbb{R}^{V\times C}, \;m=1,\ldots,M,\] where \(f_{\theta}\) denotes the frozen 3D model, and \(T\) denotes the text-prototype classifier, which will be described in Sec. 3.2. Each prediction is mapped back to the original vertex space before aggregation. All predictions are generated at inference time, and no model parameters are updated.

We then estimate vertex-level reliability from the agreement among the test-time predictions. For each augmented prediction, we first obtain the hard semantic prediction: \[\small \hat{y}^{(m)}_v=\arg\max_{c} P^{(m)}_{v,c}\] The agreement score of vertex \(v\) is defined as the fraction of predictions that vote for the majority class: \[\small \rho_v= \frac{1}{M} \max_{c} \sum_{m=1}^{M} \mathbf{1}\left[\hat{y}^{(m)}_v=c\right]\] A larger \(\rho_v\) indicates a more stable prediction, while a smaller \(\rho_v\) indicates stronger prediction variation.

We define the corresponding uncertainty as \(u_v = 1-\rho_v\) and convert the agreement score into a reliability weight \[\small w_v=\max(\rho_v,w_{\min})\] where \(w_{\min}\) prevents highly uncertain vertices from being completely ignored.

The reliability weight controls how strongly each vertex is constrained by its unary observation during inference optimization. Let \(X=\{x_v\}_{v=1}^{V}, \; x_v\in\Delta^{C-1}\) denote the optimized per-vertex class distributions, and let \(Y=\{y_v\}_{v=1}^{V}\) denote the unary observations constructed from the aggregated test-time predictions. We then define the reliability-weighted data term as \[\small E_{\mathrm{data}}(X)= \sum_{v=1}^{V} w_v \left\|x_v-y_v\right\|_2^2 \label{eq:realiability95weighted}\tag{1}\] A vertex with a high reliability weight \(w_v\) is anchored to its unary observation \(y_v\), whereas a vertex with a low \(w_v\) is less constrained by the unary term and can be further refined by the spatial and semantic consistency terms.

3.2 Label-Free Text Prototype Debiasing↩︎

The unary observations \(Y\) in Sec. 3.1 depend on the text-prototype classifier \(T\) used by the frozen open-vocabulary model. We therefore construct a label-free text classifier by replacing single-template class prototypes with prompt-ensemble prototypes.

For each test-time prediction in Sec. 3.1, the frozen 3D backbone produces a per-vertex feature \(h_v^{(m)}\in\mathbb{R}^{d}\). Open-vocabulary classification is performed by comparing this feature with a set of text prototypes. Let \(t_c\in\mathbb{R}^{d}\) denote the prototype of category \(c\). The class logit is computed by cosine similarity: \[\small z_{v,c}^{(m)} = \left\langle \bar{h}_v^{(m)}, \bar{t}_c \right\rangle\] where \(\bar{h}_v^{(m)}\) and \(\bar{t}_c\) are \(\ell_2\)-normalized features. The probability prediction \(P^{(m)}\) used in Sec. 3.1 is obtained by applying a softmax over the class logits.

A common strategy is to build each class prototype from a single template, such as “a {desk} in a scene”. However, a single prompt gives only one linguistic view of a category and can introduce wording-specific bias. Since the 3D model is frozen, such bias directly changes the open-vocabulary decision boundary and propagates to all test-time predictions. This issue is important in depth-only geometry setting, where the visual evidence is weaker than RGB input and the prediction relies more heavily on the alignment between 3D features and text prototypes.

To mitigate prompt-induced bias without labels, we construct each class prototype using a unified set of generic templates. Let \(\mathcal{Q}=\{q_k\}_{k=1}^{K}\) be the template set and \(n_c\) be the name of category \(c\). For each category, we generate \[\small s_{c,k}=q_k(n_c), \qquad k=1,\ldots,K\] and encode the prompts with the CLIP text encoder, whose language–vision alignment provides a natural classifier for open-vocabulary recognition. The class prototype is obtained by averaging normalized text embeddings followed by another normalization: \[\small t_c = \operatorname{norm} \left( \frac{1}{K} \sum_{k=1}^{K} \operatorname{norm} \left( g_{\phi}(s_{c,k}) \right) \right)\] where \(\operatorname{norm}(x)={x}/{\|x\|_2}\). The final text-prototype classifier is \(T=[t_1,\ldots,t_C]^\top\).

The same template set is applied to all categories, using only category names and generic prompts. No ground-truth labels, class-specific routing, or text-prototype training is used. The resulting classifier \(T\) reduces text-side bias in \(P^{(m)}\), thereby improving the aggregated observation \(Y\).

3.5pt

3.3 Feature-Space Semantic Consistency↩︎

The reliability-weighted data term in Eq. 1 determines how strongly each vertex should follow its unary observation. However, vertices are not independent: spatially neighboring points often share labels, and semantically similar points should remain consistent even when they are not best captured by local geometry alone. We therefore complete the inference objective with both coordinate-space and feature-space consistency regularization.

We first construct a geometric graph \(\mathcal{E}_g\) by applying \(k\)-nearest neighbors in the 3D coordinate space. This graph captures local spatial consistency and suppresses isolated noisy predictions. Its corresponding regularizer is \[\small E_{\mathrm{geo}}(X) = \sum_{(i,j)\in\mathcal{E}_g} S^{g}_{ij} \left\| x_i-x_j \right\|_2^2\] where \(S^{g}_{ij}\) denotes the geometric edge weight.

Coordinate-space proximity alone, however, does not always imply semantic consistency, especially in the depth-only privacy-preserving setting where RGB appearance cues are unavailable. To complement the geometric graph, we build a semantic graph in a learned feature space. To preserve the depth-only setting, we render depth-derived geometric views by colorizing depth values with a fixed colormap, rather than using real RGB appearance. These colorized depth renderings encode only geometric depth variation and do not contain scene texture, point-level color, or real image information. We extract features from these RGB-formatted depth renderings using a frozen DINO encoder and back-project them to 3D vertices, obtaining a feature \(\phi_v\) for each vertex. We then construct \(\mathcal{E}_s\) by \(k\)-nearest neighbors in the feature space of \(\phi_v\), with edge weights \(S^{s}_{ij}\) computed from feature similarity. The feature-space semantic consistency is defined as: \[\small E_{\mathrm{sem}}(X) = \sum_{(i,j)\in\mathcal{E}_s} S^{s}_{ij} \left\| x_i-x_j \right\|_2^2\]

Together with the reliability-weighted data term \(E_{\mathrm{data}}\), whose unary observations are computed using the debiased text-prototype classifier, the final inference objective is \[\small X^{\star}= \arg\min_X \left[ E_{\mathrm{data}}(X) + \lambda_g E_{\mathrm{geo}}(X) + \lambda_s E_{\mathrm{sem}}(X) \right]\] The final semantic label of vertex \(v\) is obtained by \(\hat{y}_v=\arg\max_c x^{\star}_{v,c}\). This objective anchors reliable predictions, refines uncertain vertices through local geometry, and enforces semantic consistency in feature space.

4 Experiments↩︎

Our experiments address three questions. First, we evaluate whether UTTO improves depth-only open-vocabulary 3D segmentation under the target privacy-preserving setting, where no real RGB input or information is used. Second, we ablate its key components to analyze the roles of uncertainty-guided optimization, text prototype debiasing, and feature-space consistency. Third, we compare this setting against non-privacy-preserving RGB-D open-vocabulary references and fully supervised geometry-only segmentation, highlighting the effect of removing RGB cues and requiring training-free open-vocabulary inference.

4.1 Setup↩︎

Dataset. We evaluate our method on ScanNet, a widely used indoor 3D scene understanding benchmark with RGB-D scans, reconstructed geometry, and dense semantic annotations. We report results on the validation sets of ScanNet20, ScanNet40, and ScanNet200, where ScanNet40 denotes the 40-class label setting on ScanNet following the NYUv2 label taxonomy. Unless otherwise specified, all evaluated open-vocabulary methods are used in an open-vocabulary manner: the target semantic categories are provided as text labels at inference time, without task-specific supervised training on the corresponding ScanNet label set.

Evaluation Metrics. We report mean Intersection over Union (mIoU) and mean Accuracy (mAcc), two standard metrics for 3D semantic segmentation.

Runtime. UTTO adds a modest inference-time overhead, taking 3.2 seconds per scene on average for test-time optimization.

Baselines. We compare methods under two sensing settings: non-private RGB-D perception and privacy-preserving depth-only perception. Here, a setting specifies what sensory information is available to the system, while an input specifies the actual evidence used by each segmentation pipeline. In the non-private setting, methods can access real RGB-D observations. In the privacy-preserving setting, methods only have access to depth observations and do not use real RGB input.

First, we include Mosaic3D [5] with its standard RGB-D input as a non-private open-vocabulary reference. This method uses real color observations and therefore does not satisfy our privacy-preserving requirement, but it provides a useful reference for understanding the performance gap associated with removing RGB appearance cues. We also apply UTTO to its RGB-D predictions, resulting in Mosaic3D+UTTO, as a diagnostic comparison under appearance-rich input.

Second, we evaluate privacy-preserving depth-only geometry inputs, which are the main focus of this work. For Mosaic3D, whose standard input includes color attributes, we remove real RGB information by replacing the RGB channels with constant zero values, ensuring that no real color or hallucinated appearance is used. We denote this variant as Mosaic3D-DepthOnly. We also evaluate OpenScene3D [8] as a geometry-only open-vocabulary 3D backbone under the same depth-only input constraint. We then apply UTTO to both backbones, resulting in Mosaic3D-DepthOnly+UTTO and OpenScene3D+UTTO. All depth-only methods are training-free at test time and do not use ground-truth labels during inference.

Third, we evaluate a pseudo-RGB privacy-preserving alternative, denoted as D2RGB+OpenSeg Fusion. This baseline follows the same privacy-preserving sensing constraint because it does not use real RGB images and starts only from depth observations. However, unlike our depth-only geometry input, it first generates pseudo-RGB views from depth-conditioned inputs using a generative model [24], applies the open-vocabulary 2D segmentation model OpenSeg [25] to the generated views, and then fuses the resulting 2D predictions into 3D using the corresponding depth maps. This baseline represents a complementary privacy-preserving strategy that attempts to recover missing appearance cues through generated visual observations. It allows us to assess whether hallucinated appearance can serve as a reliable substitute for real RGB cues under depth-only open-vocabulary 3D segmentation.

4.2 Privacy-Preserving Open-Vocabulary 3D Segmentation↩︎

Table 1 reports the main results on ScanNet20, ScanNet40, and ScanNet200. Since UTTO optimizes test-time predictions rather than training a specific 3D backbone, we evaluate it on two frozen open-vocabulary 3D backbones under depth-only input: Mosaic3D-DepthOnly, obtained by removing real RGB input from Mosaic3D, and OpenScene3D, a geometry-only open-vocabulary backbone. Under the privacy-preserving depth-only geometry setting, UTTO consistently improves both backbones across ScanNet20, ScanNet40, and the more fine-grained ScanNet200 setting. The gains are observed for both Mosaic3D-DepthOnly and OpenScene3D, indicating that UTTO is not tied to a particular model architecture or depth-only construction strategy.

The comparison also highlights the importance of real appearance cues. A controlled comparison between Mosaic3D with RGB-D input and Mosaic3D-DepthOnly shows a substantial drop when real RGB observations are unavailable, confirming that RGB appearance provides important semantic evidence for open-vocabulary 3D segmentation. Moreover, even the stronger geometry-only OpenScene3D backbone remains below the non-private Mosaic3D RGB-D reference on ScanNet20 and ScanNet40, further illustrating that depth-only open-vocabulary segmentation still faces a clear appearance gap under privacy-preserving sensing constraints. UTTO partially reduces this gap by improving depth-only predictions without reintroducing real RGB observations, additional training, or ground-truth labels.

The privacy-preserving D2RGB+OpenSeg Fusion baseline performs substantially worse than the optimized depth-only geometry methods. Although this baseline also follows the depth-only sensing constraint, it attempts to recover missing appearance cues through generated pseudo-RGB views. Its weaker performance suggests that hallucinated appearance is not a reliable substitute for real RGB cues in this setting. Instead, directly refining depth-only geometry-based predictions with uncertainty-guided test-time optimization provides stronger and more stable performance under privacy-preserving conditions. Fig. 3 provides a qualitative comparison under the privacy-preserving depth-only setting, showing that UTTO produces cleaner and more spatially consistent predictions than both pseudo-RGB fusion and the unoptimized geometry-only backbone.

On the non-private Mosaic3D RGB-D reference, UTTO brings only marginal changes. This suggests that the proposed optimization is most beneficial when RGB appearance cues are unavailable and predictions become more uncertain. Together, these results demonstrate that UTTO provides a broadly applicable test-time optimization framework for improving privacy-preserving open-vocabulary 3D perception under depth-only sensing constraints.

4.0pt

3.2pt

4.3 Ablation Study↩︎

Table 2 analyzes the contribution of different UTTO components on ScanNet20 under the privacy-preserving depth-only geometry setting. The w/o Both variant removes both text prototype debiasing and feature-space consistency, leaving only the uncertainty-guided optimization core. This variant already provides a strong refinement baseline, showing that uncertainty-guided optimization alone is effective under depth-only inference.

Adding text prototype debiasing improves both backbones when feature-space consistency is removed. This indicates that prompt-ensemble text prototypes help reduce text-side bias in depth-only open-vocabulary predictions and provide stronger unary observations.

Feature-space consistency shows a more backbone-dependent effect when used without text debiasing. It improves Mosaic3D-DepthOnly but slightly decreases performance for OpenScene3D, suggesting that feature-space priors can be helpful but may introduce noisy semantic coupling when the unary predictions are not sufficiently debiased. When text debiasing and feature-space consistency are combined, Full UTTO achieves the best performance for both backbones. These results suggest that foundation-model priors are most effective when jointly regularizing uncertainty-guided optimization, rather than being added independently.

4.4 Positioning the Depth-Only Open-Vocabulary Setting↩︎

Table 3 provides a contextual comparison across three perception regimes. This comparison is not intended as a standard equal-footing leaderboard, since the methods differ in both sensing constraints and task protocols. Instead, it is designed to contextualize the difficulty of privacy-preserving open-vocabulary 3D segmentation from geometry alone.

The non-private RGB-D reference reflects the benefit of real appearance cues for open-vocabulary segmentation, but it does not satisfy the privacy-preserving requirement because it uses real color observations. The supervised reference is Point Transformer V3 (PTv3) [26], a strong fully supervised 3D semantic segmentation model. We evaluate PTv3 with geometry-only input under a closed-set protocol. Although it follows the same geometry-only input constraint as our setting, it is trained with dense ScanNet annotations and therefore does not address open-vocabulary inference. In contrast, our UTTO variants operate under the most constrained protocol: they use depth-only geometry input, require no task-specific training, and support open-vocabulary categories at inference time. The comparison highlights that our setting combines two sources of difficulty: the absence of RGB appearance cues and the requirement of training-free open-vocabulary inference. While fully supervised closed-set geometry segmentation remains stronger, UTTO provides a privacy-preserving open-vocabulary alternative under substantially stricter supervision and input constraints.

4.5 Privacy-Preserving Semantic Goal Grounding Case Study↩︎

Motivated by the embodied application scenario illustrated in Fig. 1, we include a lightweight downstream case study to demonstrate the applicability of privacy-preserving open-vocabulary 3D perception to semantic goal grounding for downstream navigation. Given language queries over geometry-only 3D maps, UTTO identifies queried semantic regions that can serve as navigation goals without using real RGB observations. This case study is intended to demonstrate application feasibility rather than serve as a full navigation benchmark. Additional details and qualitative results are provided in the supplementary material.

5 Conclusion↩︎

We introduced UTTO, a model-agnostic uncertainty-guided test-time optimization framework for privacy-preserving depth-only open-vocabulary 3D semantic segmentation. Our experiments reveal several insights about this constrained setting. First, removing real RGB observations creates a clear appearance gap, confirming that depth-only open-vocabulary segmentation remains substantially more difficult than RGB-D perception. Second, hallucinated pseudo-RGB views do not reliably replace real appearance cues, suggesting that directly refining geometry-based predictions is a more robust strategy under privacy-preserving sensing constraints. Third, uncertainty is not merely a failure mode in RGB-free inference; it provides a useful reliability signal that allows stable regions to be preserved and ambiguous regions to be selectively refined. Finally, the contextual comparison with fully supervised geometry-only segmentation shows that a substantial gap remains when open-vocabulary inference must be performed without task-specific training. Together, these findings highlight both the difficulty and promise of privacy-preserving open-vocabulary 3D perception from depth-only geometry.

Future work may explore broader privacy-preserving sensing protocols beyond the strict RGB-free setting studied in this work. For example, extremely low-resolution or locally encoded appearance signals could provide weak semantic priors while reducing the risk of exposing privacy-sensitive content. Incorporating such privacy-filtered appearance cues into uncertainty-guided test-time optimization may further narrow the gap between depth-only and RGB-D perception.

Appendix↩︎

6 Real-Robot Semantic Goal Grounding Case Study↩︎

We provide additional details for the privacy-preserving semantic goal grounding case study introduced in the main paper. This case study corresponds to the embodied application scenario illustrated in Fig. 1 of the main paper, where an indoor robot constructs a geometry-only map, receives a class-level object-goal query, and uses UTTO to ground the queried semantic region as a navigation target. The goal of this study is to evaluate whether depth-only open-vocabulary 3D perception can produce a feasible semantic goal for downstream robot execution, rather than to benchmark a complete navigation system.

In our setup, the robot operates on a pre-reconstructed geometry-only 3D map and does not use real RGB images, point colors, textures, or image features from the test scene. Given a class-level query such as sofa, chair, desk, or door, UTTO predicts the corresponding semantic region on the 3D map. The predicted region is projected to a top-down navigation map, where connected components are converted into candidate goal poses. We then evaluate whether these generated goal candidates fall inside valid goal regions derived from coarse class-level annotations.

6.1 Geometry-Only Mapping Setup↩︎

Following the privacy-preserving setting in the main paper, the downstream perception pipeline operates only on a geometry-only 3D map. For the real-robot case study, we use a pre-reconstructed indoor mesh as the scene representation and remove all appearance information before running UTTO. The resulting map contains only geometric structure and does not include real RGB images, point colors, textures, or image features from the test scene.

This setup matches the intended deployment scenario illustrated in Fig. 1 of the main paper, where the mapping stage provides a depth-derived geometric representation and the semantic perception module is restricted to geometry-only input. In practice, such a map can be obtained from depth or LiDAR odometry/SLAM systems, where RGB sensing is treated as a prohibited modality and is disabled during mapping. Our evaluation focuses on the privacy-preserving perception stage: UTTO receives only the geometry-only map and the class-level query, regardless of the reconstruction backend used to build the mesh.

Fig. 4 shows the real-world indoor scene used in our case study. This colorized top-down visualization is provided only for human interpretation of the scene layout. The perception and goal-grounding pipeline does not use real RGB images, point colors, textures, or image features; UTTO operates on the geometry-only map.

6.2 Task Definition↩︎

We define a class-level semantic goal grounding task. We make three simplifying assumptions in this case study to isolate the semantic goal grounding problem. First, the input query is a class-level open-vocabulary target, such as sofa, chair, desk, or door, rather than a free-form language instruction or a specified object instance. This removes the need for language parsing or instance disambiguation and allows us to directly evaluate whether UTTO can ground a queried object category from geometry-only input. Second, we only consider navigation goals located inside the room and do not include hallway targets. This matches the intended object-goal setting, where the queried semantic targets correspond to indoor objects or structures within the mapped room. Third, the robot always starts from a fixed home pose in the hallway with a known pose in the pre-reconstructed map. Therefore, we do not evaluate online localization or simultaneous mapping in this case study. Together, these assumptions focus the evaluation on whether the predicted class-level semantic regions can produce feasible goal candidates from privacy-preserving geometry-only perception.

Given the pre-reconstructed geometry-only 3D map, UTTO segments the queried target class without using real RGB images, point colors, textures, or image features. The predicted class-level 3D mask is then projected onto the top-down navigation map. We extract connected components from the projected mask to obtain candidate semantic regions. To suppress small noisy fragments, we apply a component-size threshold shared by all methods and query classes. Specifically, connected components with fewer than 313 mesh vertices are removed.

For each remaining connected component, we generate one candidate navigation goal pose. Specifically, we compute the center of the predicted component and place the robot’s stopping pose 30 cm in front of this center. This produces one goal candidate for each retained predicted semantic region. The resulting goal candidates are then evaluated according to whether the robot stopping pose falls inside the valid goal region of the queried class.

6.3 Evaluation Protocol↩︎

We evaluate semantic goal grounding at the class level. For each query class, we manually annotate coarse object regions of that category in the 3D point cloud without distinguishing individual instances. These annotations are intentionally coarse and are used only to derive valid stopping regions on the top-down navigation map, rather than to serve as boundary-accurate dense 3D semantic ground truth. Therefore, we do not report 3D IoU or mIoU for the real-robot case study. Instead, we evaluate whether the predicted semantic regions can generate feasible stopping poses for the queried class.

Specifically, we project the class-level object annotations onto the top-down navigation map, dilate the projected regions by 30 cm, and intersect them with the traversable area to obtain the valid goal region for each query class. For each generated goal candidate, we check whether the robot stopping pose, placed 30 cm in front of the predicted component center, falls inside this valid goal region. A candidate is counted as a hit if its stopping pose lies inside the valid goal region of the queried class. Thus, a hit indicates that the predicted semantic component leads the robot to stop at a feasible location near a coarsely annotated object region of the requested category.

For each method and query class, we count how many generated goal candidates satisfy this criterion. This candidate goal-pose hit count evaluates whether the predicted class-level semantic regions can provide feasible navigation targets near the queried object category. It does not evaluate dense 3D segmentation quality, boundary accuracy, or instance-level grounding.

6.4 Results↩︎

Table 4 reports the real-robot semantic goal grounding results, and Fig. 5 provides qualitative visualizations for each query class. Overall, OpenScene3D+UTTO improves the candidate goal-pose hit rate from (5/11) to (6/8), corresponding to an increase from (45.4%) to (75.0%). This indicates that UTTO not only refines the predicted semantic regions, but also reduces the number of off-target goal candidates generated from noisy connected components.

For the chair query, OpenScene3D produces a single candidate and this candidate is valid. OpenScene3D+UTTO generates three candidates, two of which fall inside the valid goal region. The additional valid candidate suggests that UTTO recovers more chair-related regions from geometry-only input. The remaining non-hit candidate is likely caused by geometric ambiguity around nearby furniture structures, where depth-only observations make chair-like and table/desk-adjacent regions difficult to separate. This case shows that UTTO improves semantic coverage for the queried class, while occasional off-target candidates can still occur under depth-only perception.

For the sofa query, both methods produce one valid candidate. Although the hit count remains (1/1) for both methods, the qualitative visualization shows that OpenScene3D+UTTO yields a more complete and spatially coherent sofa prediction. This suggests that UTTO can improve the quality of the predicted semantic region even when the final candidate-level hit count is unchanged.

For the desk query, OpenScene3D generates five navigation candidates, among which only two fall within the valid goal region. Several non-hit candidates are produced from fragmented or off-target regions, and one candidate corresponds to a physically infeasible stopping pose that overlaps with the desk/furniture region. In contrast, OpenScene3D+UTTO generates three candidates, two of which fall inside the valid goal region. Although both methods recover two valid desk-related goals, UTTO yields a higher hit ratio. This suggests that UTTO helps suppress noisy fragments and produces desk predictions that are more reliable for downstream semantic goal generation.

For the door query, OpenScene3D produces a valid candidate but also generates several non-hit candidates from scattered door-like or wall-adjacent regions. OpenScene3D+UTTO produces a more compact prediction and yields a single valid candidate. This indicates that UTTO reduces spurious connected components while preserving the semantically relevant door region.

These observations are consistent with the quantitative results in Table 4. The improvement should not be interpreted as dense 3D segmentation accuracy or instance-level grounding performance. Rather, it shows that uncertainty-guided test-time optimization makes depth-only open-vocabulary predictions more useful for generating feasible semantic navigation goals.

References↩︎