1 Introduction↩︎

A 2D Scene Graph (SG) describes an image as a structured representation in which nodes correspond to detected objects and directed edges encode their pairwise relationships. A 3D scene graph lifts this abstraction into metric space. The task of online 2D-to-3D Scene Graph Generation (SGG) is to construct a global 3D SG incrementally from a stream of RGB-D frames. 3D SGs provide the high-level scene abstraction needed for downstream tasks such as embodied navigation [1], robotic manipulation [2], and spatial question answering [3]. Incremental construction supports real-time applications for which a complete scene reconstruction is unavailable or impractical [4], and lifting 2D predictions online into 3D leverages powerful 2D SGG models trained on richly annotated datasets [5] without the prohibitive cost of 3D SG annotation [6]–[8].

In the online setting, a global 3D SG is built by fusing a continuous stream of 2D observations, , a video sequence, into a persistent graph structure. With the camera’s field of view covering only a fraction of the scene at any instant, every such observation is partial and inherently uncertain. Accurately fusing these partial observations is non-trivial, as it requires reasoning about how much to trust each observation. However, existing online methods discard this information by committing to hard decisions at every stage. We identify three distinct sources of uncertainty, as illustrated in Fig. 1. (a) Observations are uncertain as objects are truncated. Relationships are particularly subject to such noise since they can only be predicted when both endpoint objects are simultaneously well observed. (b) 2D SGG models are uncertain. They produce soft class and relationship distributions that cannot be collapsed to hard predictions without loss. (c) Object representation in 3D is approximate. Back-projecting a 2D bounding box through a single noisy depth frame without committing to a full 3D reconstruction yields uncertain 3D position and spatial extent.

Offline 3D SGG methods rely on 3D point cloud input and global message-passing over the reconstructed graph, making them accurate but incompatible with real-time operation [10]–[13]. Online methods [9], [14]–[18] process frames incrementally and achieve low latency, but treat the 2D-to-3D merging stage as a fully deterministic pipeline. This rigidity overlooks the sources of uncertainty above and therefore introduces three systematic limitations. (a) A detection is either merged or rejected based on its argmax label, which prevents 2D model uncertainty from propagating into 3D. (b) Thresholding on raw 3D distance or spatial overlap treats back-projected 3D positions as exact, ignoring the inherent ambiguity in depth-based localization. (c) Hard label transfer during merging does not redistribute evidence across candidate nodes, nor complete relational evidence for sparsely co-observed edges.

We propose PUF: a plug-and-play, uncertainty-aware and training-free fusion framework that compensates for these failures simultaneously. Our framework wraps any 2D SGG model that outputs soft class and relation probability distributions, and propagates these distributions directly into our proposed merging process. 2D model uncertainty is hence preserved throughout 3D reconstruction. Scene graph node association is formulated as a probabilistic problem that jointly accounts for semantic uncertainty and 3D representation uncertainty. Each observation is matched against existing nodes through a likelihood, combining class-distribution similarity with a spatial overlap term, so that uncertain geometry attenuates rather than vetoes an association. Upon association, both node semantics and edge labels are updated through Dirichlet accumulation that redistributes evidence across all plausible candidates proportional to association likelihood. An optional relation prior compensates for poorly observed or structurally unobservable edges. PUF is representation-agnostic, and we instantiate it with both a 3D Gaussian and a 3D voxel backend to demonstrate its ability to generalize across different representations. We evaluate PUF on the 3DSSG [7], [10] and ReplicaSSG [9], [19] benchmarks, where our method significantly and consistently outperforms the strongest baseline. Overall, our main contributions are as follows:

• A plug-and-play, training-free fusion framework for online 3D scene graph generation that propagates label and representation uncertainty throughout the merging pipeline without additional training.

• A probabilistic node association that jointly models 2D scene graph uncertainty and 3D representation uncertainty, replacing hard association gates with a principled likelihood-based formulation.

• A Dirichlet evidence accumulation scheme for both node semantics and edge labels, augmented with a class-conditional prior that completes edges for sparsely and never co-observed object pairs.

• State-of-the-art results on the 3DSSG and ReplicaSSG benchmarks, improving relationship Recall@1 by 18.1 points over the strongest online baseline on 3DSSG, while maintaining real-time latency at 15 ms per frame.

2 Related Work↩︎

Offline 3D Scene Graph Generation. Early 3D scene graph methods treat the task as a global inference problem over a complete 3D point cloud. Wald  [7] introduced the 3DSSG dataset and a graph convolutional network [20] baseline that encodes PointNet [21] features extracted from fully reconstructed meshes. Subsequent works further refine node or edge representations, yet operate identically on a pre-built point cloud graph. Wang  [22] improve long-tail relation prediction via multi-modal knowledge distillation at training time, and Heo  [13] design a discriminative object feature encoder with contrastive pretraining. These methods produce dense and accurate segmentation at the point cloud level, but are incompatible with real-time operation.

Online 2D-to-3D Scene Understanding. A parallel line of work projects 2D predictions into 3D incrementally. General online lifting approaches demonstrate that rich 3D representations can be built without offline reconstruction. For example, ConceptGraphs [16] fuses 2D foundation-model detections into a 3D object graph via multi-view association. More recent open-vocabulary methods [23], [24] extend this to semantic segmentation masks, merging 2D outputs into persistent 3D structures using semantic similarity and spatial overlap.

In online 3D scene graph generation specifically, early methods rely on sparse point cloud reconstruction with SLAM technology [25], [26]. SGFN [10] and its predecessor MonoSSG [15] infer local frame graphs from incremental segmentation and merge them deterministically into a global structure. Feng  [17] introduce hyperrectangle embeddings to debias relation prediction. Kim  [14] build a sparse semantic graph from RGB-D streams without running reconstruction, but rely on hard association rules. More recently, FROSS [9] pairs a real-time 2D SGG model with Gaussian back-projection for 3D node representation. This methodology achieves faster-than-real-time operation by removing the dependency on SLAM reconstruction, but the 2D-to-3D merging is fully deterministic. SCRSSG [18] pursue a post-hoc confidence rescoring strategy that re-weights predictions using node-to-node and node-to-edge co-occurrence statistics from a global graph. This method focuses on alleviating prediction imbalance, and does not address the three sources of uncertainty identified in this work. All the above methods adopt deterministic online fusion or global message passing, which are not uncertainty-aware.

Uncertainty-Aware 3D Perception and Scene Graphs. Bayesian filtering has a long history in sensor fusion. The Joint Probabilistic Data Association Filter [27]–[29] computes per-observation marginal association probabilities across all candidate tracks simultaneously, replacing hard data association with a principled probabilistic assignment. More recent research extends this method for joint landmark and pose estimation in SLAM [30], [31], and track-let prediction in multi-target tracking [32], [33]. These techniques address spatial representation uncertainty in tracking but do not model semantic class distributions or graph-structured relationships.

In the 2D scene graph community, several works have recognized that relationship prediction is inherently uncertain. Yang  [34] propose a plug-and-play Probabilistic Uncertainty Modeling module that represents each subject-object union region as a Gaussian distribution rather than a deterministic feature vector, enabling diverse and semantically ambiguous predictions. Similarly, Li  [35] adopt Bayesian classifier reparameterization to handle annotation noise in relationship labels. These approaches improve 2D prediction quality by preserving distribution-level information during model training, yet they address uncertainty only within the 2D inference stage.

Our work bridges these two lines. Inspired by the literature on probabilistic data association, we replace hard association gates with a probabilistic node association formulation that jointly models semantic and spatial uncertainty. We then preserve the full softmax output of the 2D SGG model throughout the fusion process via Dirichlet evidence accumulation, propagating prediction uncertainty directly into the 3D scene graph rather than discarding it at the 2D-to-3D boundary. We adopt voxel- and Gaussian-based 3D representations from state-of-the-art online methods [9], [24] and demonstrate that our PUF is effective on both, and is thus generalizable across 3D representations.

3 Method↩︎

3.1 Problem Definition↩︎

We formulate online 3D scene graph generation as the incremental construction of a directed graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\), where each node \(v_i\in\mathcal{V}\) represents a 3D object with semantic label \(c_i\in\{1,\ldots,C\}\) and 3D position \(\boldsymbol{\mu}_i\), and each directed edge \((i,j)\in\mathcal{E}\) carries a relationship label \(r_{ij}\in\{1,\ldots,R\}\). Given a stream of RGB-D frames \(\{(\mathbf{I}_t,\mathbf{D}_t,\mathbf{T}_t)\}_{t=1}^{T}\), where \(\mathbf{T}_t\) is the known camera pose, the goal is to build \(\mathcal{G}\) incrementally without retaining the full video history.

3.2 Framework Overview↩︎

Figure 2 illustrates our uncertainty-aware fusion framework. At each frame, a 2D SGG model produces soft class and relation distributions, which are lifted to 3D observation nodes via either a Gaussian or voxel backend. We use the Dirichlet distribution to accumulate object and relationship evidence on the global 3D SG (Sec. 3.4). Probabilistic node association (Sec. 3.5) computes a joint association likelihood between a new observation and all existing global nodes, which replaces hard binary gates, and redistributes 2D evidence to 3D accumulators. An optional relationship prior (Sec. 3.6) completes sparsely observed edges.

Our framework is plug-and-play: it wraps any 2D SGG model that outputs soft class and relation distributions without modifying model architecture; and is training-free: it requires no additional neural-network training.

3.3 Preliminaries↩︎

2D scene graph model. We use RT-DETR detection backbone [36], [37] with EGTR [38] for 2D SGG, following FROSS [9]. RT-DETR produces object bounding boxes and feature representations at real-time latency. EGTR predicts pairwise relationship logits via an edge-conditioned transformer.

PUF departs from existing hard-fusion methods by consuming the full softmax outputs: the per-object class probability \(\hat{p}^c\in\Delta^{C-1}\) (the (\(C\)-1)-simplex) and the per-pair relation probability \(\hat{p}^r\in\Delta^{R-1}\), rather than only their argmax. In this way, the prediction uncertainty is propagated into the fusion layer.

3D representation — Gaussian. Each detected bounding box is back projected to a 3D Gaussian \(\Omega_i=(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i)\) using per-pixel depth and the camera intrinsics/extrinsics. The mean \(\boldsymbol{\mu}_i\) is the depth-weighted centroid of the back-projected pixel cloud inside the box; the covariance \(\boldsymbol{\Sigma}_i\) is the empirical covariance of the cloud. Implementation is identical to FROSS [9] for fair comparison; the details are provided in Supplementary Sec. B.

3D representation — Voxel. Each bounding box is converted to a discrete voxel set, inspired by OnlineAnySeg [24]. Depth-filtered pixels are first back-projected to the world coordinates and then quantized to a uniform grid of resolution \(\delta\). Depth filtering retains only pixels whose depth deviates by at most \(\varepsilon_d\) from the box-centre depth, isolating foreground from background clutter. The full lifting and merging procedure is detailed in Supplementary Sec. B.

3.4 Dirichlet Representation for Nodes and Edges↩︎

We use the Dirichlet distribution to represent the semantic labels of SG nodes and edges. It naturally replaces one-hot based label with a distribution over all categories, therefore carries 2D SGG model uncertainty. Its posterior update is straightforward and facilitates the evidence accumulation in the merging step.

Node class model. The class probability \(\hat{p}^c_k\) of global 3D node \(\mathcal{V}_k\) is modeled by a Dirichlet: \[\hat{p}^c_k\sim\operatorname{Dir}(\boldsymbol{\alpha}_{k}). \label{eq:node95model}\tag{1}\] where \(\boldsymbol{\alpha}_k \in \mathbb{R}^C\) is the evidence accumulator for \(\mathcal{V}_k\) over the semantic class label space, and interpreted as the concentration parameters of the Dirichlet distribution. Note that \(\boldsymbol{\alpha}_k\) is real-valued and not normalized to facilitate association update (Eq. 7 ). The normalized Dirichlet posterior mean \(\bar{\boldsymbol{\alpha}}_k = \boldsymbol{\alpha}_k / \|\boldsymbol{\alpha}_k\|_1\) gives a proper distribution over class labels, and the hard class prediction for the final output is \(c_k = \arg\max\,\bar{\boldsymbol{\alpha}}_k\).

Edge relation model. Similarly to nodes, for each directed edge \(\mathcal{E}_{ij}\), we model the relationship class probability \(\hat{p}^r_{ij}\) as: \[\hat{p}^r_{ij}\sim\operatorname{Dir}(\boldsymbol{\phi}_{ij}). \label{eq:rel95model}\tag{2}\] where \(\boldsymbol{\phi}_{ij}\in \mathbb{R}^R\) is the accumulator for edge \(\mathcal{E}_{ij}\). The final predicted relationship is the posterior mean argmax: \(r_{ij}=\arg\max\,\bar{\boldsymbol{\phi}}_{ij}\).

3.5 Probabilistic Node Association↩︎

Association likelihood. Between each pair of newly observed node \(obs\) and an existing node \(k\) in global 3D SG, we compute a factored likelihood: \[L[obs,\,k] = L_{\text{sp}}(obs,\,k)\;\cdot\;L_{\text{se}}(obs,\,k). \label{eq:likelihood}\tag{3}\] The spatial factor, unlike hard thresholds in existing methods, enters the likelihood as a continuous term that modulates association probabilities, and preserves as well as propagates 3D representation uncertainty into fusion. Its specific form depends on the representation: \[L_{\text{sp}}(obs,\,k) = \begin{cases} BC\!\left(\Omega_{obs},\Omega_k\right) & \text{(Gaussian)}, \\[4pt] |\Psi_{obs}\cap\Psi_k|\;/\;|\Psi_{obs}| & \text{(Voxel)}. \end{cases} \label{eq:spatial}\tag{4}\] For the Gaussian backend, BC denotes the Bhattacharyya coefficient and measures spatial overlap between two Gaussians, and \(BC = 1-H^2\) where \(H\) is the Hellinger distance [39]. For the voxel representation, we use the containment score as defined in Eq. 4 to measure spatial overlap between an observation with voxel set \(\Psi_{obs}\) and a global node with accumulated set \(\Psi_k\).

The semantic factor compares the Dirichlet posterior means of the observation and the global node via Jensen–Shannon divergence (JSD): \[L_{\text{se}}(obs,\,k) = \exp\!\left(-\,\frac{ JSD\!\left(\hat{p}_{obs}^c,\, \bar{\boldsymbol{\alpha}}_k\right)}{\sigma_{se}}\right). \label{eq:semantic}\tag{5}\] The bandwidth \(\sigma_{se}\) controls semantic selectivity. A larger value increases the semantic factor and encourages more divergent \(\left(\hat{p}_{obs}^c,\,\bar{\boldsymbol{\alpha}}_k\right)\) pairs to merge.

Probabilistic marginals of node association. Inspired by the Joint Probabilistic Data Association Filter [28], [29], we model the probability that the observation associates with global node \(k\) as \(\beta[obs,\,k]\). \(\beta[obs,\,k]\) is obtained by marginalizing the likelihood vector \(\{L[obs,\,k]\}_{k\in\mathcal{K}}\) together with a prior probability that \(obs\) is a new object: \[\beta[obs,\,k] = \frac{L[obs,\,k]}{Z}, \quad \beta_{birth} = \frac{\lambda_{birth}}{Z}, \quad Z = \lambda_{birth} + \textstyle\sum_{k\in\mathcal{K}} L[obs,\,k]. \label{eq:jpda}\tag{6}\] where \(\lambda_{birth}\) is a birth term encoding the prior probability of observing a new object. By construction \(\beta_{birth}+\sum_k\beta[obs,\,k]=1\). If \(\beta_{birth}>0.5\), the observation spawns a new global node; otherwise it associates with existing nodes. To ensure real-time computation we adopt per-observation independent normalization Eq. 6 rather than the expensive joint hypothesis enumeration as in the original Joint Probabilistic Data Association Filter [28]. This is also due to the fact that DETR’s non-maximum suppression mechanism [37] already prevents two detections of the same object within a single frame. We provide a detailed analysis in Supplementary Sec. A.

Association update. If the new observation merges to the global 3D SG (\(\beta_{birth}\leq0.5\)), we apply semantic and spatial updates as follows: \[\begin{align} {2} &\text{Semantic:}\quad &\boldsymbol{\alpha}_k &\;\leftarrow\; \boldsymbol{\alpha}_k + \beta[obs,\,k] \cdot \hat{p}_{obs}^c, \quad\forall\,k:\beta[obs,\,k]\geq\beta_{min}, \tag{7}\\[4pt] &\text{Spatial:}\quad &k^* &= \operatorname*{arg\,max}_k\;\beta[obs,\,k]. \tag{8} \end{align}\] where \(\beta_{min}\) is an evidence cut-off threshold. Class evidence across all candidates with non-negligible \(\beta[obs,\,k]\) is distributed to the global node \(k\), which lets node \(k\)’s Dirichlet accumulator absorb uncertainty across near-tie associations. 3D representation (Gaussian mean, covariance/voxel union) is updated only for \(k^*\) to avoid contaminating or drifting the positions of non-target nodes: a predicted label can be 0.6 chair - 0.4 table, but the object is not part chair, part table, so semantics are softly merged while spatial occupation is not.

A soft relationship transfer is applied to the SG edges. Let \(\hat{p}_{obs,j}^r\) denote the 2D relation prediction between the observation node and global node \(j\) in the current frame. The relationship evidence is redistributed to the same candidates and with the same weights as the semantic update: \[\boldsymbol{\phi}_{kj} \;\leftarrow\; \boldsymbol{\phi}_{kj} + \beta[obs,\,k]\cdot\hat{p}_{obs,j}^r, \quad \boldsymbol{\phi}_{ik} \;\leftarrow\; \boldsymbol{\phi}_{ik} + \beta[obs,\,k]\cdot\hat{p}_{i,obs}^r, \label{eq:rel95transfer}\tag{9}\] where the first term covers the observation acting as subject and the second as object. This update preserves consistency between the class and relation accumulators, giving evidence accumulation to poorly observed nodes and edges. After \(obs\) is merged to existing nodes, it is invalidated.

Otherwise, if \(\beta_{birth}>0.5\), then node \(obs\) is a newly observed object and initializes a new 3D node from the soft 2D outputs: \[\begin{gather} \boldsymbol{\alpha}_{obs} \leftarrow \hat{p}_{obs}^c . \\ \boldsymbol{\phi}_{i,obs} \leftarrow \hat{p}_{i,obs}^r\,, \quad \boldsymbol{\phi}_{obs,j} \leftarrow \hat{p}_{obs,j}^r\,. \label{eq:node95init} \end{gather}\tag{10}\] Then \(obs\) is appended to the list of global 3D nodes.

3.6 Relationship Prior for Edge Evidence Enhancement↩︎

An edge in the SG is only well-observed if both of its associated nodes are well-observed, making edges more susceptible to observation noise. To address this, we leverage Dirichlet conjugacy and propose an informative prior. For a pair with object classes \(c_i,c_j\) and 3D centroids \(\boldsymbol{\mu}_i,\boldsymbol{\mu}_j\), the prior factorizes as: \[\boldsymbol{\phi}^{prior}_{ij} = P_{cl}(r\mid c_i,c_j) \cdot P_{sp}(\boldsymbol{\mu}_i,\boldsymbol{\mu}_j) \cdot P_{ex}(c_i,c_j). \label{eq:prior}\tag{11}\] Class-conditional prior \(P_{cl}(r\mid c_i,c_j)\) is precomputed from training-set annotation counts with Laplace smoothing: \[P_{cl}(r\mid c_i,c_j) = \frac{n_{ij}^r+\varepsilon}{\sum_{r'}n_{ij}^{r'}+R\varepsilon}, \label{eq:pclass}\tag{12}\] where \(n_{ij}^r\) counts relation \(r\) between object-class pair \((c_i,c_j)\) in training data, and Laplace smoothing \(\varepsilon=0.1\). This requires a single offline pass over annotations without gradient-based training.

Spatial prior \(P_{sp}(\boldsymbol{\mu}_i,\boldsymbol{\mu}_j)\) is computed online from 3D positions, decaying the prior as spatial distance increases: \(P_{sp}(\boldsymbol{\mu}_i,\boldsymbol{\mu}_j) = e^{-\frac{\mid\boldsymbol{\mu}_i -\boldsymbol{\mu}_j\mid}{2}}\). The combination of the class and spatial priors is normalized before multiplying with \(P_{ex}(c_i,c_j)\).

Existence gate \(P_{ex}(c_i,c_j)\) is the fraction of training-set pairs of class \((c_i,c_j)\) that carry any annotated relation. It scales the prior down for class pairs that rarely interact, suppressing false-positive completions for rarely co-occurring class pairs.

For observed edges, the final distribution is the Bayesian posterior: \[\hat{p}^r_{ij}\sim\operatorname{Dir}(\boldsymbol{\phi}_{ij}^{post.}), \qquad \boldsymbol{\phi}_{ij}^{post.} = \boldsymbol{\phi}_{ij}^{prior} + \boldsymbol{\phi}_{ij}, \label{eq:posterior}\tag{13}\] where \(\boldsymbol{\phi}_{ij}\) is the accumulated evidence by Eq. 9 . For unobserved edges, the prior alone is used if \(\max(\boldsymbol{\phi}_{ij}^{prior})>0.5\). In the case no prior is used, the relationship label is predicted from Eq.@eq:eq:rel95model .

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.1 Datasets.↩︎

The 3DSSG dataset [7] extends 3RScan [6], comprising 1,482 RGB-D indoor scans with dense instance segmentation and directed inter-object relationship annotations. We follow the standard protocol of [7], [9], [10], [15], mapping 160 object categories to the 20 NYUv2 classes [40] and retaining the 7 most frequent predicate types. After mapping, the dataset contains 21,974 annotated object instances and 16,324 inter-object relationships. We use the official training, validation, and test splits throughout.

ReplicaSSG [9] extends the Replica dataset [19] with inter-object relationship annotations across 18 photorealistic indoor scenes. Its label space is adopted from Visual Genome [5] and contains 33 object and 9 relationship categories, facilitating zero-shot transfer from pretrained 2D SG models. Of the 18 scenes, 7 serve as a validation set and 11 as the test set; there is no training split.

4.1.2 Evaluation Metrics.↩︎

We adopt the recall-based protocol of [9], [15]. Object recall measures the fraction of ground-truth instances matched to a correctly classified node. Predicate recall measures correct relationship classification among pairs whose endpoints are both detected. Relationship recall requires both endpoints to be correctly detected and the predicate correctly classified. We additionally report mean recall (mRecall), which averages per-class recall to mitigate the severe predicate class imbalance present in both datasets [9], [10], [15].

4.1.3 Implementation Details.↩︎

We use EGTR [38] with RT-DETRv2-M [37] as the 2D scene graph model, following FROSS [9] for fair comparison. For 3DSSG experiments, the model is trained on 2D scene graphs extracted from the 3DSSG training split. For ReplicaSSG experiments, we use a model trained on Visual Genome [5] with no additional fine-tuning on Replica data. At inference, object detections below a confidence threshold of 0.7 are discarded and the top-10 pairwise relation scores per frame are retained, matching the FROSS configuration. All models are evaluated on an Intel XEON Gold 6534 processor with a single CPU core, and an NVIDIA L40S GPU.

3D representations. For the Gaussian backend, the back-projection and covariance estimation follow FROSS [9] without modification. For the voxel backend, we quantize depth-filtered back-projected points to a grid of \(\delta=2\) cm and retain only pixels whose depth deviates by at most \(\varepsilon_d=0.3\) m from the bounding box center depth for foreground isolation.

Hyperparameters. For Probabilistic Node Association and on both 3DSSG and ReplicaSSG, the semantic bandwidth is set to \(\sigma_{se}=0.3\); the birth density is \(\lambda_{birth}=0.4\); the minimum association weight for soft updates is \(\beta_{min}=0.05\). For the Relationship Prior, the class-conditional prior \(P_{cl}\) is precomputed offline from the 3DSSG training split, and no prior is used for ReplicaSSG since it does not have a training set. All hyperparameters are selected by grid search on the respective validation splits with relationship recall as the objective. Detailed analysis is provided in Section 4.3 for the 3D Gaussian representation and in Supplementary Sec. D.2 for the voxel.

4.2 Quantitative Results and Comparison↩︎

Tables ¿tbl:table:main953dssg? and 1 present performance and latency comparisons on the 3DSSG and ReplicaSSG benchmarks, respectively. On 3DSSG, our PUF-Gaussian variant achieves the highest relationship recall of 46.0%, improving over FROSS by 18.1 points while operating at 15 ms per frame. Compared to the SGG recall gain, the additional fusion latency is negligible, and our method remains faster than all RGB-D+SLAM baselines by an order of magnitude. This further substantiates the advantages of lifting scene graphs from 2D images over direct point cloud reasoning. On ReplicaSSG, where no relationship prior is used due to the absence of a training split, PUF-Gaussian variant consistently outperforms FROSS across all metrics. This demonstrates that our proposed fusion framework itself provides reliable gains, independently of the prior. This is further supported by our ablation studies Tab. 2 and  3 in Sec. 4.3. Our Voxel variant further demonstrates that the improvements are representation-agnostic: consistent gains over FROSS are observed with both backends across both benchmarks, confirming that the framework generalizes beyond any single 3D representation choice.

4.3 Ablation Study↩︎

4.3.1 Module Effectiveness.↩︎

Tables 2 and 3 report ablation results on 3DSSG and ReplicaSSG, respectively. We ablate three components: Dirichlet node representation, Dirichlet edge representation, and the relationship prior, across both the Gaussian and voxel backends. Probabilistic association is only enabled if the Dirichlet node representation is used.

On 3DSSG, adding the Dirichlet node representation alone improves relationship recall by 3.4 points over FROSS under the Gaussian backend, confirming that propagating 2D class uncertainty into the fusion layer benefits node association and semantic accumulation. Incorporating Dirichlet edge representation and enabling probabilistic association further improves relationship recall to 33.9%, as soft relational evidence is redistributed across plausible candidates rather than collapsed to hard votes. On top of the probabilistic fusion, the relationship prior additionally contributes a large gain: enabling it alongside the edge Dirichlet representation lifts relationship recall to 41.4%, as it completes evidence for sparsely and never co-observed pairs. The full model combining all three components achieves 46.0% relationship recall. Consistent trends are observed with the voxel backend, confirming that each component’s contribution is representation-agnostic.

On ReplicaSSG, where no relationship prior is used, the node and edge Dirichlet components each independently improve over FROSS, and combining them with the probabilistic association yields the best performance of 25.3% relationship recall under the Gaussian backend. This consistent improvement in the absence of any prior corroborates that the core uncertainty-aware fusion framework is effective on its own, and that the gains observed on 3DSSG are not solely attributable to the prior.

4.3.2 Hyperparameter choices.↩︎

Table 4 reports the sensitivity of the two main hyperparameters on the 3DSSG (w/o prior) and ReplicaSSG validation sets under the Gaussian backend. The semantic bandwidth \(\sigma_{\text{se}}\) controls the selectivity of the semantic likelihood factor (Eq. 5 ): values below 0.3 over-penalize distributional differences and fragment semantically similar nodes, while values above 0.3 over-merge distinct objects, degrading both object and relationship recall. The birth density \(\lambda_{\text{birth}}\) governs the prior probability of spawning a new global node (Eq. 6 ). A large \(\lambda_{\text{birth}}\) under-merges the graph, leaving object recall roughly intact, but depressing predicate and relationship recall as relational evidence is scattered across duplicate nodes. A small \(\lambda_{\text{birth}}\) over-merges distinct objects, harming all metrics. Notably, the same setting \((\sigma_{\text{se}}{=}0.3,\;\lambda_{\text{birth}}{=}0.4)\) is optimal on both datasets, indicating that the framework is not sensitive to precise tuning. Voxel-backend hyperparameter analysis is provided in Supplementary Sec. D.2 and exhibits consistent trends.

4.3.3 Relationship Prior on 3DSSG.↩︎

The 3DSSG dataset presents a notable co-visibility challenge: despite an average of \({\sim}245\) frames per scan, 25.3% of ground-truth object pairs accumulate fewer than 10 joint frame observations, and 2.7% never co-appear at all. Figure 3 stratifies relationship recall by the number of 2D co-observations per ground-truth triplet. Two findings stand out. First, even without the relationship prior, our uncertainty-aware fusion framework (“w/o Prior”) consistently outperforms FROSS, confirming that probabilistic node association and soft evidence redistribution alone improve recall for both well- and poorly-observed edges. This is further confirmed by a quantitative breakdown in Supplementary Sec. C. Second, the relationship prior yields its largest gains in the sparsest bins, and its marginal contribution diminishes monotonically as co-observation count increases. This suggests that the prior behaves correctly: the prior fills in evidence that the observation stream cannot provide, while deferring to accumulated observations when they are plentiful.

4.4 Qualitative Results↩︎

Figure 4 compares a representative scene from 3DSSG under FROSS and our method using the Gaussian backend. FROSS misclassifies several rare nodes in the other class (marked in red) and misses the rare predicate hanging on between curtain and wall, defaulting instead to the dominant standing on. Our method correctly recovers the object and the rare relationship, illustrating the benefit of soft evidence accumulation over hard label transfer. Additional qualitative outputs on ReplicaSSG and with the voxel backend are provided in Supplementary Sec. E, which show consistent results.

Because the Dirichlet representation maintains a full distribution over labels, we can additionally visualize per-node and per-edge predictive uncertainty via the normalized entropy \(H/H_{\max}\), where \(H_{max}\) is the entropy over an even Dirichlet distribution. As shown in the right-hand graph of Fig. 4, correctly classified nodes and edges exhibit low normalized entropy, while the few remaining errors, such as the standing on edge and the misclassified counter, carry notably higher uncertainty. This calibration property is a direct consequence of propagating the three sources of uncertainty identified in Sec. 1 through the fusion pipeline, and could be leveraged by downstream consumers to filter or re-query unreliable predictions.

5 Conclusion↩︎

We presented a plug-and-play, training-free fusion framework, PUF, for online 3D scene graph generation. Our framework replaces the deterministic merging pipelines of prior works with uncertainty-aware probabilistic association and Dirichlet evidence accumulation. By preserving 2D prediction distributions, modeling 3D representation uncertainty in node association, and optionally completing sparsely observed edges with a class-conditional prior, our framework yields substantial gains in relationship recall on both the 3DSSG and ReplicaSSG benchmarks while adding negligible latency. Its representation-agnostic design is validated with both Gaussian and voxel backends, and makes it readily applicable as a drop-in wrapper around any 2D scene graph model that outputs soft class and relation distributions.

Acknowledgements↩︎

This work has been supported by the Centre for Spatial Intelligence (RCSI) at University of Bath, the European Union under grant agreement no. 101136006-XTREME, the European Innovation Council under grant agreement no. 101257536-CEREBRIS, the MWK of Lower Saxony within Hybrint (VWZN4219) and LCIS (VWZN4704), the DFG under Germany’s Excellence Strategy within the Cluster of Excellence PhoenixD (EXC2122) and Quantum Frontiers (EXC2123).

Supplementary Materials↩︎

Overview↩︎

This supplementary document provides additional theoretical analysis, quantitative results, and qualitative visualizations that complement the main paper. The contents are organized as follows:

• Section 6 provides a formal justification for the per-observation independent normalization used in our probabilistic node association. We show that under the \(\epsilon\)-separation condition enforced by the detector’s non-maximum suppression, the factorized approximation we use recovers the exact Joint Probabilistic Data Association marginals with a bounded error of \(O(NM\epsilon)\), which is negligible for typical frame sizes.

• Section 7 details the 2D-to-3D lifting and geometric merging of the 3D backends: the Gaussian representation adapted from FROSS and the voxel representation adapted from OnlineAnySeg, making the preliminaries of the main paper self-contained.

• Section 8 introduces a frequentist completion baseline that isolates the contribution of dataset statistics from PUF’s probabilistic evidence accumulation, demonstrating that the Bayesian integration of prior evidence with accumulated observations is more effective than naive frequency-based edge completion.

• Section 9.1 reports per-class object and predicate recall on both the 3DSSG and ReplicaSSG test sets, offering a fine-grained view of results.

• Section 9.2 presents hyperparameter sensitivity analysis for the voxel backend on both datasets, complementing the Gaussian-backend analysis.

• Section 9.3 shows that PUF is robust to temperature-scaled 2D model uncertainty, and remains plug-and-play to post-hoc calibration methods.

• Section 10 provides additional qualitative comparisons between the Gaussian and voxel backends on 3DSSG and ReplicaSSG, respectively, illustrating that PUF produces consistent graph topology and interpretable uncertainty estimates across both 3D representations and datasets.

6 Per-observation Normalization for Association↩︎

We provide a detailed analysis of how and why the Joint Probabilistic Data Association Filter (JPDAF) can be factorized into per-observation normalization under non-maximum suppression (NMS).

6.1 Setup↩︎

At frame \(t\), the 2D detector produces \(N\) bounding-box detections. Each detection is converted into a 3D Gaussian / voxel set and projected into the 3D space, forming the observation set: \[\mathbf{Z} = \{z_1, \ldots, z_N\}\] Let \(\mathcal{T} = \{1, \ldots, M\}\) denote the indices of the \(M\) currently active global nodes in the 3D scene graph (, tracks in Joint Probability Data Association). A joint association hypothesis is a function: \[\theta : [N] \to \mathcal{T} \cup \{0\},\] that assigns each observation \(i\) to either a new birth \(\theta(i) = 0\) or to an existing global node \(\theta(i) = k > 0\).

6.2 Exact Joint Posterior↩︎

6.2.0.1 Feasibility.

A hypothesis \(\theta\) is feasible if each track emits at most one observation per frame: \[\Theta_{F} \;=\; \bigl\{\,\theta : [N] \to \mathcal{T} \cup \{0\} \;\big|\; \theta(i) = \theta(j) \neq 0 \;\Rightarrow\; i = j \,\bigr\}. \label{eq:feasible95set}\tag{14}\] \(|\Theta_F|\) grows super-exponentially with \(N\) and \(M\): the number of feasible hypotheses equals \(\sum_{k=0}^{\min(N,M)} \binom{N}{k}\binom{M}{k} k!\, 2^{N-k}\). For typical frame sizes (\(N \approx 20\), \(M \approx 30\)), this space of hypotheses is far too large to enumerate exhaustively in real time.

6.2.0.2 Likelihood model.

Let \(L [i, k] = L(z_i \mid k) \in (0, 1]\) denote the normalized likelihood that observation \(i\) originates from global node \(k\), and let \(\lambda_b > 0\) denote the prior birth rate. The per-factor potential is defined as \[f_i(k) \;=\; \begin{cases} L [i, k] & k \in \mathcal{T}, \\[2pt] \lambda_b & k = 0. \end{cases} \label{eq:potential}\tag{15}\] Under a Poisson birth model and conditionally independent observations given the scene state, the posterior over feasible joint hypotheses is \[P(\theta \mid \mathbf{Z}) \;=\; \frac{1}{Z_F} \prod_{i=1}^{N} f_i\!\bigl(\theta(i)\bigr), \qquad Z_F \;=\; \sum_{\theta' \in \Theta_F} \prod_{i=1}^{N} f_i\!\bigl(\theta'(i)\bigr). \label{eq:joint95posterior}\tag{16}\]

6.2.0.3 Exact JPDA marginals.

The quantity of interest is the marginal association probability of observation \(i\) with track \(k\): \[\beta [i, k] \;=\; P\!\bigl(\theta(i) = k \mid \mathbf{Z}\bigr) \;=\; \sum_{\substack{\theta \in \Theta_F \\ \theta(i) = k}} P(\theta \mid \mathbf{Z}) \;=\; \frac{L [i, k]}{Z_F} \underbrace{ \sum_{\substack{\theta \in \Theta_F \\ \theta(i) = k}} \prod_{j \neq i} f_j\!\bigl(\theta(j)\bigr) }_{=:\; C_i(k)}. \label{eq:exact95marginal}\tag{17}\] The inner sum \(C_i(k)\) requires marginalizing over all feasible completions of the remaining \(N-1\) observations subject to the constraint that no other observation is assigned to \(k\). Computing \(C_i(k)\) exactly is #P-hard in general [43], as it is equivalent to computing the permanent of a \(0/1\)-weighted matrix.

6.3 Independent Per-Observation Approximation↩︎

6.3.0.1 Factorized approximation.

Rather than summing over \(\Theta_F\), our implementation drops the feasibility constraint and factorizes the joint distribution across observations: \[\hat{P}(\theta) \;=\; \prod_{i=1}^{N} \hat{P}_i\!\bigl(\theta(i)\bigr), \label{eq:factorised95joint}\tag{18}\] where each marginal is independently normalized: \[\hat{\beta}[i, k] \;=\; \hat{P}_i\!\bigl(\theta(i) = k\bigr) \;=\; \frac{L [i, k]}{\lambda_b + \displaystyle\sum_{k' \in \mathcal{T}} L[i, k']}, \quad \hat{\beta}_i^{\,\mathrm{birth}} \;=\; \frac{\lambda_b}{\lambda_b + \displaystyle\sum_{k' \in \mathcal{T}} L[i, k']}. \label{eq:approx95marginal}\tag{19}\] The unconstrained partition function factorizes as \(Z = \prod_{i=1}^N \!\bigl(\lambda_b + \sum_{k} L [i, k]\bigr)\), replacing the intractable sum over \(\Theta_F\) with \(N\) independent scalar normalizations.

6.4 Exactness Under Non-Maximum Suppression↩︎

We now formalize the condition under which the feasibility constraint in Eq. 16 contributes negligible probability mass, so that the approximation Eq. 19 recovers the exact marginals.

6.5 Application to EGTR with NMS↩︎

6.5.0.1 NMS enforces \(\epsilon\)-separation in 2-D.

EGTR [38] applies class-conditioned non-maximum suppression before producing detections: any box \(b_j\) with \(\mathrm{IoU}(b_i, b_j) > \tau_{\mathrm{NMS}}\) and the same predicted class as a higher-scoring box \(b_i\) is suppressed. Each global node \(k\) corresponds to a single physical object. If \(k\) is visible in frame \(t\), it projects to a compact image region whose extent is determined by the object’s 3-D bounding volume and the camera intrinsics. Two distinct post-NMS detections of the same class cannot both overlap this region with \(\mathrm{IoU} > \tau_{\mathrm{NMS}}\). Since our likelihood \(L[i, k] = L_{\mathrm{sp}}(i,k)\, L_{\mathrm{se}}(i,k)\) combines a spatial Bhattacharyya coefficient / containment score (which is small for non-overlapping Gaussians / voxels) and a class-conditional semantic term (which is near zero for different predicted classes), \(\epsilon\)-separation holds with \[\epsilon \;\lesssim\; \max_{i \neq j}\, \min\bigl(L[i, k],\, L[j, k]\bigr) \;\ll\; 1. \label{eq:epsilon95bound}\tag{27}\] Proposition [prop:factorisation] then guarantees that the independent per-observation normalization Eq. 19 incurs a marginal error of at most \(O(NM\epsilon)\), which for typical frame sizes (\(N \approx 20\), \(M \approx 30\)) is negligible.

6.5.0.2 Relation to standard JPDAF.

Classical JPDAF [28] is typically applied to radar or sonar tracking where clutter density is high and the number of targets is moderate. In those settings, multiple detections of the same target are common and the feasibility constraint is necessary. Our setting is essentially different. The detector’s NMS post-processing reduces the per-frame miss-detection of the same object to negligible probability, making the feasibility constraint vacuous and the independent factorization exact in the \(\epsilon \to 0\) limit established by Proposition [prop:factorisation].

7 Details of 2D-to-3D Lifting and 3D Merging↩︎

The main paper instantiates PUF with two 3D backends: a 3D Gaussian representation adapted from FROSS [9] and a discrete voxel representation adapted from OnlineAnySeg [24]. For completeness, this section provides the implementation details of (i) the 2D-to-3D lifting that converts each 2D detection into a 3D primitive, and (ii) the geometric merging that the two backends use to fuse a local observation into the global representation.

7.1 Gaussian Backend (FROSS)↩︎

7.1.0.1 2D Gaussian representation of bounding boxes.

Each detected object \(v_i\) is first described by a 2D Gaussian \((\boldsymbol{\mu}_i^{\mathrm{2D}}, \boldsymbol{\Sigma}_i^{\mathrm{2D}})\) in the image plane. Following FROSS, the bounding box is treated as a 2D uniform distribution: the mean \(\boldsymbol{\mu}_i^{\mathrm{2D}}=(p_x,p_y)^\top\) is the box center, and the covariance matches the second moment of a uniform box of width \(W_i\) and height \(H_i\), \[\boldsymbol{\Sigma}_i^{\mathrm{2D}} = \frac{1}{12} \begin{bmatrix} W_i^2 & 0 \\ 0 & H_i^2 \end{bmatrix}. \label{eq:fross952dcov}\tag{28}\]

7.1.0.2 Back-projection to 3D.

Given the depth \(z\) at the box center and the camera intrinsics \(K\in\mathbb{R}^{3\times3}\), rotation \(R\in\mathbb{R}^{3\times3}\), and translation \(\mathbf{t}\in\mathbb{R}^3\), the 3D mean is obtained by back-projecting the 2D center and transforming it to world coordinates, \[\boldsymbol{\mu}_i^{\mathrm{3D}} = R\,K^{-1}\,(p_x,p_y,1)^\top \cdot z + \mathbf{t}. \label{eq:fross95mean}\tag{29}\] The 2D covariance is lifted with the local affine (Jacobian) approximation of the perspective projection, \[J = \begin{bmatrix} \dfrac{f_x}{z} & 0 & -\dfrac{f_x p_x}{z^2} \\[8pt] 0 & \dfrac{f_y}{z} & -\dfrac{f_y p_y}{z^2} \end{bmatrix}, \label{eq:fross95jacobian}\tag{30}\] where \(f_x,f_y\) are the focal lengths. Because \(J\) maps 3D to 2D, the covariance is back-projected through its pseudo-inverse \(J^+\). A naive inversion leaves the depth axis without variance, since the 2D box carries no depth-extent information; FROSS therefore injects a depth-axis variance equal to the average of the two recovered lateral variances, applied in the camera frame before the world rotation, \[\begin{align} \boldsymbol{\Sigma}_i^{\mathrm{3D}\prime} &= J^+\,\boldsymbol{\Sigma}_i^{\mathrm{2D}}\,J^{+\top}, \tag{31}\\[4pt] \boldsymbol{\Sigma}_i^{\mathrm{3D}\prime\prime} &= \boldsymbol{\Sigma}_i^{\mathrm{3D}\prime} + \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & \dfrac{(\boldsymbol{\Sigma}_i^{\mathrm{3D}\prime})_{1,1}+(\boldsymbol{\Sigma}_i^{\mathrm{3D}\prime})_{2,2}}{2} \end{bmatrix}, \tag{32}\\[4pt] \boldsymbol{\Sigma}_i^{\mathrm{3D}} &= R\,\boldsymbol{\Sigma}_i^{\mathrm{3D}\prime\prime}\,R^\top. \tag{33} \end{align}\] The resulting 3D Gaussian \(\Omega_i=(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i)\) with \(\boldsymbol{\mu}_i=\boldsymbol{\mu}_i^{\mathrm{3D}}\) and \(\boldsymbol{\Sigma}_i=\boldsymbol{\Sigma}_i^{\mathrm{3D}}\) is the observation node used throughout the fusion.

7.1.0.3 Gaussian merging.

In FROSS, two nodes are fused when their Hellinger distance falls below a threshold \(\delta_d\). For Gaussians \(\mathcal{N}(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i)\) and \(\mathcal{N}(\boldsymbol{\mu}_j,\boldsymbol{\Sigma}_j)\), the Hellinger distance derives from the Bhattacharyya distance \(B_D\), \[\begin{align} H_D(i,j) &= \sqrt{1-\exp\!\bigl(-B_D(i,j)\bigr)}, \tag{34}\\[4pt] B_D(i,j) &= \frac{1}{8}\,\Delta\boldsymbol{\mu}_{ij}^\top\,\boldsymbol{\Sigma}^{-1}\,\Delta\boldsymbol{\mu}_{ij} + \frac{1}{2}\ln\!\frac{\det\boldsymbol{\Sigma}}{\sqrt{\det\boldsymbol{\Sigma}_i\,\det\boldsymbol{\Sigma}_j}}, \tag{35} \end{align}\] with \(\Delta\boldsymbol{\mu}_{ij}=\boldsymbol{\mu}_i-\boldsymbol{\mu}_j\) and \(\boldsymbol{\Sigma}=\tfrac{1}{2}(\boldsymbol{\Sigma}_i+\boldsymbol{\Sigma}_j)\). The matched Gaussians are fused by a confidence-weighted integration, where the weight \(w_i\) of each node records its accumulated detection frequency across views, so that objects observed from more viewpoints contribute proportionally more, \[\begin{align} \boldsymbol{\mu}_k &= \frac{w_i\boldsymbol{\mu}_i + w_j\boldsymbol{\mu}_j}{w_i+w_j}, \tag{36}\\[4pt] \boldsymbol{\Sigma}_k &= \frac{w_i\boldsymbol{\Sigma}_i + w_j\boldsymbol{\Sigma}_j}{w_i+w_j} + \frac{w_i w_j\,(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j)(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j)^\top}{(w_i+w_j)^2}. \tag{37} \end{align}\] The spatial likelihood \(L_{\mathrm{sp}}\) of the main paper reuses the Bhattacharyya coefficient \(BC=1-H_D^2=\exp(-B_D)\) from Eq. 35 as a continuous overlap score, and the spatial update of node \(k^*\) follows Eqs. 36 –37 .

7.2 Voxel Backend (OnlineAnySeg)↩︎

7.2.0.1 Voxel lifting.

For the voxel backend, each detected mask is lifted into 3D by back-projecting its depth-filtered pixels into world coordinates and quantizing them onto a uniform grid of resolution \(\delta\), producing a voxel set \(\Psi_{obs}\). Depth filtering keeps only pixels whose depth deviates from the box-center depth by at most \(\varepsilon_d\), which isolates the foreground object from background clutter falling inside the 2D box.

7.2.0.2 Voxel merging.

OnlineAnySeg quantifies the spatial association between two voxel sets \(V_a,V_b\) through an asymmetric overlap ratio. Restricting to the voxels of \(V_b\) that are visible from the frames \(X(m_a)\) in which \(m_a\) is observed, \(\mathrm{Vis}(V_b,X(m_a))=\{v\in V_b\mid v\to X(m_a)\}\), the ratio is \[or_{(a,b)} = \frac{|V_a\cap V_b|}{|\mathrm{Vis}(V_b,X(m_a))|}, \label{eq:anyseg95overlap}\tag{38}\] which measures how much of \(m_b\) falls inside \(m_a\) within their co-visible region. Two masks are merged when their overall similarity (combining overlap ratio with semantic and geometric feature similarity) exceeds a threshold, or when enough third-view masks support the association. When merged, the new voxel set is the union of the constituent sets and the new weight is the sum of their individual weights, \[\Psi_k = \Psi_i \cup \Psi_j, \qquad w_k = w_i + w_j, \label{eq:anyseg95merge}\tag{39}\] and the global identity of every affected voxel is redirected to the merged node through an append-only mapping table. PUF adopts the voxel-set union of Eq. 39 for the spatial update of \(k^*\), and uses the simplified, single-frame containment score \(|\Psi_{obs}\cap\Psi_k|/|\Psi_{obs}|\) defined in the main paper as the spatial likelihood \(L_{\mathrm{sp}}\) in place of the visibility-restricted overlap ratio of Eq. 38 , since the per-observation likelihood already operates on the current frame.

8 Frequentist vs. Probabilistic Prior on 3DSSG↩︎

The frequentist completion baseline is designed to isolate the contribution of the dataset statistics in PUF’s relationship prior by testing how well a simple co-occurrence prior alone can recover unobserved scene graph edges. The frequentist baseline applies the same training-set co-occurrence statistics as PUF but assigns them as hard completions to unobserved edges, without Dirichlet evidence accumulation. Concretely, for each unobserved pair \((i, j)\), it looks up the empirical co-occurrence probability \(P_{ex}(r\mid c_i,c_j)\). If this exceeds the same completion threshold of 0.5 as PUF, the relation distribution \(P_{cl}(r\mid c_i,c_j)\cdot P_{sp}(\boldsymbol{\mu}_i,\boldsymbol{\mu}_j)\) is assigned as the prediction. This contrasts directly with PUF’s probabilistic relationship prior. The frequentist baseline therefore separates the effect of injecting dataset statistics, making any performance gap between it and PUF attributable specifically to the probabilistic evidence redistribution and accumulation in the latter.

Table 5 compares three edge-completion strategies on the 3DSSG test set: no prior, a frequentist prior, and our probabilistic prior. Two trends are evident. First, PUF without any prior already outperforms original FROSS, and is comparable to FROSS with the frequentist prior under both backends (e.g., 31.3% vs.% relationship recall for voxel), confirming that probabilistic node association and soft evidence redistribution provide gains independent of dataset statistics. Second, upgrading from the frequentist prior to our probabilistic prior yields a further consistent improvement (e.g., 43.1%\(\to\)​46.0% for PUF-Gaussian), demonstrating that the Bayesian integration of prior evidence with accumulated observations is more effective than naive frequency-based completion. The gap is especially pronounced for predicate mean recall, where the probabilistic prior’s ability to modulate confidence via the existence gate \(P_{ex}\) suppresses false-positive completions for rarely interacting class pairs.

9 Additional Quantitative Results↩︎

9.1 Object and Predicate Performance per Class↩︎

Table 6 reports per-class object recall on the 3DSSG test set. PUF-Gaussian achieves the highest mean object recall (65.8%) and matches or improves FROSS on the majority of classes. Notably, classes with high visual ambiguity or low frequency, such as chair, picture, and window, benefit from Dirichlet evidence accumulation, which allows soft class evidence to be aggregated across multiple partial observations rather than committed to a single argmax label. PUF-Voxel exhibits a similar pattern with a mean recall of 64.1%, confirming the representation-agnostic nature of the improvements. A few classes (e.g., bed, toilet) show minor regressions under one backend but not the other, reflecting some sensitivity of geometric overlap computation to the choice of 3D representation for objects with atypical spatial extent.

Table 7 presents per-class object recall on the ReplicaSSG test set, which has a substantially larger and more fine-grained label space (33 classes) than 3DSSG. PUF-Gaussian achieves the highest mean recall of 33.7%, improving over FROSS (28.8%) across a broad range of categories. The gains are most pronounced for small or frequently occluded objects such as bottle, shelf, and pillow, where multiple uncertain observations must be correctly fused to recover the instance. PUF-Voxel similarly improves over FROSS (30.2% vs.%), with particularly strong gains on chair and plant. Several rare categories (bench, railing, screen) remain at zero or near-zero recall for all methods, indicating that these classes are limited by 2D detection failure rather than 3D fusion.

Table 8 breaks down predicate recall by class on the 3DSSG test set. Without the relationship prior, PUF-Gaussian already improves over FROSS on the two most frequent predicates, attached to (33.4% vs.%) and standing on (55.6% vs.%), demonstrating that soft evidence redistribution benefits well-observed edges. Adding the prior yields dramatic gains on attached to (33.4%\(\to\)​61.6%) and part of (0.0%\(\to\)​13.3%), both of which suffer from poor co-observation and are therefore heavily reliant on prior completion. The connected to predicate remains at 0.0% across all online methods, likely due to 2D SGG failure. Consistent trends hold for the voxel backend, with the prior providing large lift on the same sparse predicates.

Table 9 reports per-class predicate recall on the ReplicaSSG test set, where no relationship prior is used. PUF-Gaussian outperforms FROSS on six of eight predicates and achieves the highest mean predicate recall of 26.2%. The most substantial improvements appear on above (44.4% vs.%), which depends on accurate spatial reasoning that benefits from uncertainty-aware node association. The against and attached to predicates remain at 0.0% for all methods, reflecting the near-absence of these relations in the test scenes, and the difficulty of zero-shot transfer from Visual Genome for such spatially subtle predicates. PUF-Voxel shows modest regressions on on and with relative to FROSS, suggesting that the discrete voxel overlap metric is less effective than the Gaussian Bhattacharyya coefficient for capturing the fine spatial distinctions required by these predicates.

9.2 Hyperparameters on Voxel Backend↩︎

Table 10 reports the sensitivity of the semantic bandwidth \(\sigma_{se}\) and birth density \(\lambda_{birth}\) on the 3DSSG validation set (w/o prior) under the voxel backend. The optimal setting is \(\sigma_{se}=0.2\) and \(\lambda_{birth}=0.3\), which differs slightly from the Gaussian backend (\(\sigma_{se}=0.3\), \(\lambda_{birth}=0.4\)). This shift suggests that the voxel containment score provides a sharper spatial signal than the Gaussian Bhattacharyya coefficient, so a tighter semantic bandwidth is needed to prevent over-merging of spatially overlapping but semantically distinct objects. Similarly, a lower birth density compensates for the voxel backend’s tendency to produce higher spatial overlap scores, which would otherwise suppress new-node creation.

Table 11 presents the corresponding hyperparameter analysis on the ReplicaSSG validation set under the voxel backend. The optimal configuration is \(\sigma_{se}=0.4\) and \(\lambda_{birth}=0.4\), matching the Gaussian backend on this dataset. This convergence is consistent with the zero-shot evaluation setting of ReplicaSSG, where the 2D model’s class distributions are broader and less peaked than on 3DSSG, making the fusion layer less sensitive to the choice of spatial backend. The relationship recall peaks at 13.9% and gradually degrades on both sides of the optimum, with the steepest drop occurring at high \(\sigma_{se}\) values that over-merge distinct objects. The trends mirror those observed for the Gaussian backend in Table 5 of the main paper.

9.3 Sensitivity to Uncertainty Calibration↩︎

Uncertainty mis-calibration is a known issue with the softmax outputs from deep neural networks [44], , the predictions can be overconfident in high-probability classes. To verify if PUF is sensitive to such mis-calibrations, we evaluate PUF with temperature scaling [44], applying \(T \in \{0.5, 1.0, 1.5\}\) to calibrate softmax outputs before fusion. Results in Tab. 12 show robustness across all temperatures on both datasets.

Temperature scaling is a post-hoc uncertainty calibration method, with which PUF remains plug-and-play and training-free. More advanced methods [45] would give more accurate uncertainty calibration but require additional network retraining to varying degrees. The integration with these methods is therefore an interesting topic for future research.

10 Additional Qualitative Results↩︎

10.1 More Qualitative Results on 3DSSG↩︎

Figure 5 compares PUF’s qualitative outputs under the Gaussian and voxel backends on a representative 3DSSG scene. Both backends correctly recover most object instances and relationship edges present in the ground truth. The Gaussian backend produces slightly lower normalized entropy on most nodes and edges (e.g., floor: 0.10 vs., sofa: 0.14 vs.), reflecting its finer-grained spatial overlap modeling via the Bhattacharyya coefficient. The lamp near the sofa (labeled as other in 3DSSG) is a rare object and misclassified with the voxel backend, but also carries high uncertainty with the Gaussian representation. Despite these minor differences, the two backends produce consistent graph topology and class predictions for this scene, corroborating the representation-agnostic nature of our framework. The entropy values annotated on each node and edge further demonstrate that PUF’s Dirichlet representation provides interpretable, per-prediction confidence estimates regardless of the underlying 3D representation.

10.2 Qualitative Results on ReplicaSSG↩︎

Figure 6 visualizes PUF’s predictions on two ReplicaSSG scenes (Office0 and Room0) under both the Gaussian and voxel backends. In the Office0 scene, both backends correctly detect all major objects and recover the dominant spatial relationships such as near and with. In the more cluttered Room0 scene, both backends successfully reconstruct the majority of the scene graph structure. False predictions (marked in red) carry notably higher normalized entropy in both cases, consistent with the calibration behavior observed on 3DSSG in the main paper. A major source of error is missing objects (, atypical objects screen in Office0 and table in Room0) due to 2D detector failure. Although cannot be mitigated by PUF, this type of error will diminish as more powerful detection models are available. Overall, the qualitative results on ReplicaSSG confirm that PUF generalizes to unseen environments in a zero-shot setting, and that the uncertainty estimates remain informative across both backends and datasets.

References↩︎

---

1. Corresponding author.↩︎