1 Introduction↩︎

Understanding the semantics of 3D environments is fundamental for embodied AI, robotics, and augmented reality. An agent must recognize objects, maintain their identities over time, infer relationships and support language-based queries for grounding and planning, e.g., “pick up the mug next to the laptop”. Scene graphs are a natural representation for this goal. They compactly encode entities as nodes and their interactions as edges, enabling downstream reasoning about “what is where” and “what is interacting with what.”

Recent work on 3D scene graphs has largely evolved along two complementary but currently disconnected directions: Semantic-rich offline reconstruction prevents querying unless the whole process is complete, while real-time reconstruction supports querying but only with limited semantic information. The former utilizes offline open-vocabulary systems and prioritizes rich labels, attributes, free-form descriptions, and functional relations [1]–[5], typically leveraging CLIP [6] embeddings and vision-language reasoning. These methods require completing instance-level reconstruction before enriching the scene with captions, labels, and inter-object relations. The latter are real-time mapping systems that incrementally update object graphs but rely on closed sets of semantics or shallow feature matching, trading the semantic richness of recent large language models (LLM) or vision-language models (VLM) for responsiveness [7]–[9].

The core technical challenge of combining these two paradigms lies in the instability of incremental tracking. Since objects are often fragmented or merged incorrectly under partial views and occlusions, it is impractical to query a VLM during exploration. Not only would this degrade the system’s responsiveness, it would also waste massive computational resources on analyzing duplicate or unstable object tracks. Consequently, previous systems have to wait for a fully consolidated offline reconstruction before applying VLM reasoning.

Many semantically rich pipelines already build their geometric map incrementally [1], [3]. What they lack is this combination of a robust enough 3D scene representation to make on-the-fly enrichment worthwhile, and an efficient scheduling mechanism to make it affordable.

Our key observation is that semantic reasoning does not need to wait for a stable graph; it can actively contribute to stabilizing it. This behavior is enabled by two factors. First, rather than the pairwise greedy association used by prior methods, we accumulate per-voxel votes for each object track, producing substantially more robust object tracks, reducing the instability that motivated postponing semantics in the first place. Second, rather than brute-forcing VLM calls on every object at every frame, we treat VLM reasoning as a selective, asynchronous background process, one that not only enriches the graph with fine-grained attributes, but also actively repairs it by detecting and merging duplicate object tracks through semantic loop closure.

We therefore propose an asynchronous architecture in which a lightweight online mapper runs concurrently with two background VLM agents: a Critic Agent that detects and merges duplicate object tracks through semantic loop closure, and a Description Agent that attaches fine-grained visual attributes via a multi-target frame scheduler that amortizes VLM cost across informative frames. The resulting scene graph grows in semantic richness as exploration continues, yet never blocks online operation. Our method thus occupies the middle ground between offline open-vocabulary reconstruction and real-time closed-set mapping. It builds the graph incrementally during exploration, enriching semantics asynchronously rather than at frame rate.

To demonstrate the effectiveness of our approach, we extensively evaluate it on open-vocabulary semantic segmentation and three visual grounding benchmarks. In summary, we make the following contributions:

• We introduce ThinkGraphs, to our knowledge the first asynchronous architecture for open-vocabulary 3D scene graph construction. It decouples lightweight online mapping from heavyweight VLM reasoning, meaning the graph remains queryable during exploration and is continually enriched without blocking it.

• We propose a VLM-in-the-loop mechanism for semantic loop closure that detects and merges duplicate object tracks caused by incremental association drift, analogous to geometric loop closure in SLAM.

• We introduce a multi-target frame scheduler that selects a small number of informative frames for the Description Agent, each depicting multiple undescribed objects, so that a single VLM call enriches several nodes at once.

• We demonstrate that our incremental method outperforms all batch-based open-vocabulary scene graph methods on semantic segmentation and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25, showing that deferring all semantics to a post-hoc stage is unnecessary.

2 Related Work↩︎

Incremental 3D scene graphs. The incremental construction of 3D scene graphs focuses on updating structured spatial representations as new observations arrive. Kimera [10], [11] first demonstrated real-time metric-semantic mapping with hierarchical spatial layers, a design later extended by Hydra [8], which pioneered incremental scene graphs by integrating signed distance fields with hierarchical graphs of objects, places, and rooms. S-Graphs+ [12] introduced a factor-graph formulation for hierarchical optimization, while SceneGraphFusion [13] fuses geometric over-segmentation with a Graph Neural Network (GNN) for incremental semantic prediction. Wu et al. [9] propose a sparse RGB-only variant that combines SLAM-based point tracking with multi-view feature aggregation. Further incremental methods either reuse temporal history [14] or incorporate knowledge from past observations [15] to enhance scene understanding. All of these methods remain restricted to closed-set vocabularies, limiting their applicability in open-world settings. More recently, Clio [7] introduced language interfaces into incremental mapping, averaging open-set features across temporal object tracks to form clusters relevant to the task, although it was limited to a predefined task list.

Compared to prior incremental scene graph mappers, our goal is to keep the graph open-vocabulary and queryable while exploration is ongoing. We achieve this goal by decoupling lightweight online tracking from heavyweight semantic enrichment: the mapper maintains stable object identities and geometry, while background VLM agents progressively refine labels and attributes without blocking the main loop; spatial relations are updated deterministically from 3D geometry.

Batch-processing of scene graphs. Many open-vocabulary 3D systems represent semantics as language-aligned features stored in dense maps or spatial memories [16]–[21]. In contrast, batch scene graph pipelines commit these signals to discrete object nodes and relations, enabling compositional and relational queries.

ConceptGraphs [1] pioneered open-vocabulary 3D scene graphs by combining CLIP [6] embeddings with SAM2 [22] masks to construct 3D objects. Subsequent works extended this paradigm by adding hierarchical multi-level reasoning across rooms and floors [2], object affordances [4], and relational question answering [3]. Open3DSG [23] takes a learning-based approach by distilling 2D vision-language features into a graph neural network, producing language-aligned node and edge embeddings for open-vocabulary object and pairwise relationship prediction. PoVo [5] takes a different approach, operating on geometric superpoints rather than per-frame masks and merging them into 3D instances through combined semantic and mask coherence.

While some of these methods build their geometric map incrementally [1], [3], all defer semantic enrichment to a post-hoc stage and therefore require a complete reconstruction before the graph becomes queryable. Our method instead exposes an intermediate graph throughout exploration and improves it incrementally: Semantic refinement is asynchronous and revisitable, so the system can answer queries early with coarse semantics and later update the same nodes or edges as better evidence emerges.

VLM reasoning in 3D scene understanding. Large vision-language models bring contextual reasoning capabilities that go beyond the embedding similarity used for object association and labeling. In open-vocabulary scene graphs, ConceptGraphs [1] queries a VLM to caption each object node and an LLM to infer spatial edges, while BBQ [3] formulates object retrieval as deductive reasoning over a scene graph structure using an LLM.

For 3D visual grounding, LLM-Grounder [24] decompose complex natural-language queries with an LLM into sub-goals and iteratively invokes a 3D visual grounder (e.g., OpenScene [16] or LERF [25]) to localize targets in reconstructed scenes. SQA3D [26] benchmarks situated question answering that requires joint spatial and common-sense reasoning on 3D scans. Effective visual grounding in these pipelines often relies on visual prompting strategies such as Set-of-Mark [27], which overlays numeric tags on segmented image regions so that a VLM can refer to specific objects without fine-tuning.

Existing 3D mapping methods generally employ VLM/LLM reasoning in one of two ways: post-hoc enrichment after reconstruction, or synchronous query-time inference that blocks execution until the model responds. Our work targets the missing middle ground: We treat VLM reasoning as a background process that runs continuously, is scheduled selectively, and integrates results when available. This makes semantics available during mapping while controlling the computation via coverage-based frame selection rather than per-object per-frame querying.

Object-level association and merging. A key challenge in incremental scene graph mapping is maintaining consistent object identities as observations accumulate over time. Prior work addresses this through object-level SLAM [28], multi-object tracking and fusion [29], [30], and semantic object-based loop closure [31], [32]. However, association remains challenging in the presence of perceptual ambiguity and viewpoint changes.

In open-set mapping, ROMAN [33] aligns object submaps via graph-theoretic data association but assumes object nodes are already formed, while OpenVox [34] formulates incremental instance association as probabilistic voxel inference and Octree-Graph [35] proposes chronological group-wise merging with informative feature selection.

Most scene graph systems, whether incremental [8], [13] or batch-based [1], [5], rely on greedy pairwise association in learned embeddings (e.g., CLIP [6] or DINO [36]) and 3D geometric overlap. Such greedy strategies involve making early, local commitments based on limited evidence, which result in the accumulation of false negatives that lead to fragmented object tracks [3], [8]. BBQ [3] addresses this by consolidating duplicates in a periodic merge step, but repeated instances and viewpoint changes can still cause missed associations.

In contrast, our method couples probabilistic instance tracking with a VLM-based semantic loop closure mechanism that explicitly detects and corrects fragmented object tracks during online operation (3.3).

3 Method↩︎

We introduce ThinkGraphs, an incremental open-vocabulary 3D scene graph pipeline that fuses geometric and visual evidence from sequentially posed RGB-D frames \(\{(I_t^{\text{rgb}},\, I_t^{\text{depth}},\, \theta_t)\}_{t=1}^{N}\) (2), where \(\theta_t\) is the camera pose. Our system incrementally constructs a scene graph \(G_t = (\mathcal{V}_t, \mathcal{E}_t)\), where each node \(v_i \in \mathcal{V}_t\) corresponds to a confirmed object track \(T_i\), represented by a 3D point cloud \(\mathbf{P}_i\), a selected CLIP visual embedding \(\mathbf{f}_i^\star\), a consensus label \(\ell_i^\star\), and VLM-enriched attributes \(\mathcal{A}_i\). Edges \(\mathcal{E}_t\) encode spatial predicates derived from 3D geometry. Rather than requiring a complete reconstruction, the pipeline maintains a continuously queryable graph throughout exploration via three components: (i) a frontend that extracts grounded instance observations \(\mathcal{O}_t\), (ii) a backend that lifts and associates them into persistent 3D object tracks, and (iii) asynchronous VLM agents that refine the scene graph without blocking the mapping loop.

3.1 Frontend for grounded instance extraction↩︎

Given an RGB image \(I_t^{\text{rgb}}\) at time \(t\), we query Qwen3-VL-2B-Instruct [37] to produce a set of noun prompts \(\mathcal{L}_t = \{ \ell_j^t \mid j=1,\dots,C_t \}\), as its context-aware prompts yield fewer missed detections than tag-based alternatives (2). We feed these prompts into Grounded-SAM v2 [22], [38], [39], where Grounding-DINO [39] predicts text-conditioned boxes for each prompt, and SAM2 [22] then refines each box into a 2D instance mask. The frontend outputs a set of \(K_t\) grounded instances \(\mathcal{O}_t = \{ (M_k^t,\, \ell_k^t,\, q_k^t) \}_{k=1}^{K_t}\), where each mask \(M_k^t\) is paired with its matched prompt \(\ell_k^t \in \mathcal{L}_t\) and confidence \(q_k^t\). This ensures every mask carries a grounded label from the start, which our backend relies on for association and label assignment.

3.2 Backend for 3D Association and Track Representation↩︎

We adopt OpenVox’s [34] probabilistic voxel-consistency formulation as our association backbone, as voxel-level voting accumulates evidence across multiple observations, filtering out noisy detections over time, whereas greedy pairwise matching commits to early associations that can accumulate drift. The underlying voxel map is a sparse hash map that allocates cells only when RGB-D observations project points into them, so storage scales with observed surface rather than a prespecified volume. We adapt it for our open-vocabulary setting by replacing its SBERT caption pipeline with CLIP text-label embeddings for track association, computed directly from the detector’s label strings. These embeddings are invariant to viewpoint, unlike visual features. We therefore store visual embeddings separately for retrieval and grounding. Beyond the change of embedding, we introduce stricter observation filtering and confidence-weighted label voting to suppress noise and obtain more stable track representations.

Track scoring. Before association, we filter each 2D mask (erosion, subtraction of contained segments, area and confidence thresholds) and retain only the dominant 3D cluster via DBSCAN [40], which serves solely as a local per-mask denoising step. We then maintain a sparse voxel map that continuously records the spatial occupancy history of each track. Given a new observation \(O = (M_k^t,\, \ell_k^t,\, q_k^t) \in \mathcal{O}_t\), we compute an association score \(S(T_i, O)\) for each candidate track \(T_i\) using a weighted combination of geometric and feature similarities: \[S(T_i, O) = \lambda_{\text{geo}} \cdot S_{\text{geo}} + \lambda_{\text{feat}} \cdot S_{\text{feat}}. \label{eq:total95score}\tag{1}\] The geometric score \(S_{\text{geo}}\) represents a probabilistic voxel-consistency vote in the intersecting volume: \[S_{\text{geo}}(T_i,O) = \frac{1}{|V(O)|}\sum_{v\in V(O)}\frac{c_{v,i}}{c_v}, \label{eq:geo95score}\tag{2}\] where \(c_{v,i}\) is the number of times track \(T_i\) has been observed in voxel \(v\), \(c_v = \sum_j c_{v,j}\) is the total observation count across all tracks in that voxel, and \(V(O)\) is the set of voxels occupied by observation \(O\). The feature score \(S_{\text{feat}}\) is the cosine similarity between CLIP text embeddings of the incoming observation label and the track’s current consensus label. With \(\lambda_{\text{geo}}=0.8\) and \(\lambda_{\text{feat}}=0.2\), the observation is merged into the highest-scoring track if \(S \ge \tau_{\text{assoc}}\). Otherwise, a new tentative track is instantiated. To prevent noisy detections from cluttering the scene graph, these tentative object tracks are promoted to confirmed nodes only after accumulating at least \(N_{\text{conf}}\) successful associations, effectively filtering out transient noise.

Confidence-weighted label voting. As observations accumulate, the same track may receive different label predictions across frames. To resolve these into a single consensus label, each track \(T_i\) maintains a label histogram \(\mathcal{H}_i\) over the labels observed so far. When a new observation with grounded label \(\ell\) and detection confidence \(q\) is associated to \(T_i\), the histogram is updated as \[\mathcal{H}_{i}(\ell) \;\leftarrow\; \mathcal{H}_{i}(\ell) + q, \label{eq:label95vote}\tag{3}\] and the consensus label is the highest-scoring label: \[\ell_i^\star = \underset{\ell}{\arg\max}\; \mathcal{H}_{i}(\ell). \label{eq:aggregated95label}\tag{4}\] Weighting votes by detection confidence, rather than uniform counting, ensures that high-quality detections naturally suppress noisy or ambiguous predictions.

Visual embedding selection. To support downstream retrieval and grounding tasks, each track \(T_i\) aggregates a bank of CLIP visual embeddings \(\mathcal{B}_i = \{\mathbf{f}_n^{\text{vis}}\}\), extracted from tight crops of its associated 2D segments. To ensure embedding quality, we apply two complementary filters. First, distant or heavily occluded viewpoints produce low-quality embeddings and should be discarded. An adaptive area gate maintains a running maximum segment area \(A_i^\star\) over the track history and only admits semantic/CLIP updates whose visible 2D segment area \(A_n\) satisfies \(A_n \geq \rho_\text{area} \cdot A_i^\star\). Second, we select a single representative embedding via text-guided scoring. Relying solely on the largest crop is insufficient: For instance, a large view of a wall may still be dominated by occluding furniture, confounding the CLIP feature. Instead, we encode the track’s consensus label \(\ell_i^\star\) (4 ) with the CLIP text encoder to obtain a semantic anchor \(\mathbf{f}_{\ell_i^\star}^{\text{text}}\) and retrieve the most semantically aligned view: \[\mathbf{f}_i^{\star} = \mathbf{f}_{s^*}^{\text{vis}}, \text{~where~} s^* = \arg\max_n\, {\mathbf{f}_n^{\text{vis}}}^\top \mathbf{f}_{\ell_i^\star}^{\text{text}}. \label{eq:text95guided}\tag{5}\] Spatial edges. To support relational grounding queries, we derive directed spatial edges deterministically from the axis-aligned bounding boxes of confirmed object tracks, following a similar approach to BBQ [3], as spatial relations can be reliably determined from geometric heuristics. Each edge encodes a directional predicate (e.g., left, under) and pairwise distance between target and anchor. Horizontal directions are resolved relative to a virtual room-center anchor to ensure viewpoint invariance.

3.3 Asynchronous Agentic Refinement↩︎

Enriching every node at every frame with a VLM would be prohibitively expensive, while deferring all enrichment to a post-hoc stage would mean that the system would not be queryable during exploration. We therefore introduce an asynchronous refinement layer comprising two VLM background agents that refine the graph without blocking the online loop (3). The Critic Agent detects and merges duplicate object tracks caused by incremental association errors, while the Description Agent attaches fine-grained visual attributes to each node.

Critic agent. Similarly to pose drift in SLAM, incremental object associations accumulate errors over time, spawning duplicate or fragmented tracks. Prior consolidation methods [3] rely on feature similarity and 3D overlap, but embeddings struggle to distinguish distinct instances of the same class or to re-identify single objects across varying viewpoints. Resolving such an ambiguity requires both fine-grained visual evidence and a broader scene context, motivating the use of a VLM as a verifier. We therefore introduce the Critic Agent: an asynchronous in-the-loop verifier that performs merge-only identity corrections in two stages. First, a lightweight candidate scheduling step proposes high-confidence duplicate pairs. Then, a visual verification step authorizes or rejects each merge.

To avoid overwhelming the VLM with spurious proposals, we apply a two-gate filtering mechanism to each pair of tracks \((T_i, T_j)\), including tentative ones, with observation counts \(m_i, m_j\) (the number of successfully associated detections for each track) and 3D bounding boxes \(B_i, B_j\). A geometric gate first discards any pair with \(\min(m_i, m_j) < 2\) or \(\mathrm{IoU}_{3\mathrm{D}}(B_i, B_j) < \tau_{\text{iou}}\), pruning noisy single-observation detections. Surviving pairs pass through a semantic gate that scores each pair by the cosine similarity \(s_{ij}\) between the SBERT [41] embeddings of their consensus labels. We use SBERT rather than CLIP here, as CLIP gives uniformly high similarity even for distinct labels, making threshold filtering unreliable.

Pairs with \(s_{ij} \ge \tau_{\text{sem}}\) enter a candidate buffer that decouples tracking from verification. For each buffered pair, we track the peak alignment via a running maximum \(\hat{s}_{ij} \leftarrow \max(\hat{s}_{ij},\, s_{ij}^{(t)})\), capturing the best-observed compatibility as new evidence arrives. The buffer is flushed periodically, ranking candidates by \(\hat{s}_{ij}\) and dispatching the top-\(K_{\text{critic}}\) to the verifier. At the end of the sequence, all remaining valid pairs are flushed to ensure complete coverage. When merging two tentative tracks results in a combined observation count that exceeds \(N_{\text{conf}}\), the merged track is automatically promoted to a confirmed node.

For each selected pair, we first keep high-visibility views of each track, measured by visible mask area. If both tracks are visible in the same frame, we choose the shared frame where both are most visible. Otherwise, we choose two high-visibility views with similar camera direction, camera position, object depth, and temporal proximity. To ground VLM attention without obscuring texture details, we employ Set-of-Mark (SoM) prompting [27], overlaying high-contrast SAM mask boundaries and unique identifiers directly onto the image. The model receives these marked images alongside a context tuple \(\mathcal{C} = \{ \text{ID}_{i,j}, \text{Label}_{i,j}, \hat{s}_{ij}, \text{geometry cues} \}\) that supplies geometric cues to reduce hallucinations from ambiguous masks, and returns a structured decision \(\mathcal{D} = \{ a \in \{\mathrm{\small Merge}, \mathrm{\small Keep}\}, r \in \text{String} \}\). When the verifier returns \(a=\mathrm{\small Merge}\), the Critic unifies point clouds, features, and label histograms of both tracks in the scene graph.

Description agent. While the tracker maintains object identities and coarse text labels, downstream tasks such as visual grounding require deeper semantic understanding. We introduce a Description Agent that runs asynchronously in a dedicated worker thread to enrich each scene graph node with fine-grained visual attributes, such as material, color, texture and physical state, which are critical for disambiguating instances of the same class.

Since a single frame typically depicts multiple objects, a small set of well-chosen frames can cover all targets while amortizing VLM cost and preserving scene context that individual crops would lack. We therefore introduce a multi-target frame scheduler that buffers 2D bounding boxes over a tumbling window of \(W_\text{desc}\) frames and identifies targets that lack a description or whose best visible area substantially exceeds their last-described area. It then greedily selects up to \(K_\text{desc}\) frames that maximize target coverage. A target \(T\) can be covered in frame \(f\) only if its normalized screen area \(\hat{A}_{T,f}\) is within a fraction \(\tau_\text{view}\) of its best view in the window. Each candidate frame is scored by \[U(f) = \sum_{T \in \mathcal{T}_f} w_T \cdot \ln\!\bigl(1 + \gamma\,\hat{A}_{T,f}\bigr), \label{eq:utility}\tag{6}\] where \(\mathcal{T}_f\) is the set of targets coverable in frame \(f\), \(w_T\) is a per-target weight that down-weights background classes and \(\gamma\) controls the area sensitivity. The greedy loop selects the highest-scoring frame, marks its targets as covered, and repeats until the budget is exhausted.

Selected frames are also rendered with SoM overlays and submitted in a single multi-image VLM call, along with each target’s current label, description, and attribute estimates as context. For each marked object, the model returns a refined label (, “winged armchair” rather than “chair”) and visual attributes (material, color, state, texture). Attributes accumulate in a per-node histogram mirroring the confidence-weighted voting of 3 , so repeated observations reinforce consistent evidence.

4 Experiments↩︎

We evaluate on open-vocabulary 3D semantic segmentation and 3D visual grounding, then analyze component contributions through ablations.

Datasets. For semantic segmentation, we evaluate on Replica [42] (eight photorealistic scenes: room0–room2, office0–office4) and ScanNet [43] (eight real-world RGB-D sequences: 0011_00, 0030_00, 0046_00, 0086_00, 0222_00, 0378_00, 0389_00, 0435_00). For visual grounding, we use three referring expression benchmarks with ScanNet: ScanRefer [44] (natural-language descriptions), Sr3D+ [45] (template-generated), and Nr3D [45] (free-form human-written). For both tasks, we follow the evaluation protocol of BBQ [3]; the eight ScanNet scenes are chosen to match their grounding setup. We adopt the same frame sampling stride of 5 for Replica and 10 for ScanNet.

Metrics. For segmentation, we report mean per-class accuracy (mAcc), mean intersection-over-union (mIoU), and frequency-weighted IoU (f-mIoU), following the open-vocabulary protocol of BBQ [3] with EVA02-CLIP [46] features. Per-point predictions are matched to ground truth via nearest-neighbor association in 3D. Generic categories (other, otherfurniture) are excluded. For grounding, we use GPT-4o [47] and report accuracy at IoU thresholds A@0.25 and A@0.5 on ScanRefer, and at A@0.1 and A@0.25 on Sr3D+ and Nr3D. The full evaluator prompt and an ablation over evaluator models are provided in the supplementary.

Implementation. For our CLIP embeddings we use the EVA02-CLIP [46] variant. Qwen3-VL-2B-Instruct [37] runs locally; for the Agents we use GPT-5-mini [48], called via its cloud API. Experiments use a single workstation (AMD Ryzen 7 9700X, NVIDIA RTX 5090, 64 GB RAM). We set the voxel backend to a resolution of 4 cm with an adaptive area ratio \(\rho_{\text{area}}{=}0.5\). Association balances geometry and feature similarity (\(\lambda_{\text{geo}}{=}0.8\), \(\lambda_{\text{feat}}{=}0.2\)) at a threshold \(\tau_{\text{assoc}}{=}0.4\); tentative tracks are promoted after \(N_{\text{conf}}{=}6\) successful associations. The semantic gate is set to \(\tau_{\text{sem}}{=}0.8\) and the IoU gate to \(\tau_{\text{iou}}{=}0.01\), with utility scale \(\gamma{=}100\). The Critic Agent flushes up to \(K_{\text{critic}}{=}20\) candidate pairs every \(W_{\text{critic}}{=}50\) frames. The Description Agent operates over tumbling windows of \(W_{\text{desc}}{=}30\) frames, selecting up to \(K_{\text{desc}}{=}3\) frames per window. The remaining hyperparameters are listed in the supplementary.

5 Results↩︎

We evaluate our method on two tasks: open-vocabulary 3D semantic segmentation and 3D visual grounding. We then analyze the contribution of individual components through ablations.

5.1 Open-Vocabulary 3D Semantic Segmentation↩︎

1 compares our method with prior open-vocabulary 3D methods on Replica and ScanNet. On Replica, our incremental method outperforms all prior methods on every metric (0.58 mAcc, 0.37 mIoU, 0.61 f-mIoU), with the largest gain on f-mIoU (+0.13 over BBQ-CLIP). On ScanNet, we obtain 0.75 mAcc, 0.44 mIoU, and 0.46 f-mIoU, achieving the highest mIoU and f-mIoU while remaining competitive with OpenVox (0.76) on mAcc. We attribute these gains to two factors. First, the voxel-consistency association with CLIP text-label embeddings produces more stable track identities than the pairwise visual similarity used by prior methods. Second, the text-guided feature selection in 5 ensures that each track is represented by the view most consistent with its consensus label, producing cleaner per-point semantic assignments. 4 illustrates this effect over time: Metrics improve steadily as additional frames are processed, demonstrating that the incremental pipeline converges to high-quality predictions without requiring a complete reconstruction.

4pt

5.2 3D Visual Grounding↩︎

We evaluate on three referring expression benchmarks of increasing difficulty: the template-based Sr3D+, the free-form Nr3D, and the natural-language ScanRefer. To handle the complexity of free-form and natural-language evaluation, we utilize GPT-4o [47] as our automated evaluator.

Sr3D+ and Nr3D. On both referral benchmarks (¿tbl:tab:sr3d95nr3d95overall?), our method substantially outperforms all baselines, improving over Open3DSG [23] by 15.3 A@0.25 on Sr3D+ and 18.8 on Nr3D. Performance remains consistent across difficulty and viewpoint splits (see supplementary for per-split breakdowns).

ScanRefer. On the ScanRefer benchmark ([tbl:tab:scanrefer]), our method achieves 52.9% A@0.25 and 37.6% A@0.5, outperforming Open3DSG [23] by 18.3 and 6.6 absolute points respectively. This reflects the combined effect of our core components: our backend provides stable track identities and high-quality representative visual embeddings, the Critic Agent reduces node fragmentation so that queries match the correct consolidated object, and the Description Agent provides fine-grained attributes that help disambiguate same-class instances.

5.3 Ablation Studies↩︎

To isolate individual component contributions, we conduct ablations on the eight Replica scenes.

Pipeline design choices. 2 ablates the backend pipeline on Replica. Switching from RAM++ [51] to Qwen3-VL [37] as the frontend tagger yields the largest mAcc gain (+0.09), as context-aware prompts produce fewer missed detections. Text-guided embedding selection drives the largest mIoU gain (+0.09): Naively selecting the largest-area crop as the representative embedding, as done in BBQ [3], is sensitive to foreground clutter. Scoring against the consensus label produces cleaner views. The adaptive area gate adds a further 0.06 mAcc by filtering low-quality observations, while replacing SBERT with CLIP text-label embeddings provides a smaller benefit but unifies the visual and textual feature space.

Agent contributions to grounding. 3 incrementally adds the two asynchronous agents to isolate their effect on Nr3D grounding accuracy. The base system includes the full backend with spatial relations, but no VLM agents. Adding the Critic Agent alone lifts overall A@0.25 from 38.5% to 39.3%, with gains concentrated on easy and view-independent splits where fragmented object tracks are the primary bottleneck (5). The Description Agent has a larger effect on hard referrals (+3.5 A@0.25), where fine-grained attributes are needed to disambiguate same-class instances. Combining both agents yields the best results in all splits (41.8% A@0.25 overall, +7.1 on hard), confirming that the two agents address complementary failure modes. An ablation isolating the architecture from the frontend VLM (a lighter RAM++ tagger) is provided in the supplementary, showing the gains are not merely due to a stronger tagger.

Runtime and cost. The online mapping path averages 1.45 s per keyframe, with the Critic and Description agents running off the critical path so they never block the online mapping loop. The multi-target scheduler keeps VLM usage low, about 26 calls per scene on average, more than two orders of magnitude fewer than naive per-object-per-keyframe prompting (\(\sim\)​9,300 on a typical scene). The primary runtime bottleneck is the frontend, not the agents. A full analysis is provided in the supplementary.

5.4 Limitations↩︎

Our framework has two main limitations. Firstly, the front-end models (Qwen3-VL + Grounded-SAM) prevent real-time operation. Replacing these models with more efficient alternatives would be the most direct way of overcoming this issue. Secondly, asynchronous agents inevitably lag behind the mapping frontier, meaning that objects discovered during fast exploration may remain unenriched for several frames. This limits the quality of grounding early in a sequence.

6 Conclusion↩︎

We present ThinkGraphs, an asynchronous architecture for open-vocabulary 3D scene graphs that decouples lightweight online mapping from heavyweight VLM reasoning. By running semantic refinement in the background, the scene graph remains queryable during exploration and grows in richness over time. A probabilistic voxel backbone with text-guided embedding selection provides stable object identities, while a Critic Agent corrects fragmentation through semantic loop closure, and a Description Agent attaches fine-grained attributes via multi-target frame scheduling. Our method outperforms all batch-based open-vocabulary scene-graph methods on semantic segmentation on Replica and ScanNet, and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25, showing that an incremental architecture can match and exceed batch-based alternatives. Future work could extend this asynchronous agent design with additional specialized agents, for example for richer relation prediction, higher-level scene reasoning, or task-oriented graph refinement, while also exploring its use in other structured representations that require both responsiveness and semantic depth.

Acknowledgements↩︎

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Deniz Bickici and Michael Pabst. This work was supported by the Alexander von Humboldt Foundation funded by the German Federal Ministry of Research, Technology and Space, and by Deutsche Forschungsgemeinschaft (DFG) under Germany’s Excellence Strategy EXC 2120/2 (390831618).

7 Per-Split Grounding Results↩︎

4 5 6 expand the overall grounding numbers from the main paper (Tabs. 2 and 3) into per-split breakdowns. On Nr3D and Sr3D+ [45] (4 5), ThinkGraphs outperforms BBQ [3] across nearly all splits. The one exception is Sr3D+’s hard split at A@0.1, where BBQ has a slight advantage (33.3 vs. 32.0). However, ThinkGraphs still leads at A@0.25 (32.0 vs. 22.7). For ScanRefer [44] (6), we analyze performance by split. Our approach reaches 71.8% A@0.25 on the unique split. Performance remains also strong on the more challenging multiple split, achieving 49.0% A@0.25, indicating that the attributes of the Description Agent effectively disambiguate instances belonging to the same class.

To isolate the contribution of our architecture, we replace the Qwen tagger with the more lightweight RAM++ [51] tagger and rerun the full system on Nr3D. Even with this weaker frontend, ThinkGraphs reaches 33.6 A@0.25, still exceeding Open3DSG (23.0) and BBQ (19.0). The 18.8 A@0.25 overall gain over Open3DSG thus splits into 10.6 from the architecture and 8.2 from the Qwen3-VL tagger, confirming that the improvement does not stem solely from a stronger tagger.

8 Grounding Model Ablation↩︎

Our main-paper results use GPT-4o [47] as the grounding evaluator, following the protocol of BBQ [3]. To test sensitivity to the evaluator, we re-evaluate the same ThinkGraphs scene graphs with GPT-5.4 [52] and Llama-4 Maverick [53] (7) on Nr3D. Overall accuracy stays within a narrow band around GPT-4o, ranging from a 1.9 A@0.25 drop with Llama-4 Maverick to a 3.6 gain with GPT-5.4, confirming that our improvements do not depend on a particular evaluator.

9 Agent Behavior Analysis↩︎

We provide an additional analysis of the agents’ behavior.

Critic Agent behavior. 6 shows representative decisions of the Critic Agent. In the first two cases, the agent correctly identifies duplicate tracks caused by viewpoint changes, such as a leather chair observed from two angles or a partially visible appliance fragmented across frames. In both cases, the agent reasons about spatial offset, containment, and visual consistency to authorize the merge. These are precisely the cases where greedy CLIP-similarity and 3D-IoU association fails: the viewpoint change lowers both embedding similarity and geometric overlap, so a pairwise matcher leaves the object fragmented, whereas the Critic recovers the merge from visual evidence. The third case illustrates a correct rejection. Two visually similar boxes stacked vertically are kept as distinct instances due to their low containment (7.1%) and vertical separation (0.23 m). Conversely, this is a case where IoU- or containment-based merging would erroneously fuse two distinct same-class instances; the Critic keeps them separate. The fourth case illustrates a failure mode in which the axis-aligned bounding box of a larger object overestimates its spatial extent. This causes a smaller, nearby box to appear highly contained and be incorrectly merged as a fragment.

To quantify these behaviors across scenes, 8 categorizes all Critic decisions into three types: deduplications, which merge two confirmed tracks referring to the same object and remove a redundant node; absorptions, which fold an unconfirmed fragment into an existing confirmed track, improving its geometry without changing the node count; and promotions, which merge two unconfirmed fragments whose combined observations exceed \(N_{\text{conf}}\) and recover an otherwise lost object. Across the eight scenes, 60 out of 144 calls (41.7%) are accepted. Absorptions dominate (38 of 60), followed by 17 deduplications and 5 promotions. Because promotions partially offset deduplications in the total confirmed count, the Critic’s full contribution is best measured through the downstream grounding ablation in Tab. 5 of the main paper.

5pt

Description Agent behavior. We illustrate the Description Agent’s refinement (Sec. 3.3 of the main paper) using scene0389_00 from ScanNet [43]. 7 shows the SoM-annotated [27] frames sent to the agent along with the current track labels. The agent refines coarse labels into fine-grained descriptions. For instance, two tracks both labeled board are disambiguated into padded ironing board (wall-mounted) and folding luggage rack (metal with straps), while nightstand becomes wooden nightstand (stained two-drawer). Beyond labels, the agent returns structured attributes (material, color, state, texture) that support the disambiguation of same-class instances during grounding. In one case (track_2), the agent incorrectly overrides the correct consensus label clothes hanger with wall-mounted reading lamp. Since the backend consensus label and the agent’s description coexist in the scene graph, incorrect descriptions do not destroy the original label.

10 Incremental Improvement↩︎

To demonstrate how our system improves over time, we report how the semantic segmentation scores evolve during exploration. We report metrics at four coverage checkpoints on both Replica and ScanNet (9). On Replica, performance improves steadily across all metrics, reaching 0.53 mAcc at just 50% coverage, already surpassing the final scores of most batch-based baselines in Tab. 1 of the main paper. On ScanNet, early-stage metrics start lower (0.09 mIoU at 10%) but recover quickly once diverse rooms are reached, with mIoU and f-mIoU more than doubling between 10% and 30% coverage.

3pt

11 Qualitative Results↩︎

Semantic segmentation. 8 compares our open-vocabulary semantic segmentation predictions with the ground truth for three Replica scenes. Room 2 accurately recovers most object boundaries, although large objects such as rugs remain a known challenge for incremental association. These objects span many frames and viewpoints, which makes it difficult to consolidate them into a single, consistent track. Office 3 shows strong overall coverage, but fine-grained objects (e.g. small desk items or thin wall-mounted fixtures) are occasionally missed when they fall below observation filtering thresholds or receive too few confident detections to be promoted to confirmed tracks. Office 4 reveals a notable evaluation artifact: objects assigned to generic ground-truth categories (other) are penalized even when our prediction is semantically plausible, such as a surface correctly identified as a table. Across all three scenes, remaining errors concentrate on small or heavily occluded objects and large ambiguous surfaces, while the majority of each scene is segmented correctly without requiring complete reconstruction.

Grounding. We also provide multiple examples on the grounding task in 9, covering Nr3D, Sr3D+, and ScanRefer. Each example shows the predicted bounding box alongside the ground truth for a given natural-language query.

12 Runtime and Cost Analysis↩︎

We profile end-to-end latency and VLM cost across all eight ScanNet sequences used in our temporal benchmark. All timings are reported with visualization, evaluation, and exports disabled in our latency-benchmark mode. 10 summarizes the per-scene breakdown.

Online path. Across all scenes, the online path averages 1.45 s per keyframe, of which the frontend contributes 1.10 s (76% of online time), making it the primary optimization target.

Asynchronous agents. Both VLM agents run in dedicated background threads, decoupled from per-keyframe processing. On average, the Description Agent takes 22.7 s to complete, with a mean lag of 28.9 keyframes (\({\sim}289\) original frames at stride 10). The Critic Agent takes an average of 19.1 s to complete, with a mean lag of 27.7 keyframes (\({\sim}277\) original frames). These lags are expected by design: the agents trade immediacy for selectivity by processing only high-value targets identified by the schedulers.

VLM call budget. Across the eight scenes, the system runs 205 VLM calls in total: 61 Description-Agent and 144 Critic pair-verification calls (60 merges, 41.7% acceptance). The multi-target scheduler is what keeps enrichment cheap: a naive per-object-per-keyframe strategy would need \({\sim}9{,}300\) calls on a single typical scene (, 39 tracks \(\times\) 238 keyframes), whereas our Description Agent covers all targets in only 4–16 calls per scene. The total end-to-end wall-clock time across all runs is 3,561 s (59.4 min), of which 305 s (8.6%) is spent draining pending asynchronous work after the last keyframe.

4pt

13 Frontend Design Choices↩︎

Why text-conditioned segmentation? Following prior open-vocabulary scene graph methods [4], [5], we adopt Grounded-SAM v2 [22], [38], [39] as our segmentation frontend. This is a natural fit for our pipeline, which anchors association, label voting, and embedding selection on per-mask text labels (Eqs. 1, 3, 5 of the main paper): each mask inherits a grounded label directly from the detection stage, requiring no separate labeling step. Category-agnostic alternatives such as standalone SAM2 [22] would require post-hoc labeling and tend to fragment objects into multiple regions (10), introducing noise into downstream components.

Why Qwen3-VL as the proposal model? Grounded-SAM requires a set of textual object candidates for each frame. Since label stability has a significant downstream impact, the choice of proposal model is critical to avoid the cascade of noisy or overproduced tags throughout our backend processes: association, voting, and embedding selection. Switching from RAM++ [51] to Qwen3-VL-2B-Instruct [37] yields the largest single improvement in our backend ablation (+0.09 mAcc, Tab. 4 of the main paper). RAM++ typically overproduces object names, introducing irrelevant categories that destabilize incremental tracking. Qwen produces concise, object-centric labels (“chair”, “table”, “shelf”) that provide a reliable anchor for the full pipeline. Its compact size enables fully local deployment without relying on external APIs, whereas GPT-based captioning, despite being viable, introduces substantial latency that is unsuitable for per-frame operation.

14 Prompts↩︎

We provide the exact prompts used for the frontend tagger, asynchronous VLM agents, and grounding evaluator.

15 Full implementation details↩︎

For completeness, 11 summarizes the key implementation parameters used in our system beyond those reported in the main paper. The released code will provide the complete configuration and default settings.

References↩︎