1 Introduction↩︎

() focuses on linking natural language expressions to their corresponding objects in complex 3D environments, demanding robust language understanding and spatial reasoning, and serving as a key component in applications such as augmented reality [1], [2], embodied navigation [3]–[5], intelligent agents [6], [7], robotic perception [8]–[11], and autonomous driving [12], [13].

In this task, most existing methods adopt a segment-then-select paradigm [14]–[19], where 3D object proposals are first extracted via scene-level segmentation and then matched to the textual query. Despite strong benchmark performance, such two-stage approaches suffer from inherent limitations, including error propagation from imperfect segmentation and high computational cost, often redundant in cluttered scenes with fine-grained relational cues.

Given a language description for localizing a target object, the contextual cues typically correspond to only a restricted region of the 3D scene. As illustrated in Fig. 1, instead of reasoning over all candidate objects (e.g., all chairs in the scene), the grounding model should focus only on spatially relevant areas guided by the anchor objects and spatial relations described in the utterance. This mirrors human spatial cognition, where people interpret referring expressions by using salient anchor objects and relational cues to narrow attention to the most relevant region of a scene rather than exhaustively comparing every possible object.

Additionally, human utterances are often free-form and exhibit ambiguity, incompleteness, and implicit spatial reasoning. Spatial relations may be partially specified or omitted, relying on assumptions that are intuitive for humans but challenging for models. This variability complicates grounding, as models must resolve ambiguity and infer missing geometric cues.

Consider the red chair in Fig. 1. The utterance may rely on the door to identify the target, yet in point clouds the door can be ambiguous, e.g. sharing color with the wall or partially missing due to occlusion and capture artifacts. Models that depend on the door as an anchor are thus prone to confusion. To improve localization, additional contextual cues are needed. Relations with nearby objects, such as a desk or bookshelf, provide complementary constraints that help disambiguate candidates and yield more stable bounding box estimation in cluttered or imperfect scenes.

Motivated by these problems, we propose PruneGround , a novel plug-and-play, prune-then-ground framework that leverages a () to prune irrelevant regions of a 3D scene, reformulate descriptions, and optimize a () for effective grounding. By focusing computation on language-relevant regions, our framework enables a target-centric and more accurate grounding process.

We devise a three-stage pipeline for effective ():

(i) (): We first render global multi-view RGB and depth images of the 3D scene and leverage a () to identify regions that are semantically relevant to the input query. This step effectively narrows the spatial search space and reduces unnecessary computation over irrelevant areas (Sec. 3.2).

(ii) (): We further render side views and cropped top-view regions centered on the pruned areas, and employ the () to decompose complex multi-anchor expressions into structured single-anchor statements. In addition, we augment the language descriptions with explicit spatial relations between the target and surrounding anchors that may be implicit or absent in the original utterance (Sec. 3.3).

(iii) (): Finally, we repurpose a detection-pretrained spatial () for language-conditioned grounding by aligning point cloud and linguistic representations within the pruned region and adapting the decoding scheme to produce the target prediction in a structured language form (Sec. 3.4).

By simply integrating ()-based scene pruning and description reformulation with an effective ()-based grounding framework, our plug-and-play PruneGround   achieves state-of-the-art performance across the three most popular () benchmarks - ScanRefer [1], Nr3D, and Sr3D from ReferIt3D [20]. Notably, it demonstrates strong generalization by delivering state-of-the-art results on all three ScanRefer settings (Tab. 1) and on 9 out of 10 Nr3D/Sr3D settings (Tab. 2).

2 Related Works↩︎

2.0.0.1 Two-stage vs. One-stage ().

Pioneering ScanNet-based benchmarks such as ScanRefer [1] and ReferIt3D [20], [21] have advanced (). Early supervised methods typically use separate language and scene encoders followed by cross-modal fusion.

(i) Two-stage approaches [14]–[17], [19], [20], [22]–[30] follow a proposal-and-selection paradigm: a 3D detector or segmentor first generates candidate proposals, then the proposal best matching the query is selected. By decoupling object localization and language grounding, they leverage strong geometric priors and mature detection pipelines through explicit candidate reasoning.

(ii) In contrast, one-stage approaches [31]–[39] directly localize the target in the 3D scene conditioned on the query. This enables end-to-end optimization and tighter language-geometry interaction, while avoiding error propagation from explicit proposal generation.

2.0.0.2 ()- and ()-based ().

To enhance language understanding in (), recent works incorporate ()s [38], [40]–[46] and ()s [47]–[53] into the grounding pipelines.

(i) Zero-shot approaches exploit pretrained ()s for open-vocabulary () without task-specific 3D supervision. ZSVG3D [38] uses ()-driven visual programming and modular 3D reasoning; VLM-Grounder [47] performs iterative grounding over stitched multi-view images and projects ensembles to 3D boxes; and SeeGround [54] converts 3D scenes into query-aligned rendered views with spatial textual descriptions to better match () inputs.

(ii) Fine-tuned approaches embed ()s/ ()s into 3D grounding networks with task-specific box decoders. LIBA [48] mitigates point-language granularity mismatch via multi-granularity alignment, dynamic cross-modal adapters, and ()-guided hierarchical reasoning. ReGround3D [44] introduces a reasoning-oriented 3D grounding benchmark and a ()-based chain-of-grounding framework that interleaves reasoning and localization.

2.0.0.3 Structured language reasoning for ().

Accurate () requires reasoning over attributes, relations, and viewpoint-dependent expressions. Recent works improve language grounding by converting free-form utterances into structured semantic forms. ViewRefer [40] uses ()s to enrich queries with view-aware templates, while EDA [36] decomposes descriptions into fine-grained semantic elements for dense alignment with point cloud instances. ViewSRD [55] restructures multi-anchor expressions into simpler single-anchor statements to disentangle target-anchor relations.

Similarly, we decompose multi-anchor utterances into single-anchor relations. Unlike purely text-based restructuring [40], our method renders multi-view images and uses a () to decouple relations, remove redundancies, and supplement missing geometric cues, yielding richer perspective-aware descriptions.

3 PruneGround↩︎

3.1 Overall Architecture↩︎

As shown in Fig. 2, given a 3D scene represented by point coordinates and color features \(\mathcal{S} \in \mathbb{R}^{N \times 6}\) and a natural language query \(\mathcal{Q}\)¹, our goal is to localize the referred target object. Instead of grounding over the entire scene, we propose a three-stage framework that progressively reduces spatial redundancy and refines language understanding.

(i) We introduce () (Sec. 3.2), where multi-view renderings (top and oblique views) are aligned with the query using a pretrained () to identify semantically relevant regions. Irrelevant 3D points are removed, yielding a pruned scene that reduces clutter and narrows the search space.

(ii) We present () (Sec. 3.3). Using top and side views of the pruned scene, we identify target and anchor objects and reformulate complex expressions into structured single-target\(\rightarrow\)single-anchor relations. Redundant statements are filtered, and the query is enriched with geometric cues inferred from multi-view semantics.

(iii) Given the pruned scene and reformulated description, we develop () (Sec. 3.4), a language-conditioned grounding model repurposed from a detection-pretrained spatial (), which jointly reasons over language and point cloud representations to predict the target object.

3.2 Language-Guided Spatial Pruning (sec:LGSP)↩︎

Let an indoor 3D scene represented as a point cloud with coordinates and color, \(\mathcal{S} \in \mathbb{R}^{N \times 6}\), where the \(x\)-\(y\) plane aligns with the floor and the \(z\)-axis points upward, opposite gravity. This canonical alignment enables consistent multi-view rendering and spatial reasoning.

3.2.0.1  Global Multi-View Rendering.

The () module first renders a set of structured multi-view observations from \(\mathcal{S}\). Specifically, we generate one top-view image that serves as a canonical spatial layout of the scene, together with four oblique views. The top view captures the global spatial arrangement along the horizontal plane but may lack semantic details of vertically structured objects. To complement this, each oblique view provides an RGB image encoding object semantics and a corresponding depth map encoding geometric structure.

To construct the oblique views, we compute the axis-aligned bounding box (AABB) of the scene to define its spatial extent. The camera centers are placed at the midpoints of the four upper edges of the AABB. Each camera is oriented toward the center of the opposite vertical face, ensuring that the field of view covers the corresponding side of the scene (see Appendix Fig. 4). RGB and depth images are rendered simultaneously under this configuration. Implementation details of the camera intrinsics and rendering settings are provided in the Appendix Sec. 7. The final multi-view representation thus consists of one top-view image, four oblique RGB images, and four corresponding oblique depth maps, as shown in Fig. 2.

3.2.0.2  Language-Guided Region Selection.

The rendered multi-view images, together with the language query \(\mathcal{Q}\), are fed into a frozen (). Conditioned on the textual query, the () analyzes the semantic and spatial correspondences across views and predicts a relevant 2D bounding region on the top-view image that is most consistent with the description.   \[\begin{align} \boldsymbol{\textcolor{green}{\mathbf{bbox}^{2D}}} &= (x_{\min}, x_{\max}, y_{\min}, y_{\max}) \\ &= \boldsymbol{\textcolor{cyan}{\mathrm{VLM}}} (\mathcal{V}_{\text{global}}, \boldsymbol{\textcolor{orange}{\mathcal{Q}}}) \end{align} ~\] where \(\mathcal{V}_{\text{global}}\) denotes the global multi-view input, and \((x_{\min}, x_{\max}, y_{\min}, y_{\max})\) define the boundaries of the retained region.

Details of the structured prompting strategy for the () are provided in Fig. [fig:prompt952d95bbox95estimation] of Appendix Sec. 8.

3.2.0.3  Post-processing and 3D Projection.

To ensure spatial stability, we apply post-processing to the predicted 2D bounding box in the top view by enforcing a minimum area constraint. If the box falls below this threshold, it is proportionally expanded to maintain sufficient coverage. The refined box is then projected back into 3D space. Since the top view lies on the \(x\)-\(y\) plane (with a default resolution of \(512 \times 512\)), we retain the full extent along the \(z\)-axis and select points whose \((x, y)\) coordinates lie within the box, discarding the rest to obtain the pruned point cloud \(\mathcal{S}'\).

For compact scenes where the area or pruned region is smaller than \(1\,\text{m} \times 1\,\text{m}\), we retain the entire scene or expand the region accordingly to avoid unnecessary information loss.

3.3 Multi-View-Conditioned Description Reformulation (sec:MCDR)↩︎

Natural language descriptions in () are inherently human-centric and often reflect intuitive reasoning rather than complete spatial specifications. As a result, relevant objects or spatial relations may be omitted or only implicitly expressed. While ViewSRD [55] decomposes expressions into target-anchor pairs to facilitate relational learning, it is purely text-driven and does not condition on visual semantics. Consequently, it may miss spatial cues that are visually evident but linguistically absent. To address this, we propose an () module that restructures descriptions into target-anchor relations under multiview visual conditioning. By incorporating semantic cues from top and side views of the pruned region, our method simplifies queries into single-target\(\rightarrow\)single-anchor statements while recovering missing spatial information from scene geometry.

3.3.0.1 Local Multi-View Rendering.

For the pruned scene \(\mathcal{S}'\) represented by a 3D bounding box, we render two side-view RGB images from the \(x\)- and \(y\)-axis directions of the selected top-view image’s bounding box. For the camera aligned with the \(x\)-axis, the viewing direction is set to pass through the centers of the two faces parallel to the \(y\)-axis. The camera is then gradually elevated to reduce occlusions from surrounding furniture. To further mitigate occlusions from structural elements (e.g., walls), the camera orientation is constrained to face outward from the center of the original scene. The camera along the \(y\)-axis is configured using the same procedure. In this step, depth maps are discarded to simplify the input and emphasize semantic content. A visualization of the camera configuration and rendering pipeline is provided in Fig. 5 of Appendix Sec. 7.

3.3.0.2 Description Simplification and Augmentation.

Given rendered local multi-view images \(\mathcal{V}_{\text{local}}\) and the original description \(\mathcal{Q}\), we continue to leverage a frozen () to perform structured query reformulation: \[(t, \mathcal{T}, \mathcal{A}, \hat{\mathcal{S}}) = \boldsymbol{\textcolor{cyan}{\mathrm{VLM}}}(\mathcal{V}_{\text{local}}, \boldsymbol{\textcolor{orange}{\mathcal{Q}}}).\]

The () decomposes the query \(\mathcal{Q}\) into four components: (i) the target object name \(t\), (ii) a set of attributes of the target object \(\mathcal{T}\), which describe properties such as color, material, and shape, (iii) a set of anchor object names \(\mathcal{A}\), and (iv) a set of target–anchor relational statements \(\hat{\mathcal{S}}\), each expressed as: \(\langle \text{target}, \text{target attributes} \rangle + \langle \text{spatial relation} \rangle + \langle \text{anchor}, \text{anchor attributes} \rangle.\)

By prompting the () to augment relational information using the local multi-view images, the resulting relational statements incorporate not only reformulated content from the original description but also additional target–anchor pairs. Specifically, anchors that do not explicitly appear in the original description but are observable in the local multi-view images may be introduced, leading to enriched relational supervision grounded in visual context. Details of the structured prompting strategy for the VLM query are provided in Fig. [fig:prompt95MCDR] of Appendix Sec. 8.

3.4 LLM-based Grounder (LLM-Grounder)↩︎

Given a reformulated referring expression and a pruned 3D point cloud as input, we build upon the SpatialLM framework [56], originally developed for unconditional 3D object detection, and repurpose it for language-conditioned grounding. The architecture follows an Encoder–Projector–LLM design.

A point cloud encoder (Sonata [57], a PTv3 variant with 108.8M parameters [58]) first extracts geometric features from the pruned scene. Unlike the original SpatialLM, which processes point clouds without linguistic conditioning, we inject the referring expression into the multimodal sequence to enable target-aware reasoning. Specifically, the geometric features are projected via a Multilayer Perceptron (MLP) into the language embedding space and concatenated with the text token embeddings of the referring expression, forming a unified multimodal input sequence. The () (Qwen2.5-0.5B [59]) processes this sequence and predicts a single target object matching the referring expression, in a structured language format: \[\texttt{<}c,\; p_x, p_y, p_z,\; s_x, s_y, s_z\texttt{>}\] where \(c\) denotes the object name, \((p_x, p_y, p_z)\) the center position, and \((s_x, s_y, s_z)\) the size of the bounding box along each axis. This contrasts with the enumerative output of the original SpatialLM, which produces structured tokens for all detected objects in the scene. Finally, the predicted bounding box parameters are mapped back to the original scene coordinate space to obtain the final grounding result.

4 Experiments↩︎

4.1 Experimental Settings↩︎

4.1.0.1 Datasets.

We evaluate our framework on three widely used () benchmarks. (i) ScanRefer contains 51,583 human-written referring expressions for 800 ScanNet scenes [21]. Following the official split, we use 36,665 samples for training and 9,508 for validation. Based on object uniqueness within a scene, the dataset is divided into unique (the target class appears once) and multiple (the target class appears multiple times) subsets. (ii) Nr3D consists of 37,842 human-written referring expressions grounded to annotated objects across 641 ScanNet scenes (511 for training and 130 for validation), covering 76 object classes. Each expression refers to a target object among same-class distractors. For evaluation, the dataset is partitioned into easy (one distractor) and hard (multiple distractors) subsets. Expressions are further categorized as view-dependent or view-independent, depending on whether grounding requires a specific viewpoint. (iii) Sr3D contains 83,570 automatically generated referring expressions from 1,018 training and 255 validation ScanNet scenes. Unlike Nr3D, Sr3D uses predefined templates and relies purely on spatial relations to distinguish same-class objects. We follow the standard evaluation protocol consistent with Nr3D.

4.1.0.2 Implementation Details.

We use Qwen2.5-VL-3B-Instruct [60] as the frozen () for both () and (). The top-view, oblique-view, and side-view images are rendered at a resolution of \(512 \times 512\) pixels with a field of view (FoV) of \(90^\circ\) (see Fig. 4 and Fig. 5 of Appendix Sec. 7). () adopts the Qwen-based SpatialLM [56] (Qwen2.5-0.5B) and initialize from its pre-trained checkpoint. Following [56], we shift the point cloud and target bounding box coordinates to ensure non-negative values, then quantize them into 1,280 bins at 2.5cm resolution. The LLM predicts integer values, which are mapped back to continuous coordinates via inverse transformation. All experiments are implemented in PyTorch and run on an NVIDIA A100 80 GB GPU. Further details of the pre-trained model initialization and training procedure are provided in Appendix Sec. 10.

4.2 Experimental Results↩︎

4.2.0.1 ScanRefer dataset.

Tab.  1 compares our method with prior approaches on the ScanRefer benchmark. On the challenging Multiple subset, where multiple candidates share the same class as the target, our pruning strategy effectively removes objects in irrelevant regions, achieving 57.6% Acc@0.25 and 49.4% Acc@0.5. As a result, our method achieves 63.8% Acc@0.25 and 56.1% Acc@0.5 overall, establishing a new state of the art in all three settings: Unique, Multiple, Overall.

2pt

2pt

4.2.0.2 Nr3D and Sr3D datasets.

As shown in Tab.  2, on the challenging Hard subset - where the target object is distracted by other objects of the same class - our approach achieves 68.8% and 76.5% on Nr3D and Sr3D respectively. As a result, we establish a new state of the art in 9 out of 10 settings, with Overall accuracies of 75.1% and 81.5%, respectively.

4.2.0.3 Qualitative analysis.

As shown in the qualitative examples (Fig. 3), () effectively prunes irrelevant regions, reducing candidate objects and directing attention to language-relevant areas. Meanwhile, () rewrites descriptions into a simpler, more structured form, enriching them with additional attributes and spatial relations-even beyond the original input-while maintaining consistent length and semantic density. Together, these components enable () to better align language with the target object for accurate bounding box prediction without being distracted by irrelevant candidates or missing spatial relationships in the original utterance.

4.3 Ablation Study↩︎

2pt

4.3.0.1 Impact of each proposal module.

As shown in Tab. 3, incorporating () consistently improves performance across both Unique and Multiple subsets by removing irrelevant regions, yielding a +3.2% Acc@0.5 gain in overall accuracy. Building on this, () further boosts results through description reformulation: simplification alone provides noticeable improvements across both subsets (Overall +1.7% Acc@0.5), while combining simplification and augmentation delivers the best performance, bringing an additional +2.1% Acc@0.5 gain in overall.

4.3.0.2 Candidate Reduction Rate.

Next, we conduct an ablation study to validate the effectiveness of the proposed () module. We first evaluate it on the Sr3D Hard subset, where each query contains at least two distractor objects from the same category, providing a challenging setting for candidate reduction before grounding. A candidate is considered removed if the pruned scene contains no spatial region belonging to that object. As shown in Table 4, our module reduces grounding candidates from 2.43 to 1.28 on Sr3D Hard and from 5.72 to 1.62 on the ScanRefer Multiple split, effectively narrowing the grounding search space.

4.3.0.3 Target recall of LGSP under varying numbers of oblique views.

In Tab. 5, we evaluate how the number of oblique views affects relevant-region detection, where success is defined as covering \(>\)​75% of the target. Performance consistently improves with more views and further benefits from depth. Notably, using two opposing oblique views with depth already yields strong gains, while four views achieve the best result of 94.7% (+22.1% over the 1-view RGB baseline). Based on this, we adopt four oblique views with depth as a good trade-off between accuracy and efficiency, with optional box expansion to increase target coverage in practice.

2pt

4.3.0.4 VLM selection justification

While larger VLMs such as Qwen2.5-VL-72B, used in SeeGround [54], and frontier APIs such as ChatGPT-5 and Gemini 3 Pro offer stronger general-purpose reasoning, they incur substantial latency (\(>\)​10s/sample) and cost, limiting large-scale evaluation. As shown in Tab. 6, we compare four VLMs with a no-pruning baseline, which retains the full \(512\times512\) top-view image, across pruned area, target recall, and runtime. All four VLMs achieve similar recall (\(94.7\)–\(95.7\%\)) while removing \(83.7\)–\(88.3\%\) of irrelevant regions, suggesting that the pruning task is largely saturated and gains little from frontier-scale reasoning. Qwen2.5-VL-3B-Instruct [60] runs in \(1.4\)s per stage, about \(8\)–\(9\times\) faster than the alternatives, while remaining within \(1\%\) recall of the best model.

5 Conclusion↩︎

In this paper, we introduce PruneGround , the first framework to leverage spatial pruning and detection-pretrained spatial () repurposing for (). It integrates three key components: (i) () to remove irrelevant regions using a frozen () with semantic and geometric cues, (ii) () to reformulate the description into a simpler form while augmenting it with semantic and relational spatial information, and (iii) (), an ()-based grounding model that aligns point cloud features with linguistic representations for target-selective prediction. Our plug-and-play framework achieves state-of-the-art results on three most popular point cloud benchmarks (ScanRefer, Nr3D and Sr3D). Notably, it demonstrates strong generalization by delivering state-of-the-art results on all three ScanRefer settings and on 9 out of 10 Nr3D/Sr3D settings.

Limitations↩︎

Our pipeline is limited by error propagation across its three stages. Errors in () can produce incomplete or noisy candidates, while mistakes in language understanding in () may misinterpret attributes or spatial relations. Since () relies on these intermediate outputs, early-stage errors can accumulate and amplify, reducing robustness in cluttered scenes, ambiguous descriptions, and fine-grained spatial reasoning tasks (see Fig. 7 of Appendix Sec. 12).

Furthermore, future work should focus on developing more lightweight and adaptable approaches to improve generalization across diverse real-world environments.

Acknowledgement↩︎

This work was supported in part by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645; by Coefficient Giving; by SwissAI; by Hector Foundation; by Schmidt Sciences; by the Frontier Model Forum and AI Safety Fund;by the Canadian AI Safety Institute Research Program at CIFAR; by the Canadian AI Safety Institute Research Program at CIFAR through a Catalyst Award; by the Survival and Flourishing Fund; by the Cooperative AI Foundation; and by the Division of Engineering Science at the University of Toronto. Resources used in preparing this research project were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.

6 Cognitive Motivation↩︎

Our motivation is also consistent with evidence from human visual cognition. Humans rarely interpret a complex scene by exhaustively comparing all objects of the same category. Instead, visual search is guided by selective attention [67], [68]: early perceptual processing extracts broad cues in parallel, while more complex object identification and relational reasoning are applied only to a limited set of task-relevant regions. In situated language understanding, linguistic cues further modulate this attentional process. Eye-tracking studies show that listeners rapidly direct their gaze toward objects mentioned or implied by an unfolding utterance, indicating that language incrementally guides visual attention in the surrounding environment [69], [70].

This cognitive process is especially relevant to (). Spatial language typically describes a target object, or located object, with respect to one or more reference objects, such as "the chair closest to the door" or "the chair under chalkboard". Cognitive studies of spatial language suggest that understanding such expressions involves identifying the relevant objects, assigning a reference frame, and evaluating the spatial relation between the target and the anchor object rather than globally reasoning over every possible candidate in the scene [71], [72].

Inspired by this human-like strategy, our PruneGround   avoids treating all object candidates as equally relevant. Given a referring expression, it first uses the mentioned anchors and spatial relations to narrow the search space to a language-conditioned local region (()). The model then performs fine-grained grounding within this region, augmenting with candidates that are spatially and semantically plausible (()). This design mirrors how humans use linguistic context to guide attention toward informative scene regions, reducing unnecessary comparison with irrelevant distractors while preserving the relational cues needed for accurate localization. Consequently, the localization (()) provides a cognitively grounded alternative to exhaustive segment-then-select pipelines, improving both accuracy and robustness in cluttered 3D environments.

7 Camera Setup Visualization↩︎

7.0.0.1 Global View Rendering.

We provide a visualization of the camera setup for Global View Rendering - a part of () in Sec. 3.2. As shown in Fig. 4, four cameras are placed at the centers of the four upper edges of the 3D bounding box, each oriented toward the center of the opposite face.

7.0.0.2 Local View Rendering.

We provide a visualization of the camera setup for Local View Rendering - a part of () in Sec. 3.3. Given a 3D bounding box, we render side views in both the horizontal and vertical directions. As shown in the first example in Fig. 5, for each direction there are two possible camera positions that can cover the box; however, we select the one closer to the center of the original scene (the center of the black box). Note that the camera is then slightly elevated to obtain a clearer view and to prevent occlusion by surrounding furniture.

8 VLM Query Structure↩︎

In this section, we describe the query structure, which consists of two tasks.

8.0.0.1 Prompt for 2D Bounding Box Estimation.

The Fig. [fig:prompt952d95bbox95estimation] shows the prompt for 2D bounding box estimation. Given the input description, the scene size \((m, n, k)\), and the top-view image resolution \((w, h)\), the objective is to estimate the corresponding 2D bounding box on the top-view image. This prompt is used in for Language-Guided Region Selection in () (Sec. 3.2).

messages = [
    {
        "role": "user",
        "content": [

            {"type": "image", "image": "topview.png"},

            {"type": "image", "image": "view_1_rgb.png"},
            {"type": "image", "image": "view_1_depth.png"},

            {"type": "image", "image": "view_2_rgb.png"},
            {"type": "image", "image": "view_2_depth.png"},

            {"type": "image", "image": "view_3_rgb.png"},
            {"type": "image", "image": "view_3_depth.png"},

            {"type": "image", "image": "view_4_rgb.png"},
            {"type": "image", "image": "view_4_depth.png"},

            {
                "type": "text",
                "text": f"""
Task:
Given the description: "{description}", Determine the relevant region in the topview image as a 2D bounding box.

Image order:
The images appear in the following order:
1. topview.png  
2. view_1_rgb.png  
3. view_1_depth.png  
4. view_2_rgb.png  
5. view_2_depth.png  
6. view_3_rgb.png  
7. view_3_depth.png  
8. view_4_rgb.png  
9. view_4_depth.png  

Scene coordinate system:
- X axis: horizontal width = {m} meters
- Y axis: vertical length = {n} meters
- Z axis: upward height = {k} meters

Topview image:
- topview.png is a topview rendering of the scene
- width = {w} pixels corresponding to the X dimension
- height = {h} pixels corresponding to the Y dimension

Image coordinate system:
- (0, 0) is the top-left corner of the topview image
- x increases to the right
- y increases downward

Oblique views:
- Each oblique view consists of two consecutive images:
first the RGB image, then the corresponding depth image.
- The cameras are placed at the midpoint of the four top 
edges of the axis-aligned 3D bounding box.

View mapping:
- view_1 (images 2-3: view_1_rgb.png, view_1_depth.png)  
  camera looks from bottom to top in the topview
- view_2 (images 4-5: view_2_rgb.png, view_2_depth.png)  
  camera looks from right to left in the topview
- view_3 (images 6-7: view_3_rgb.png, view_3_depth.png)  
  camera looks from top to bottom in the topview
- view_4 (images 8-9: view_4_rgb.png, view_4_depth.png) 
  camera looks from left to right in the topview

Reasoning procedure:
Step 1: Identify possible target objects and anchors mentioned in the description.  
Step 2: Use the topview and oblique RGB-depth images to determine plausible spatial regions containing target candidates.  
Step 3: Determine the minimal bounding region that covers those candidates.

Bounding box requirements:
- cover all candidate regions that could plausibly contain the target object
- exclude regions that are unlikely to contain the target
- include anchors mentioned in the description only when they are spatially close to the target candidates
- if the anchors are far from the target candidates, they should not be included
- the bounding box should be as tight as possible while still covering all plausible target candidates
- the bounding box must not exceed the scene boundary and should avoid covering large empty areas

If you are uncertain, prefer a slightly larger box that still excludes clearly irrelevant regions.

Output format:
- Return ONLY four values: x_min, x_max, y_min, y_max
- All coordinates must be in pixel coordinates of the topview image.
- Output ONLY the four numbers separated by commas.
- Do not output any explanation or additional text.
"""
            },
        ],
    }
]

8.0.0.2 Prompt for ().

The () query for Description Simplification and Augmentation - a part of () (Sec. 3.3) is described in Fig. [fig:prompt95MCDR]. Given rendered local multi-view images and the original description, we prompt the () to augment the relational information by using the local multi-view images, thereby incorporating not only reformulated content from the original description but also additional target–anchor pairs.

messages = [
    {
        "role": "user",
        "content": [

            {"type": "image", "image": "side_1.png"},
            {"type": "image", "image": "side_2.png"},

            {
                "type": "text",
                "text": f"""
Task:

You are given two side-view images of a pruned 3D scene.

Images:
1. side_1.png
2. side_2.png

Description: "{description}"
Your goal is to analyze the description and the two side views to extract the target object and anchors, and reformulate the description into structured single target-single anchor utterances.

Reasoning procedure:
Step 1 -  Analysis
Analyze the description and side views to identify:
- the target object
- the list of anchors explicitly mentioned
in the description
Step 2 - Visual Anchor Detection
From the two side-view images, identify additional anchors that are visible in the scene but NOT mentioned in the description.
Step 3 - Reformulate Description
Reformulate the description into a list of single target - single anchor utterances.
Each utterance should describe the spatial relation between the target and one anchor.

Output Structure:
- target name
- anchors name:
[list of anchors mentioned in the description]
- anchors not in description but visible:
[list of anchors visible in side views but not mentioned in the description]
- reformulated_description:
    + target properties
    (short and concise description of the target's properties)
    + augmented target properties
    (additional target properties that can be inferred
    from the side views but are not explicitly described)
    + single target single anchor utterances
    (list of utterances where each describes the relation between the target and one anchor)
    + augmented single target - single anchor utterances
    (list of utterances using anchors that are not mentioned in the description but detected in the side views)

Constraints:
- target properties and augmented target properties must be short and concise.
- the TOTAL number of single target - single anchor utterances must be at most 5.
- each utterance must follow this format:
target (+properties) + spatial relation + anchor (+properties)
- Use only clear spatial relations such as:
left of, right of, in front of, behind, next to, under, on, ...
- Do not produce explanations.
- Return only the structured output.

Example target properties:
- the chair is wooden and yellow
Example utterance:
- the chair is behind the black coffee table
"""
            },
        ],
    }
]

9 Additional Ablation Study↩︎

9.0.0.1 Inference analysis.

In this study, we report the runtime and resource consumption of PruneGround   during inference. We evaluate our pipeline on GPU RTX A100, reported by Tab. 7.

9.0.0.2 Performance with other grounders.

Since our pruning mechanism is designed as a plug-and-play module, the pruned regions can be directly used as input to other grounders. To verify that reducing the search space improves grounding performance, we retrain ScanRefer [1] and ReferIt3D [20], which are foundational works that introduced both the methods and datasets for the () task. Both methods follow the segment-then-select paradigm. As shown in Tab. 8, integrating our language-based pruning consistently improves their performance, demonstrating its effectiveness for (). Additionally, we apply our pruning module to 3DVG-Transformer [22], a representative one-stage approach, and observe consistent improvements.

9.0.0.3 Ablation study of proposed modules on Nr3D and Sr3D.

We further provide an ablation study of each proposed module on the Nr3D and Sr3D datasets in Tab. 9, complementing the study on ScanRefer dataset in Tab. 3. Consistent improvements are observed across both datasets, confirming the effectiveness of the proposed modules.

2pt

9.0.0.4 Anchor verification analysis.

To assess whether () introduces meaningful spatial information rather than spurious content, we analyze the anchors before and after reformulation. For each augmented anchor (i.e., anchors not present in the original description but added by ()), we verify its existence against ScanNet object annotations within the pruned region. An augmented anchor is marked as verified if an instance of its class label actually exists in the pruned scene, and hallucinated otherwise. We perform this analysis on 500 randomly sampled descriptions from each dataset.

As shown in Tab. 10, () increases the average number of anchors per description from 2.14 (original) to 3.22 (reformulated) on ScanRefer, with similar trends on Nr3D and Sr3D. Among the augmented anchors, average 93.7 are verified as existing in the pruned scene, and 93.7% are spatially relevant to the target. The hallucination rate remains below 6.3%, demonstrating that visual conditioning effectively constrains the VLM to ground augmentations in actual scene content rather than fabricating plausible-sounding but non-existent references. This verification analysis confirms that the additional anchors introduced by () are predominantly grounded in the scene and contribute genuine spatial information, supporting the downstream grounding improvements.

4pt

9.0.0.5 Human evaluation of Reformulated Descriptions.

To assess the linguistic quality of () beyond downstream grounding accuracy, we conduct a human study comparing three description variants: (i) the Original free-form utterance, (ii) a Text-only Reformulation produced by prompting the same VLM with the original description but no visual input, and (iii) () (ours), which conditions the VLM on multi-view renderings of the pruned scene.

We randomly sample 150 descriptions from ScanRefer (50), Nr3D (50), and Sr3D (50). Three annotators with backgrounds in computer vision and natural language processing independently rate each description on a 5-point scale (1 = poor, 5 = excellent) along four dimensions: (i) Faithfulness – whether the reformulation preserves the original intent without distorting the target referent; (ii) Completeness – whether useful spatial information is added relative to the original; (iii) Spatial Precision – whether the spatial relations are accurate and specific (e.g., “left of” vs.vague “near”); (iv) Conciseness – whether the description is compact and free of redundancy. Annotators are provided with the corresponding 3D scene visualization and the ground-truth target bounding box. Inter-annotator agreement is measured via Krippendorff’s \(\alpha\).

2pt

As reported in Tab. 11, () achieves the highest scores on Faithfulness, Completeness, and Spatial Precision, outperforming both the Original utterance and the Text-only Reformulation. Notably, while Text-only Reformulation also enriches the original description with additional content, it underperforms () on Faithfulness and Spatial Precision – a gap attributable to its lack of visual grounding, which causes the VLM to fabricate plausible-sounding but inaccurate spatial relations. In contrast, () leverages multi-view scene context to anchor augmentations in actual geometry, yielding spatially precise and faithful descriptions. The Conciseness score of () is slightly lower than the Original, since reformulation inevitably expands the utterance, however, this trade-off is justified by the substantial gains in Completeness and Spatial Precision, which translate into improved downstream grounding. These results indicate that the improvements brought by () are not merely an artifact of training-time supervision, but stem from genuine enhancements in the linguistic quality of the referring expressions themselves.

10 Model Structure and Training Procedure↩︎

10.0.0.1 Qwen2.5-VL-3B-Instruct.

We use this frozen () for inference inside () and (). The language backbone consists of 36 transformer layers [73] with a hidden size of 2048, SiLU activations [74], RMS normalization [75], 16 attention heads, and grouped-query attention with 2 key–value heads. The model uses a vocabulary of 151,936 tokens, tied input–output embeddings, bfloat16 precision, rotary positional embeddings with \(\theta_{\mathrm{RoPE}} = 1{,}000{,}000\), and mRoPE scaling for multimodal position encoding, supporting a maximum context length of 128,000 tokens. Although the configuration specifies a sliding-window size of 32,768 tokens, sliding-window attention was disabled during inference. The visual encoder contains 32 layers [76] with hidden size 1280, 16 attention heads, \(14 \times 14\) spatial patches, spatial merge size 2, and selected full-attention blocks at layers 7, 15, 23, and 31; its output is projected to the 2048-dimensional language-model hidden space. We used the model strictly for inference, with caching enabled and without any parameter updates or additional training.

10.0.0.2 SpatialLM.

As (), we use SpatialLM1.1-Qwen-0.5B, a spatial-language causal decoder model. SpatialLM is designed with an Encoder–MLP–LLM architecture. We use Qwen2.5-0.5B [59] as the base LLM, Sonata [57] as the point-cloud encoder, and a two-layer MLP as the projector. The language backbone follows a Qwen-style transformer architecture [73] with 24 hidden layers, hidden size 896, intermediate size 4864, SiLU activations [74], RMS normalization with \(\epsilon=10^{-6}\) [75], and 14 attention heads with grouped-query attention using 2 key-value heads. The model uses a vocabulary of 151,936 tokens with tied input-output embeddings, rotary positional embeddings with rope_theta\(=1{,}000{,}000\), a maximum context length of 32,768 tokens, and no sliding-window attention. Spatial inputs are represented using dedicated point start, end, and point tokens, and encoded with a Sonata point backbone. The point encoder consumes 6-channel point features and uses five stages with channel widths \([48, 96, 192, 384, 512]\), depths \([3, 3, 3, 12, 3]\), multi-head attention widths \([3, 6, 12, 24, 32]\), patch size 1024, stride-2 downsampling between stages, masked-token support, and spatial ordering based on z-order, transposed z-order, Hilbert, and transposed Hilbert curves. Point features are mapped into the language-model space through an MLP projector.

10.0.0.3 Sonata.

As a frozen encoder of (), we used the encoder-only configuration. The encoder consists of five hierarchical stages with channel dimensions \([48, 96, 192, 384, 512]\), depths \([3, 3, 3, 12, 3]\), with a patch size of 1024 at each stage and stride-2 down-sampling between stages. The model uses pre-normalization, an MLP expansion ratio of 4, learned query/key/value biases, mask tokens, stochastic depth with a drop-path rate of 0.3, and no attention or projection dropout. Flash attention [77] is enabled for efficient inference, while relative positional encoding, attention upcasting, and softmax upcasting are disabled. Sonata serializes points using multiple space-filling curve orders, including Z-order, transformed Z-order, Hilbert order, and transformed Hilbert order, with order shuffling enabled.

10.0.0.4 Training Protocol.

Since ScanRefer, Nr3D, and Sr3D are all built upon the ScanNet dataset, we adopt a two-stage training strategy.

In Stage 1, the model is first pre-trained on the SpatialLM synthetic dataset, then further trained for 3 epochs on the same dataset with class labels mapped to the ScanNet taxonomy, and finally fine-tuned on ScanNet for 30 epochs. All steps in this stage are conducted on the object detection task by [56].

In Stage 2, the model is fine-tuned on the () task for 10 epochs with a batch size of 4 on each dataset (ScanRefer, Nr3D, Sr3D) separately. We use the AdamW optimizer with a learning rate of 1e-4, a cosine scheduler, and a warm-up ratio of 0.03; all other hyperparameters remain unchanged. All components (encoder, projector, and LLM) are jointly trainable.

11 Pipeline Visualization↩︎

In this section, we provide visualizations of each step in our PruneGround  pipeline in Fig.6: (i) (), (ii) (), (iii) ().

12 Failure Case Analysis↩︎

In this section, we discuss the remaining limitations of our proposed framework.

12.0.0.1 Rendering failures leads to pruning failures.

First, rendering oblique views of complex indoor scenes remains challenging, particularly under low lighting, with incomplete point clouds, or in multi-room layouts. In such environments, objects are frequently occluded by walls or surrounding furniture, resulting in low-quality renderings. As illustrated in Fig. 7 (a), these degraded images can adversely affect the performance of the VLM in identifying language-relevant regions, then lead to final incorrect localization.

References↩︎