Robust 3D Alignment of Generative Reconstructions via Partial Monocular Observations
July 01, 2026
Abstract
Aligning generative 3D reconstructions with partial monocular observations is a critical but under-explored challenge in computer vision. This task is inherently ill-posed due to severe asymmetries between noisy, sparse monocular inputs and dense generative priors, whose scale ambiguity and geometric hallucinations, combined with the lack of initial overlap, render traditional registration pipelines ineffective. To resolve these issues, we propose a training-free and interpretable geometric alignment framework that grounds generative 3D priors via a 3D similarity transformation (Sim(3)), which can recover accurate metric scale and pose. Specifically, we introduce an explicit scale factor to resolve metric ambiguity and employ a coarse-to-fine alignment strategy, leveraging geometry-aware descriptors for robust initialization and a decoupled closed-form solver for precision refinement. In addition, we introduce a Hallucination Filtering operation to effectively suppress outliers caused by hallucinated geometry. To evaluate alignment performance under these extreme conditions, we introduce GenPMOAlign–Where2Place, a rigorous benchmark specifically designed for Generative-to-Partial Monocular Observational Alignment. Experiments demonstrate that our method achieves stable and accurate registration, substantially outperforming both classical geometric pipelines and state-of-the-art learning-based baselines. Code and the benchmark will be publicly released.
3D registration ,point cloud registration ,similarity transformation ,generative reconstruction ,robotic manipulation
1 Introduction↩︎
Recent advances in embodied intelligence and robotic foundation models have significantly expanded the capability of robots to operate in open-world environments [1], [2]. To achieve reliable manipulation beyond constrained laboratory settings, robots must acquire a comprehensive understanding of scene geometry and object structure. In grasping, collision avoidance, and physical interaction, they must reason not only about visible surfaces but also about the overall shape, volume, and potential contact regions of objects [3]–[5]. However, the observations available in real-world scenarios are often limited to monocular or sparse sensory inputs. Occlusions, restricted viewpoints, and sensor noise frequently yield incomplete geometric information, which alone is insufficient for reliable planning and execution.
Unlike traditional reconstruction pipelines that rely on multi-view observations or dedicated depth sensors, recent generative 3D models can recover relatively complete object geometry from a single image or limited observations while plausibly inferring occluded regions [6]–[9]. By providing dense and complete object representations beyond the current sensor field of view, generative priors offer richer geometric constraints for downstream tasks such as grasp planning, motion planning, collision checking, and scene interaction.
Despite their advantages, generative 3D reconstructions cannot be directly deployed in robotic systems. The generated geometry typically resides in a normalized canonical space rather than the metric physical space in which robots perceive and interact with the world [10], [11]. Specifically, generative models often produce geometrically plausible shapes without reliable metric scale, and their coordinate systems are internally defined and decoupled from real-world coordinates [6], [7]. Consequently, the same reconstructed shape may correspond to either a real mug or a miniature toy replica, making it difficult to determine its true physical size and spatial location from the generated model alone. This discrepancy between canonical geometry and metric observations forms a fundamental canonical-to-metric gap. For robotic applications, such inconsistency directly affects downstream decision making. Even when the reconstructed shape is visually accurate, scale errors may lead to incorrect gripper configurations, failed collision checking, or inaccurate motion planning. From a geometric perspective, the problem therefore requires not only recovering the coordinate transformation between the canonical and metric spaces, but also estimating the missing absolute scale. Consequently, establishing a consistent mapping between generative priors and metric observations has become a prerequisite for deploying generative 3D models in real-world robotic systems.
Among these discrepancies, metric scale ambiguity is arguably the most fundamental obstacle. Unlike conventional point cloud registration, where both inputs are acquired from physical sensors and naturally share a common metric space [12], generative reconstructions are inherently scale-free. As a result, the alignment problem extends beyond rigid pose estimation and requires jointly recovering scale and pose under severe partial observations [13], [14]. Moreover, geometric hallucinations introduced by generative models, together with sparsity, noise, and substantial density mismatch in sensory observations, further increase the difficulty of reliable alignment.
Registering generative 3D reconstructions in canonical space to metric sensory observations is therefore an emerging yet under-explored problem and a key enabler for geometry-aware robotic manipulation, including grasping [4], [15], collision avoidance [5], and interaction [2]. As shown in Fig. 1 (A1) and (A2), the inputs to this task exhibit pronounced cross-domain discrepancies: the partial point cloud inferred from monocular or depth estimation [16] is typically sparse, noisy, and visibility-limited, whereas the prior produced by the generative model is dense and complete yet often contains geometric hallucinations and lacks a true metric scale [6], [7]; moreover, the two often have little to no initial overlap. As depicted in Fig. 1 (C1), traditional rigid registration methods that rely on overlap and correspondence assumptions are ineffective in this setting. Specifically, direct ICP-style approaches [17]–[19] are brittle under sparse and noisy inputs, often resulting in incorrect scale and rotation, as shown in Fig. 1 (B1). On the other hand, learning-based registration methods [20], [21], despite leveraging global context, can still fail due to contextual mismatch, prior bias, and density discrepancies, leading to noticeable misalignment in real scenes, as illustrated in Fig. 1 (B2).
To address these challenges, we propose a training-free and interpretable geometric alignment framework that grounds generative 3D priors into the metric world via a 3D similarity transformation (Sim(3)), which can recover accurate scale and pose, as presented in Fig. 1 (B3), (B4), and (B5). Specifically, we explicitly model the scale factor to resolve metric ambiguity and adopt a coarse-to-fine alignment strategy: we first obtain a robust initialization using geometry descriptors that are insensitive to hallucinations, and then optimize with a decoupled closed-form Sim(3) solver to handle zero-overlap cases and decoupled coordinate systems. In addition, we introduce a Hallucination Filtering operation that restricts optimization to observation-supported correspondences, suppressing outliers induced by hallucinated geometry. To evaluate alignment under these extreme conditions, we build GenPMOAlign–Where2Place, a rigorous benchmark tailored to generative-to-partial monocular observational alignment on cluttered indoor home or office images. As demonstrated in Fig. 1 (C2), our approach achieves stable and accurate registration, substantially outperforming classical geometric pipelines and strong learning-based baselines.
Our contributions are threefold:
We formalize the canonical-to-metric gap for generative 3D models in robotics as a partial-to-complete 3D similarity transformation registration problem under unknown scale, density mismatch, and low or zero overlap.
We propose a robust, training-free, and interpretable Sim(3) alignment framework that jointly estimates scale and pose while accounting for generative hallucinations and cross-domain density mismatch, enabling accurate recovery of metric scale and pose.
We construct GenPMOAlign–Where2Place to benchmark the cross-domain Sim(3) registration task from generative 3D priors to partial monocular or depth observations, and show that under this extreme regime our method consistently outperforms classical geometric pipelines and learning-based baselines.
2 Related Work↩︎
Classical Point Cloud Registration Classical point cloud registration aims to estimate a rigid or similarity transformation to align two or more point clouds, which are generally categorized into local registration and global registration. Specifically, ICP [17] alternates correspondence matching and error minimization for fine alignment, but is initialization-sensitive and can get trapped in poor local minima under large pose changes or partial overlap. To improve geometric modeling, GICP [22] unifies point-to-point and point-to-plane formulations and enhances robustness. LiTAMIN2 [23] introduces a symmetric KL-divergence term into the ICP objective to compare local distribution shapes. Voxelized GICP [18] and KISS-ICP [19] further emphasize efficient implementation and practical deployability. In recent years, classical frameworks have continued to advance in robust global estimation and efficient search [24]–[26]. More commonly, correspondence-driven pipelines first establish candidate correspondences using handcrafted descriptors (e.g., FPFH [27]), then obtain an initial pose via robust estimators such as RANSAC [28], and finally refine it with fast global optimization methods like FGR [29]. Moreover, TEASER [30] and GNC [31] provide stronger robustness and outlier rejection. In addition, max-clique-based approaches [32]–[34] further improve robustness by selecting mutually consistent correspondence subsets. From a geometric perspective, classical registration methods have achieved remarkable success when both source and target point clouds originate from physical sensors and exhibit similar geometric properties [17], [22]. Their robustness largely stems from explicit geometric constraints and interpretable optimization objectives. However, these methods generally assume sufficient overlap, relatively consistent point densities, and reliable initialization [20]. Consequently, their performance often degrades under severe partial observations, cross-domain discrepancies, and density mismatch.
Overall, classical methods are training-free and interpretable, but depend on sufficient overlap, consistent density, and reliable initialization, and thus degrade under cross-domain, partial, or density-mismatched settings.
Learning-Based Point Cloud Registration Driven by the success of deep learning in 2D vision, point cloud registration shifts from handcrafted features to learned features and correspondences. Early works learn local descriptors and keypoints (3DMatch [12]), while sparse or voxel convolution enables dense feature extraction (FCGF [35], D3Feat [36]). PARENet [37] boosts rotational robustness with position-aware rotation-equivariant features. For correspondence modeling, recent methods emphasize global consistency and coarse-to-fine filtering (DGR [38], CoFiNet [39], Predator [20]), with additional correspondence pruning and confidence estimation (ICC [40]). Transformer-based models further strengthen long-range context modeling and improve robustness in low-overlap, cluttered scenes (GeoTransformer [21] and follow-ups [41], [42]). Compared with classical methods, learning-based approaches seek to learn discriminative geometric representations directly from large-scale data [20], [35]. By leveraging learned features and contextual reasoning, these methods often demonstrate improved robustness to noise, clutter, and moderate overlap reduction [21], [39]. More recently, foundation-model-based registration methods have emerged as a promising direction for improving cross-domain generalization through large-scale semantic priors [43]–[45].
Despite these advances, cross-domain and zero-shot generalization remains challenging. In particular, SpinNet [46] and BUFFER [47] target more general descriptors and zero-shot settings. ZeroReg [43] uses 2D foundation-model semantics for object-level zero-shot registration. RARE [44] leverages foundation models for zero-shot registration and refinement. BUFFER-X [45] is training-free, improving cross-domain generalization with high efficiency. RAP [48] further explores generative flow-matching registration that directly produces registered point clouds across diverse scales and sensor setups.
Generative 3D Reconstruction for Robotics Recent advances in generative 3D reconstruction have enabled the recovery of complete object geometry from a single image or sparse observations [6]–[9]. Leveraging large-scale generative priors, these methods can infer plausible shapes for occluded regions and produce dense, semantically meaningful 3D representations. Such capabilities have attracted growing attention in robotics, where complete geometric understanding is essential for grasp planning, manipulation, scene interaction, and embodied reasoning [1], [3], [4], [15].
Several recent studies have explored the integration of generative 3D models into robotic systems [4], [15]. By providing shape completion beyond visible observations, generative priors can significantly improve object understanding under severe occlusion and limited viewpoints. However, unlike CAD models or sensor-acquired point clouds, generative reconstructions are often represented in normalized canonical coordinates and may contain hallucinated structures, scale ambiguity, and topological inconsistencies [6]–[9]. These characteristics introduce substantial challenges when aligning generative priors with real-world observations and remain largely under-explored in existing registration literature.
Scale-Aware Cross-Domain Alignment Scale-aware alignment has become increasingly important in heterogeneous registration problems, where source and target data are represented in different coordinate systems and possess inconsistent metric scales. Unlike conventional SE(3) registration, Sim(3)-based alignment jointly estimates scale, rotation, and translation, making it more suitable for cross-domain geometric matching [13], [14]. This capability is particularly important when aligning generative reconstructions with real observations, as generative models generally lack physically meaningful metric information [10], [11].
Heterogeneous data alignment (e.g., CAD-to-depth scans [49], [50] or generative priors-to-real observations [11]) typically requires solving Sim(3) to jointly estimate rotation, translation, and scale. Classical solutions build on generalized Procrustes analysis [13] or scale-separation techniques, and methods such as TEASER++ [30] improve robustness under high outlier rates via max-clique constraints, but still rely on accurate correspondences and high geometric fidelity. Category-level approaches (e.g., NOCS [10]) alleviate scale ambiguity by mapping observations into a normalized canonical space, yet depend on category-specific supervision and dense, high-quality observations, limiting generalization to open-world and cross-category settings. Hallucinated geometry from generative and foundation models often exhibits scale drift, structural incompleteness, and topological inconsistencies, making uniform Sim(3) solvers prone to failure under severe partiality and density mismatch [8], [9]. Unlike most prior Sim(3) alignment methods that assume high-fidelity correspondences, dense observations, or learned canonical mappings [10], [49], [50], the problem considered in this work is substantially more challenging. The source geometry may contain hallucinated structures, while the target observation is sparse, noisy, and incomplete. These characteristics jointly introduce scale ambiguity, density mismatch, and low-overlap correspondence uncertainty, motivating the development of a dedicated alignment framework tailored to generative-to-partial observational registration.
Hence, we propose a training-free and interpretable geometric alignment framework that grounds generative 3D priors into the metric world via a Sim(3), thereby recovering accurate scale and pose.
3 Methodology↩︎
We propose a scale-stabilized Sim(3) registration framework for aligning generative canonical 3D reconstructions with partial monocular observations, as illustrated in Fig. 2. Unlike previous methods that jointly optimize scale and pose until convergence, we decouple scale estimation into a guarded global initialization and a scale-locked local refinement. Our approach prevents the generative model from collapsing to fit partial observations and avoids the common shrinking degeneracy.
3.1 Observation Model and Problem Formulation↩︎
The system input consists of a monocular RGB image, a semantic mask, and an estimated depth map \(D(u,v)\). We model the projection from 2D pixels to 3D space using the pinhole camera model with intrinsic matrix \(K\).
For valid pixels \((u, v)\) within the target mask, the back-projected point in the camera frame is \[x_{cam}(u,v) = D(u,v) \cdot K^{-1} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}.\] After back-projection, we transform the point set \(X_{cam} = \{x_{cam}^i \in \mathbb{R}^3\}_{i=1}^N\) to obtain the observational point cloud \(P_{obs} = \{ q_i \in \mathbb{R}^3 \}_{i=1}^N\) and sample a source point cloud \(P_{rec} = \{ p_j \in \mathbb{R}^3 \}_{j=1}^M\) from the canonical mesh generated by SAM 3D [7]. We then estimate a similarity transformation \(S = \{s, R, t\} \in \mathrm{Sim}(3)\) that aligns the canonical shape to the metric observation: \[\min_{s, R, t} \mathcal{J} = \sum_{(i,j) \in \mathcal{C}} \left\| q_i - (s R p_j + t) \right\|^2,\] where \(\mathcal{C}\) represents the correspondence set, \(s \in \mathbb{R}^+\) is a uniform scale factor, \(R \in SO(3)\) is the rotation, and \(t \in \mathbb{R}^3\) is the translation.
3.2 Guarded Global Sim(3) Estimation↩︎
We first preprocess the data using FPFH, instead of learning-based descriptors, to keep the method training-free. However, in this task the significant scale discrepancy between the unit canonical source \(P_{rec}\) and the metric observation \(P_{obs}\) makes vanilla FPFH matching unstable. To address this issue, we introduce a heuristic scale factor \(s_{init}\) before correspondence search. We scale \(P_{rec}\) by \(s_{init}\) so that the support radius for normal estimation and FPFH extraction covers geometrically comparable regions. The descriptor \(\Phi(p)\) then becomes approximately transformation-invariant: \[s_{init} = \frac{\| \mathrm{extent}(P_{obs}) \|}{\| \mathrm{extent}(P_{rec}) \|},\] \[\Phi(p) \approx \Phi(s_{init} R p + t).\]
We further propose a Hallucination Filtering operation to suppress unsupported surfaces in the generative prior. Specifically, we restrict correspondence construction and optimization to regions that are geometrically consistent with the monocular or depth observation, and discard areas with insufficient observation support, preventing hallucinations from biasing scale and pose estimates.
We then use a RANSAC framework to estimate the global alignment from the FPFH correspondences. For a minimal subset of correspondences, the optimal Sim(3) parameters can be derived in closed form. The translation is decoupled via centroid alignment, the rotation is solved by SVD, and the scale is obtained by minimizing the residual variance: \[\Sigma_{pq} = \sum_{(i,j)\in\mathcal{C}} (q_i - \bar q)(p_j - \bar p)^\top,\] \[s^* = \frac{\operatorname{tr}(R^\top \Sigma_{pq})}{\sum_j \| p_j - \bar p \|^2},\] \[R^* \in \arg\max_{R\in SO(3)} \operatorname{tr}(R \Sigma_{pq}),\] \[t^* = \bar q - s^* R^* \bar p.\]
In the generative-to-partial setting, unconstrained Sim(3) estimation often yields degenerate scales, for example \(s \rightarrow 0\), to minimize error on noisy partial observations. To avoid this failure mode, we introduce a guarded estimation strategy. A hypothesis is considered valid only if its estimated scale falls within a physically plausible range relative to the heuristic initialization, namely \(s^* \in [\lambda_{\min} s_{init}, \lambda_{\max} s_{init}]\). If the scale violates this constraint, the hypothesis is rejected during the RANSAC loop, preventing hallucinated geometry from distorting the metric scale estimate. The accepted solution is denoted by \(S_{est} = \{ s^*, R^*, t^* \}\).
3.3 Scale-Locked Local Refinement↩︎
The standard approach typically performs joint optimization of \((s, R, t)\) on the local alignment obtained through RANSAC. However, we find that allowing the scale \(s\) to update during local refinement leads to significant shrinkage of the generated model, because the optimizer attempts to minimize the distance to the incomplete surface of \(P_{obs}\).
To address this issue, we propose a Scale-Locked Refinement. We fix the global scale \(\hat{s}\) estimated from the guarded RANSAC stage and restrict the local optimization to the \(SE(3)\) manifold. The refinement objective becomes \[\min_{R, t} \sum_{(i,j) \in \mathcal{K}} \left\| q_i - (\hat{s} R p_j + t) \right\|^2,\] where \(\mathcal{K}\) denotes the local correspondence set used during refinement.
By treating the scale as a hard constraint during the fine-tuning phase, we force rotation and translation to align the observed regions while preserving the global geometric integrity of the hallucinated parts. The final output is a metric-aligned object representation that is both geometrically stable and robust to hallucinated geometry.
4 Experiment↩︎
4.1 Benchmarks and Evaluation Metrics↩︎
Benchmarks We construct GenPMOAlign–Where2Place, a benchmark for generative-to-partial monocular alignment. It consists of 100 cluttered indoor home or office scenes, each with one image and 2–4 paired masks, totaling 338 point cloud pairs. For Where2Place [1], we use the provided RGB images and object masks. We reconstruct a complete canonical mesh with SAM 3D [7] conditioned on the image and mask, yielding a normalized canonical shape that may contain hallucinated geometry in occluded regions. We next estimate dense depth from the same image using Depth Anything [16] and back-project masked depth to obtain a partial object point cloud along with the surrounding scene point cloud. Since the generative mesh resides in a normalized canonical space while the observation point cloud is in metric space, this process inherently introduces scale inconsistency. Furthermore, the presence of partial observations and hallucinated geometry makes the alignment problem ill-posed under rigid transformations, leading to the extreme generative-priors-to-partial-monocular-observations registration setting.
Evaluation Metrics We evaluate alignment quality using both 2D and 3D metrics. The 2D metrics include Boundary F-score, Center Drift, and IoU, which measure image-space consistency by comparing the projected silhouette of the aligned reconstruction with the target mask. Accurate contour alignment is essential for downstream robotic tasks such as grasping and interaction. The 3D metrics include Chamfer Distance, UniChamfer Distance, Fitness, Normal Consistency, and Depth Mean Absolute Error (Depth MAE), which capture bidirectional geometric error, observation-to-prediction geometric error, overlap quality, normal consistency, and residual inlier error, respectively. Scale recovery is quantified by Depth MAE, the mean absolute error between the rendered depth of the aligned mesh and the estimated depth map.
4.2 Generative-to-Partial Monocular Observational Alignment↩︎
| Method | 2D Metrics | 3D Metrics | |||||||
|---|---|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-10 | Boundary F \(\uparrow\) | Center Drift \(\downarrow\) | IoU \(\uparrow\) | Chamfer \(\downarrow\) | Fitness \(\uparrow\) | Normal Cons. \(\uparrow\) | Depth MAE \(\downarrow\) | Unichamfer \(\downarrow\) | |
| FGR | 0.2562 | 17.4885 | 0.5030 | 31.5524 | 0.3265 | 0.6638 | 34.7210 | 97.91 | |
| GICP | 0.3651 | 14.4621 | 0.5713 | 21.6842 | 0.4004 | 0.7385 | 25.5326 | 22.89 | |
| ICP P2Plane | 0.3466 | 13.7205 | 0.5725 | 22.7666 | 0.4079 | 0.7298 | 25.9668 | 26.95 | |
| ICP P2Point | 0.3477 | 10.2559 | 0.6032 | 17.4429 | 0.4420 | 0.7516 | 20.5719 | 18.67 | |
| Predator | 0.2599 | 19.3787 | 0.5009 | 28.4041 | 0.8094 | 0.7033 | 32.4227 | 15.5069 | |
| GeoTransformer | 0.2155 | 24.9377 | 0.4082 | 40.8592 | 0.6404 | 0.6496 | 41.1876 | 31.4545 | |
| RAP | 0.2013 | 24.5546 | 0.3964 | 44.0998 | 0.5284 | 0.6115 | 42.1329 | 38.1860 | |
| BUFFER-X | 0.0253 | 113.53 | 0.0683 | 432.61 | 0.0121 | 0.5469 | 361.02 | N/A | |
| Ours | 0.3926 | 9.1658 | 0.6443 | 14.8803 | 0.4777 | 0.7541 | 20.0198 | 16.02 | |
4pt
GenPMOAlign–Where2Place Tab. 1 reports quantitative results on GenPMOAlign–Where2Place with the updated protocol, including RAP pan2025rap? and BUFFER-X [45]. Importantly, all learning-based baselines are evaluated under our unified preprocessing protocol rather than their original off-the-shelf input pipelines: we apply the same mask-guided partial observation construction, pair centering, heuristic scale initialization, and normalized input generation before model-specific inference. Therefore, the reported differences mainly reflect each method’s registration behavior under a common generative-to-partial setting, instead of dataset-specific preprocessing advantages from official implementations. Our method achieves the best performance among all methods on Boundary F-score, Center Drift, IoU, Chamfer Distance, Normal Consistency, and Depth MAE, while Predator attains higher Fitness and UniChamfer under the same protocol, indicating that high overlap scores alone do not guarantee better silhouette or metric-depth alignment in this setting. Compared with rigid baselines (FGR/GICP/ICP), the Sim(3)-based formulation better handles canonical-to-metric scale mismatch and yields substantially improved silhouette and geometric alignment quality.
We also observe that several learning-based methods (e.g., Predator and GeoTransformer) can obtain relatively high Fitness, but this does not consistently translate to better global alignment quality in our setting, as reflected by weaker 2D consistency and larger Chamfer/Depth errors. UniChamfer further reveals this behavior: methods with high overlap scores may still remain less accurate on observation-supported surfaces under severe partial-to-complete mismatch. Overall, these updated results support our main claim that explicitly modeling scale and constraining registration with observation-consistent geometry provides the most reliable alignment in this challenging regime. Consistent with the quantitative trends, Fig. 3 shows that our method produces more stable pose and silhouette alignment on representative instances.
From the 3D metrics, standard rigid registration methods often fail to achieve accurate alignment, yielding higher Chamfer Distance and lower Fitness, primarily due to the scale ambiguity inherent to monocular reconstruction. Moreover, solvers restricted to \(SE(3)\) cannot bridge this scale gap and frequently converge to poor local minima, where object centers align but boundaries drift. In contrast, our method jointly estimates scale \(s\), rotation \(R\), and translation \(t\) in \(Sim(3)\), leading to lower geometric error and more accurate dense alignment. Beyond scale, partial-to-complete matching induces low or zero overlap: hallucinated regions lack correspondence in single-view observations, making methods such as FGR sensitive to non-overlapping geometry and degrading registration stability. By enforcing consistency only on observation-supported surfaces, our approach suppresses hallucinations and remains robust under low overlap, which is further reflected by improved 2D metrics, namely higher IoU and Boundary F-score. Consistent with these quantitative results, Fig. 3 provides qualitative comparisons on representative object instances. Traditional methods commonly exhibit pose drift and silhouette misalignment under this cross-domain setting, while our method more reliably recovers plausible scale and pose.
General Registration. To test whether our design generalizes beyond the target generative-to-partial setting, we evaluate it on the standard 3DMatch, 3DLoMatch, and KITTI Odometry benchmarks under the established evaluation protocols in prior work. Following common practice, we report representative baseline results together with our scores. Our method achieves 92.0 on 3DMatch and 36.0 on 3DLoMatch, remaining competitive on generic pairwise registration while being designed for the harder canonical-to-metric alignment problem with partial observations (PARENet reports higher indoor recall on both benchmarks). On KITTI, our pipeline attains 99.82% registration recall with mean RRE/RTE of 0.14\(^\circ\)/0.045 m on successful pairs, matching the strongest results reported in BUFFER-X [45].
| Metric | FPFH+FGR | FPFH+Quatro | FPFH+TEASER++ | Predator | GeoTransformer | BUFFER | PARENet | Ours |
|---|---|---|---|---|---|---|---|---|
| 3DMatch | 62.53 | 8.22 | 52.00 | 90.60 | 92.00 | 92.90 | 95.00 | 92.00 |
| 3DLoMatch | 15.42 | 1.74 | 13.25 | 62.40 | 75.00 | 71.80 | 80.50 | 36.00 |
| KITTI RR (%) | 98.74\(^\ddagger\) | 99.10\(^\ddagger\) | 98.92\(^\ddagger\) | 89.9\(^\dagger\) | 92.0\(^\dagger\) | 99.64\(^\ddagger\) | 99.82\(^\ddagger\) | 99.82 |
3pt
4.3 Real-World Robotic Evaluation↩︎
| Object | GraspGen | Ours | GICP | ZeroGrasp | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 (lr)8-10 (lr)11-13 Category | \(S\) | \(F\) | \(S\%\) | \(S\) | \(F\) | \(S\%\) | \(S\) | \(F\) | \(S\%\) | \(S\) | \(F\) | \(S\%\) |
| cup | 6 | 4 | 60 | 10 | 0 | 100 | 7 | 3 | 70 | 8 | 2 | 80 |
| spoon | 3 | 7 | 30 | 7 | 3 | 70 | 6 | 4 | 60 | 7 | 3 | 70 |
| plate | 6 | 4 | 60 | 9 | 1 | 90 | 5 | 5 | 50 | 6 | 4 | 60 |
| hammer | 3 | 7 | 30 | 8 | 2 | 80 | 5 | 5 | 50 | 8 | 2 | 80 |
| cube | 5 | 5 | 50 | 10 | 0 | 100 | 7 | 3 | 70 | 9 | 1 | 90 |
| mouse | 4 | 6 | 40 | 8 | 2 | 70 | 5 | 5 | 50 | 6 | 4 | 60 |
| Average | - | - | 45.0 | - | - | 86.7 | - | - | 58.3 | - | - | 73.3 |
3pt
We adopt a real-world robot grasping task, which is highly sensitive to metric scale, pose, and local geometry, to perform a task-driven holistic evaluation of generative-to-partial registration and the effectiveness of the proposed method. Experiments are conducted on a UR5 6-DoF manipulator with a Robotiq 2F-85 gripper and an eye-to-hand monocular RGB camera. We evaluate six everyday objects (cup, spoon, plate, hammer, Rubik’s cube, and mouse), which exhibit thin structures, symmetries, and self-occlusions. For each object, we reconstruct a sparse point cloud from a single-view observation and execute predicted grasps on the real robot to measure success rates. GraspGen [4] predicts grasps directly from the unregistered partial point cloud. For GICP [22] and our method, we first align the observation to the generative prior and then run GraspGen on the registered point cloud. ZeroGrasp [15] performs end-to-end shape completion and grasp prediction. Notably, all baselines except ours and GICP require task-specific training. Tab. 3 summarizes the grasp success rates of different methods across object categories. Our method achieves the best average success rate, outperforming both classical geometric registration and learning-based baselines. It remains reliable on thin and symmetric objects and under noisy, sparse observations, while requiring no task-specific training or fine-tuning. As shown in Fig. 4, the system can still produce executable poses and trajectories in a cluttered scene with distractors and challenging contact constraints, achieving stable and precise placement. Overall, by explicitly aligning generative 3D priors into metric space, our approach bridges generative reconstruction and physical interaction, enabling robust generalization to unseen objects and novel scenes.
4.4 Ablation Experiments↩︎
| 2D Metrics | 3D Metrics | |||||||
|---|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-9 | Boundary F \(\uparrow\) | Center Drift \(\downarrow\) | IoU \(\uparrow\) | Chamfer \(\downarrow\) | Fitness \(\uparrow\) | Normal Cons. \(\uparrow\) | Depth MAE \(\downarrow\) | |
| ✔ | 0.3583 | 10.6200 | 0.6040 | 22.7666 | 0.4407 | 0.7453 | 21.8500 | |
| ✔ | 0.3428 | 10.6500 | 0.6099 | 17.4429 | 0.3986 | 0.7352 | 25.7100 | |
| ✔ | ✔ | 0.3926 | 9.1658 | 0.6443 | 14.8803 | 0.4777 | 0.7541 | 20.0198 |
0pt 0pt
| 2D Metrics | 3D Metrics | ||||||
|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-8 | Boundary F \(\uparrow\) | Center Drift \(\downarrow\) | IoU \(\uparrow\) | Chamfer \(\downarrow\) | Fitness \(\uparrow\) | Normal Cons. \(\uparrow\) | Depth MAE \(\downarrow\) |
| Voxel Size = 50 | 0.4082 | 9.5558 | 0.6412 | 15.0004 | 0.4816 | 0.7502 | 18.4154 |
| Voxel Size = 20 | 0.3629 | 9.9808 | 0.6235 | 15.7202 | 0.4663 | 0.7239 | 21.9356 |
| Voxel Size = 30 (Ours) | 0.3926 | 9.1658 | 0.6443 | 14.8803 | 0.4777 | 0.7541 | 20.0198 |
0pt 0pt
| 2D Metrics | 3D Metrics | ||||||
|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-8 | Boundary F \(\uparrow\) | Center Drift \(\downarrow\) | IoU \(\uparrow\) | Chamfer \(\downarrow\) | Fitness \(\uparrow\) | Normal Cons. \(\uparrow\) | Depth MAE \(\downarrow\) |
| SAM-3D (default) | 0.3926 | 9.1658 | 0.6443 | 14.8803 | 0.4777 | 0.7541 | 20.0198 |
| TripoSR | 0.3518 | 8.7146 | 0.6893 | 20.0526 | 0.9029 | 0.7689 | 44.3529 |
0pt 0pt
Impact of Scale Recovery As shown in Tab. 4, when we remove explicit scale estimation and only perform ICP refinement in \(SE(3)\), the overall alignment quality degrades substantially. Specifically, 3D geometric error increases and overlap quality drops, while 2D silhouette consistency also deteriorates with larger center drift. This result indicates that omitting scale recovery leads to global geometric drift and boundary mismatch under scale inconsistency between canonical and metric spaces, confirming that explicit scale estimation is indispensable for bridging the canonical-to-metric gap and achieving stable, accurate alignment.
Role of ICP Refinement in Alignment Quality As shown in Tab. 4, when ICP refinement is removed and only the initial Sim(3) solution is retained, the alignment quality significantly degrades. Both 3D Chamfer and Fitness worsen, indicating that without ICP refinement, residual errors are not effectively suppressed, leading to a decline in local geometric alignment quality. Meanwhile, the 2D metrics also decrease, with particularly noticeable drops in silhouette consistency and IoU, demonstrating the importance of ICP refinement in improving local geometric fidelity and overall alignment accuracy.
Robustness to Voxel Downsampling Tab. 5 evaluates the impact of voxel downsampling on GenPMOAlign–Where2Place. As the voxel size varies over a wide range from 20 to 50, our method exhibits only minor fluctuations in both 2D and 3D metrics, indicating strong robustness to point-density changes. Meanwhile, the default setting with voxel size 30 achieves the best or near-best performance on most key metrics (Tab. 5), suggesting a favorable trade-off between preserving geometric detail and suppressing noise.
Local Geometry Prior (SAM-3D vs.TripoSR). Tab. 6 studies whether our alignment pipeline is sensitive to the source of canonical geometry. We swap SAM-3D with TripoSR [6] for single-view mesh reconstruction while keeping all other components unchanged. TripoSR achieves higher silhouette consistency (IoU of 0.6893) and overlap quality (Fitness of 0.9029), together with lower center drift (8.71 px). SAM-3D remains advantageous in bidirectional Chamfer distance and depth consistency (Depth MAE of 20.02 mm vs.44.35 mm), reflecting that mesh priors with stronger metric-depth alignment are still preferable when dense depth supervision is available. Overall, the results confirm that our registration core is not tied to a specific reconstruction backend and can ingest alternative single-view priors such as TripoSR, with expected trade-offs between silhouette alignment and metric depth fidelity.
Guarded Estimation Ablation (Mechanism Validation). We analyze the Sim(3) scale-control mechanism by varying guard range and scale-locking. Collapse reports the fraction of samples with \(s^{*}<0.3\) (pathological shrinkage after heuristic scaling); Clamped reports the fraction whose RANSAC scale is projected back into the guard interval. This ablation validates whether guarded estimation suppresses pathological scales, rather than replacing the main benchmark in Tab. 1.
| Setting | Guard Range on \(s^{*}\) | Scale-locked Refine | Collapse (\(s^{*}<0.3\)) \(\downarrow\) | Clamped \(\uparrow\) |
|---|---|---|---|---|
| Guarded (default) | \([0.5,1.5]\) | Yes | 0.0000 | 0.9464 |
| Guarded (tight) | \([0.8,1.2]\) | Yes | 0.0000 | 0.9732 |
| Guarded (medium) | \([0.5,1.5]\) | Yes | 0.0000 | 0.9554 |
| Guarded (loose) | \([0.2,2.0]\) | Yes | 0.9643 | 0.9375 |
| Unguarded | \((0,+\infty)\) | Yes | 0.9643 | 0.0000 |
| No-guard + no-lock | \((0,+\infty)\) | No | 0.9911 | 0.0000 |
| Guard + no-lock | \([0.5,1.5]\) | No | 0.0000 | 0.9821 |
| No-guard + lock | \((0,+\infty)\) | Yes | 0.9732 | 0.0000 |
4pt
Key Findings. (1) Guarding is critical for preventing pathological scales: collapse drops from 96.43%–99.11% (unguarded variants) to 0% (guarded variants with moderate/tight ranges). (2) Overly loose bounds behave similarly to unguarded estimation, indicating that an effective guard must be sufficiently tight. (3) Scale locking alone does not solve collapse; the dominant factor is whether global Sim(3) scale is explicitly constrained. Overall, this ablation supports the core claim that guarded estimation improves robustness by explicitly filtering implausible scale hypotheses.
5 Conclusion↩︎
In this study, we propose a novel geometric alignment framework for aligning generative 3D reconstructions with partial monocular observations. By explicitly modeling scale, using geometry-aware descriptors for robust initialization, and employing refined optimization, our method effectively mitigates scale ambiguity and geometric hallucinations, significantly enhancing registration accuracy. Through the construction of the GenPMOAlign–Where2Place benchmark and experimental validation, the results demonstrate that our approach outperforms traditional geometric methods and state-of-the-art learning-based methods under extreme conditions. This work provides a stable and reliable solution for applying generative 3D models in robotic perception, offering useful insights and tools for addressing complex 3D reconstruction and alignment problems in computer vision. Additional visualizations, theoretical guarantees, and detailed derivations are provided in the supplementary material.
Impact Statement↩︎
This research aims to advance embodied intelligence by developing a unified robotic system capable of both navigation and manipulation tasks. By demonstrating that cross-task learning can mutually enhance different embodied capabilities, our research contributes to building more efficient and versatile AI agents. This unified framework approach helps reduce the complexity and cost of deploying robotic systems, making them more applicable to real-world scenarios. More broadly, this work advances the integration of embodied intelligence capabilities into a single model, paving the way for more powerful and adaptive autonomous systems that better meet human needs in various domains.
CRediT authorship contribution statement↩︎
Author contribution details will be provided in the final manuscript. All authors have read and agreed to the published version of the manuscript.
Code and Data availability↩︎
All code, model weights, and datasets will be made available upon publication.
Conflicts of Interests↩︎
The authors declare no conflict of interest.
Generative AI Disclosure↩︎
During the preparation of this manuscript, the authors used ChatGPT (OpenAI) solely for language editing and clarity improvement of the text. All content generated with the assistance of this tool was reviewed and edited by the authors, who take full responsibility for the final manuscript. No generative AI tools were used to create, modify, or enhance any images in this work. All figures are original and were generated by the authors in accordance with Elsevier’s policy on the use of generative AI for images.