1 Introduction↩︎

Recovering the 3D pose and shape of objects from images is a fundamental challenge in computer vision, with relevant applications in robotic perception [3] and manipulation [4]. A practical pipeline typically first retrieves a CAD model that approximately matches the target object, and then estimates a 9D alignment (rotation, translation, and anisotropic scale) to register it to the image observation [1], [2]. The alignment step is typically solved by establishing correspondences between the CAD model and the observed image, followed by robust pose estimation from the resulting matches.

Early works cast this as a supervised learning problem, using end-to-end [1], [5] or render-and-compare [6], [7] training setups, which lack generalization to out-of-distribution images and require costly pose-annotated data. Advances in foundation models for visual perception have enabled zero-shot CAD alignment through pretrained backbones that produce high-dimensional features for correspondence estimation, improving generalization across categories and distributions [8], [9]. However, features from visual foundation models are primarily appearance-based and lack grounding in the 3D object geometries. This results in a domain gap, leading to suboptimal alignment accuracy.

Weak supervision via synthetic renders of CAD models has been used to align this feature space with the geometry-focused Normalized Object Coordinates (NOCs) [2], [10], yet models trained on synthetic and isolated data struggle to generalize to the clutter, occlusion, and appearance variation present in real scenes. Beyond training data, nearest-neighbor matching in high-dimensional semantic feature spaces, as used in existing works, is costly in both memory and computation and fails to exploit the rigid geometric structure of the problem. This produces noisy matches that often require expensive iterative refinement to recover accuracy.

In this paper we propose SUFLECA (Scaling Up Feature LEarning for CAD-to-image Alignment), which addresses these limitations through two key contributions. First, it scales NOC-supervised feature learning to a large mixture of real and synthetic datasets, substantially improving generalization. The resulting low-dimensional feature space reduces both matching cost and memory footprint, making it well suited for robotic applications. Second, we introduce a correspondence estimation algorithm that replaces conventional semantic-only nearest-neighbor matching with a method that enforces mutuality and geometric consistency. Together, these contributions enable zero-shot alignment accuracy that surpasses supervised methods while maintaining sub-second runtime, as shown in 1.

2 Related Work↩︎

2.1 Normalized Object Coordinates (NOCs)↩︎

NOCs [10] represent each object point in a category-level canonical coordinate frame, allowing pose estimation to be formulated as dense coordinate prediction followed by 3D registration. OmniNOCS [11] introduced large-scale NOC supervision for pose estimation, although it draws heavily from single-object datasets such as Objectron [12]. Vision backbones implicitly encode 3D-awareness [13], motivating the use of NOC supervision as an auxiliary signal on top of frozen features to shape a high-dimensional feature space for correspondence estimation [2]. However, this remains limited in coverage of occlusions and real-synthetic pairings, which are particularly important in CAD alignment, where a synthetic CAD model is aligned to a real object observation that may be affected from partial occlusion.

2.2 Supervised CAD-to-image alignment↩︎

Supervised approaches to single-image CAD alignment span three paradigms. Direct regression methods predict 9D pose in a single feed-forward pass. Total3DUnderstanding [14] jointly regresses layout, boxes, and meshes; and Mask2CAD [5] couples detection with image-shape embeddings. Render-and-compare methods refine an initial CAD pose by minimising the difference between the rendered hypothesis and the input image. 3D-RCNN [15] pioneered this idea with differentiable rendering and CAD priors, while SPARC [6] and MultiObj-SPARC [7] achieve strong results through sparse transformer-based 9D pose updates. In contrast, indirect methods first estimate correspondences and then recover pose through registration. ROCA [1] regresses NOCs from images to establish such correspondences and is trained end-to-end. More recently, CosCAD [16] combines cross-modal CAD retrieval with pose alignment by integrating image, CAD, and text features in a shared representation space, improving retrieval and alignment under occlusion. However, all of these methods rely on direct pose supervision, which limits their scalability.

2.3 Zero-shot CAD-to-image alignment↩︎

A parallel line of work aims to remove the dependence on pose-annotated training data. DiffCAD [17] trains cascaded diffusion models on synthetic data for scale, pose, and shape estimation, while SDFit [9] fits a morphable SDF through iterative render-and-compare optimization. However, both methods require per-category priors and are computationally expensive. Related category-level approaches [3], [18] aim to estimate optimal object pose and shape by progressively pruning outliers to recover the largest set of compatible measurements. Our correspondence filtering is inspired by this line of work, but these methods still rely on active shape models and keypoint detectors, which are difficult to obtain for most categories. Among other works, Diorama [8] uses an additional off-the-shelf model for scale inference and targets open-world scene-level layout, but its accuracy degrades in real-world scenes with occlusions. Recently, ZeroCAD [2] trains a NOC-supervised adapter over a frozen visual backbone for zero-shot 9D alignment. However, training exclusively on synthetic single-object scenes across nine categories leaves a substantial domain gap, and its simple nearest-neighbor matching admits many-to-one geometrically inconsistent correspondences.

3 SUFLECA↩︎

3.1 Problem Definition↩︎

Given an RGB image, camera intrinsics, and a candidate CAD model retrieved from off-the-shelf methods, the goal is to estimate a 9D transformation from the CAD coordinate frame \(\mathcal{X}_{\mathrm{cad}}\) to the camera coordinate frame \(\mathcal{X}_{\mathrm{cam}}\), as follows: \[\mathcal{X}_{\mathrm{cam}} = R S \mathcal{X}_{\mathrm{cad}} + t, \label{eq:problem}\tag{1}\] where \(R \in SO(3)\), \(S = \mathrm{diag}(s_x, s_y, s_z)\) is an axis-aligned anisotropic scale matrix, and \(t \in \mathbb{R}^3\). We extend this standard formulation and also output a score, \(\mathcal{S}_\mathrm{fit} \in \mathbb{R}\), that measures the quality of the estimated alignment, which is useful for ranking alignments or multi-view aggregation.

3.2 Overall CAD Fitting Pipeline↩︎

SUFLECA follows the standard indirect CAD alignment pipeline used in prior work [1], [2], [8], [9].

Given the input image, the pipeline first extracts an object mask and computes per-pixel descriptors. The same descriptors are also extracted for the candidate CAD model. Image-to-CAD correspondences are then established in feature space, lifted to 3D using monocular depth estimates, and passed to a RANSAC-based Procrustes solver to estimate the alignment in 1 . SUFLECA improves two key modules of this pipeline: the feature model, which is learned at scale and unified across real and synthetic data as described in 3.3, and the correspondence estimation module, which adds geometric consistency as described in 3.4. Throughout this paper, zero-shot refers specifically to the alignment stage: object detection and segmentation, depth estimation, and CAD retrieval are treated as upstream components inherited from prior work.

3.3 Scaling Unified NOC-aligned Feature Learning↩︎

Multi-Dataset Setup  We adopt a scaling paradigm similar to [11], focused primarily on the indoor domain and augmented with synthetic data. Since correspondence for CAD alignment occurs between a real image and a synthetic CAD render, we augment, wherever CAD annotations are available, each real image with a synthetic counterpart in which the CAD objects are rendered at approximately the same pose. The synthetic counterparts are rendered with randomized backgrounds and object textures, with some objects occasionally replaced by visually similar ones. This yields a larger and more balanced training set with comparable numbers of real and synthetic samples, sharing similar occlusion patterns and scene context, which encourages the model to learn domain-invariant features. Compared with  [2], which trains on 300K synthetic images of single objects across 9 categories, our dataset comprises 674K images from 12 datasets (see 1) with an average of 3.38 annotated objects per frame across a broader range of categories, providing denser and more diverse supervision. Furthermore, while most datasets we train on originate from SLAM-focused camera trajectories, the inclusion of casual imagery [20], [26], [27] further broadens the data distribution.

NOC Derivation and Verification  Most datasets do not provide NOC maps directly, but instead supply object poses in 3D from which NOCs must be derived. To this end, we render the ground-truth object onto the image to obtain an approximate bounding box, from which a mask is extracted using SAM2 [31]. The masked region is then backprojected into 3D using available depth maps or a monocular metric depth estimator [32], after which NOCs are computed following [11] (see the middle row in 2). To ensure annotation quality, we verify the computed NOCs against the overlaid CAD NOCs and reject object instances whose mean error exceeds a threshold. Frames for which the total number of valid NOC pixels falls below a threshold are discarded entirely. For datasets derived from high frame-rate cameras [19], [22], [25], we subsample frames at regular intervals and additionally enforce a minimum camera pose displacement to ensure sufficient viewpoint diversity.

Model Architecture  As shown in 3, our model builds upon a pretrained perception encoder, kept frozen during training, that provides rich patch-level features from images. These patch features are upsampled using a Dense Prediction Transformer (DPT) [33], which fuses multi-scale outputs from different layers of the encoder to produce denser feature maps of dimension \(\mathrm{dim}\). The DPT features are decoded by a lightweight NOC head: a binned classifier [10], [11] that partitions each coordinate axis (\(x\), \(y\), \(z\)) of the NOC space into \(m\) uniform bins, estimating value as the expectation over the bin probabilities. This supervision encourages the DPT features to be spatially smooth and geometrically meaningful. The training loss comprises a categorical cross-entropy term between the predicted bin probabilities and the ground-truth bin, and an \(\ell_1\) regression term on the expected NOC, \[\mathcal{L}_{\mathrm{NOC}} = \mathcal{L}_{\mathrm{CE}} + \lambda \| \hat{\mathcal{N}} - \mathcal{N} \|_1,\] where \(\mathcal{L}_{\mathrm{CE}}\) is the cross-entropy loss, \(\hat{\mathcal{N}}\) is the predicted NOC value, \(\mathcal{N}\) is the ground-truth NOC value, and \(\lambda\) is a weighting coefficient. The NOC head is discarded at test time, and the \(\ell_2\)-normalized DPT features are used as descriptors for correspondence estimation in SUFLECA. Optionally, these features can be concatenated with the output layer features of the vision backbone, following [2], to mitigate forgetting caused by weak supervision over a limited set of categories. We refer to this variant as SUFLECA-blend.

3.4 Geometrically-consistent Correspondence Estimation↩︎

To align a CAD model \(C\) to a masked image \(I\), we start by featurizing the image using the aforementioned feature model \(\mathcal{M}\), yielding per-patch features interpolated to pixel level, \(\mathcal{F}_I = \mathcal{M}(I)\). Unlike prior works that aggregate 2D features across multiple renders of \(C\) to obtain 3D features over a mesh [2], [9], a process that is both arbitrary and lossy, we keep the matching task entirely within the 2D domain. To this end, we maintain a small set of \(n\) pre-rendered views of \(C\) with precomputed features and point maps. Perception encoders have been shown to encode global pose information in an image [8], which we exploit to select the closest render to the query image prior to estimating correspondences. For that, we start by using a monocular metric depth estimator [32] to lift the masked image pixels into a 3D point cloud \(\mathcal{Q}_i\), and apply Farthest Point Sampling (FPS) to obtain a representative subset \(\hat{\mathcal{Q}}_i \subset \mathcal{Q}_i\) of \(N\) points for correspondence estimation. Then, given the set of pre-rendered views \(\mathcal{V}_C\) from distinct fixed viewpoints, we select the view \(V^*\) that maximises the mean nearest-neighbor cosine similarity over \(\hat{\mathcal{Q}}_i\), \[V^* = \mathop{\mathrm{argmax}}_{V \in \mathcal{V}_C} \frac{1}{|\hat{\mathcal{Q}}_i|} \sum_{q \in \hat{\mathcal{Q}}_i} \max_{p \in V} \mathcal{F}_I(q)^\top \mathcal{F}_V(p), \label{eq:render}\tag{2}\] where \(\mathcal{F}_V(p)\) is the precomputed feature at pixel \(p\) of view \(V\) and \(\mathcal{F}_I(q)\) is the feature at pixel \(q\) in \(I\).

Using the selected render feature map \(\mathcal{F}_{V^*}\) and image feature map \(\mathcal{F}_I\), we need to estimate correspondences between the lifted render and observed object pixels. While nearest-neighbor matching suffices for the aforementioned coarse view selection, it is unreliable for correspondence estimation, as it produces geometrically inconsistent and many-to-one matches. Mutual nearest-neighbor matching reduces these ambiguities but is overly conservative. We therefore adopt mutual \(k\)-nearest-neighbors matching, retaining a correspondence only when the render and image descriptors lie within each other’s top-\(k\) neighbors, with duplicates resolved by greedy assignment to the highest-scoring source match. However, this only considers the semantic feature space and does not enforce geometric consistency; and since the transformation includes anisotropic scaling, pairwise distance preservation is not directly applicable. Therefore, we first estimate an approximate scale to normalize the correspondence set before applying pairwise distance preservation. Concretely, let \(\{(\mathbf{p}_i, \mathbf{q}_i)\}_{i=1}^{M}\) be a set of putative correspondences, where \(\mathbf{p}_i \in \mathbb{R}^3\) are source points from \(V^*\) and \(\mathbf{q}_i \in \mathbb{R}^3\) are target points from \(I\). We seek a boolean inlier mask \(\mathbf{m} \in \{0,1\}^M\) that retains correspondences geometrically consistent with the unknown 9D transformation. Denoting by \(\mathbf{s}^2 = (s_x^2, s_y^2, s_z^2)^\top\) the squared anisotropic scale vector, we obtain the following from pairwise differences, \[\begin{align} \|\mathbf{q}_i - \mathbf{q}_j\|^2 = &\,\boldsymbol{\Delta}_{ij}^\top \mathrm{diag}(\mathbf{s}^2)\, \boldsymbol{\Delta}_{ij} = (\mathbf{s}^2)^\top \left(\boldsymbol{\Delta}_{ij} \odot \boldsymbol{\Delta}_{ij}\right)\\ \qquad \boldsymbol{\Delta}_{ij} = &\,\mathbf{p}_i - \mathbf{p}_j, \end{align}\] which yields a linear system in \(\mathbf{s}^2\), \[\phi_{ij} = \mathbf{f}_{ij}^\top \mathbf{s}^2, \;\; \mathbf{f}_{ij} = \frac{\boldsymbol{\Delta}_{ij} \odot \boldsymbol{\Delta}_{ij}}{\|\boldsymbol{\Delta}_{ij}\|^2}, \;\; \phi_{ij} = \frac{\|\mathbf{q}_i - \mathbf{q}_j\|^2}{\|\boldsymbol{\Delta}_{ij}\|^2}.\] Solving this system in the presence of outliers requires robust initialisation. To this end, we obtain an isotropic scale estimate by computing the per-pair isotropic log-scale \(\psi_{ij} = \frac{1}{2}\log \phi_{ij}\) over all \(\binom{M}{2}\) pairs and taking the histogram mode, assuming that inlier pairs coarsely concentrate in a common bin. This gives the isotropic log-scale \(\hat{\ell}_\mathrm{iso} = \mathrm{mode}\{\psi_{ij}\}\), from which we initialise \(\hat{\mathbf{s}}^2 = \exp(2\hat{\ell}_\mathrm{iso}) \cdot \mathbf{1}\). From this, we refine the anisotropic log-scales \(\boldsymbol{\hat{\ell}}\) via Iteratively Reweighted Least Squares (IRLS). Since this estimate degrades when correspondences do not cover all axes sufficiently, we shrink towards the more robust isotropic estimate, leading to \[\tilde{\boldsymbol{\ell}} = \alpha\boldsymbol{\hat{\ell}} + (1-\alpha)\hat{\ell}_\mathrm{iso}\mathbf{1}, \qquad \tilde{\mathbf{s}}^2 = \exp(2\tilde{\boldsymbol{\ell}}),\] where \(\alpha \in [0,1]\) controls the trust placed in the anisotropic estimate. We then build a consensus matrix \(\mathbf{G} \in \{0,1\}^{M\times M}\), whose entries compare the observed squared distance against the value predicted by \(\tilde{\mathbf{s}}^2\), \[\begin{align} G_{ij} = &\,\mathbf{1}\!\left[\,\left|\frac{\|\mathbf{q}_i - \mathbf{q}_j\|^2 - \tilde{d}_{ij}^2}{\tilde{d}_{ij}^2}\right| < \beta\,\right], \;\; G_{ii} = 0,\\ \tilde{d}_{ij}^2 = &\,(\tilde{\mathbf{s}}^2)^\top \left(\boldsymbol{\Delta}_{ij} \odot \boldsymbol{\Delta}_{ij}\right), \end{align}\] where \(\beta>0\) is the relative inlier threshold. The principal eigenvector \(\mathbf{e}\) of \(\mathbf{G}\) scores each correspondence by its consistency with the inlier group, following which correspondences are retained by, \[\mathbf{m} = \mathbf{1}[\mathbf{e} \geq \delta \cdot \max(\mathbf{e})],\] where \(\delta \in (0,1]\) is the relative gating threshold. The \(\mathbf{m}\)-masked set with cardinality \(\hat{M}\) is passed to a space-partitioning RANSAC [34] that uses Procrustes registration under anisotropic scaling to recover the final alignment.

Alignment quality  Unlike prior works [1], [2], which use the CAD retrieval score as a proxy for alignment quality, SUFLECA estimates \(\mathcal{S}_\mathrm{fit}\) from the registration residuals, providing a more principled measure of alignment quality. The information matrix \(\mathbf{\Lambda} \in \mathbb{R}^{9 \times 9}\) is computed analytically from the Jacobians of the registration residuals with respect to the parameter vector \(\mathbf{x} = [\delta\boldsymbol{\theta}, \delta\mathbf{t}, \delta\mathbf{s}]\), where \(\delta\boldsymbol{\theta}\) is a left rotation perturbation, \(\delta\mathbf{t}\) is an additive translation, and \(\delta\mathbf{s}\) is an additive perturbation on the diagonal scale. The residual for each correspondence \(i\) is \(\mathbf{r}_i = \mathbf{q}_i - RS\mathbf{p}_i - t\), with per-correspondence Jacobian, \[\mathbf{J}_i = \begin{bmatrix} [\mathbf{u}_i]_\times & -\mathbf{I}_3 & -R\,\mathrm{diag}(\mathbf{p}_i) \end{bmatrix} \in \mathbb{R}^{3 \times 9},\] where \(\mathbf{u}_i = RS\mathbf{p}_i\) is the rotated and scaled source point and \([\cdot]_\times\) denotes the skew-symmetric cross-product matrix. The information matrix is then \[\mathbf{\Lambda} = \frac{1}{\sigma^2}\sum_i \mathbf{J}_i^\top \mathbf{J}_i, \quad \sigma^2 = \frac{\sum_i \|\mathbf{r}_i\|^2}{3\hat{M} - 9}, \label{eq:logdet}\tag{3}\] from which the alignment quality score is \(\mathcal{S}_\mathrm{fit}=\log\det(\mathbf{\Lambda})\).

4 Evaluation↩︎

4.1 Implementation Details↩︎

Our feature model uses DUNE-B [13] as the frozen perception encoder and produces DPT patch features with \(\mathrm{dim}=384\). These are fed to the NOC head, which consists of two shallow convolutional layers and classifies each coordinate into \(m=64\) bins. We train for \(50\) epochs using AdamW with \(\lambda=0.33\) and a learning rate of \(1.1 \times 10^{-4}\). We also train a smaller variant, SUFLECA-S, which uses DUNE-S [13] as the encoder with \(\mathrm{dim}=256\) and \(m=50\). For alignment, we maintain \(n = 6\) pre-rendered fixed views per CAD model, and set \(N = 512\), \(k=13\), \(M = 256\), \(\alpha=0.5\), \(\beta=0.01\), and \(\delta=0.05\). All hyperparameters are empirically determined and remain consistent across all experiments, conducted on a workstation with an NVIDIA RTX 5090 GPU.

4.2 Methodology↩︎

We evaluate SUFLECA across complementary settings. ScanNet25k [1] serves as the main benchmark for alignment accuracy, while CO3D [38] is adapted to assess generalization to categories not observed during weak supervision (4.4). We further include ablation studies to isolate the contributions of our design choices (4.5), analyze robustness to inexactly retrieved CAD models (4.6), and perform an efficiency analysis (4.7). While our comparisons primarily focus on zero-shot methods, we also include results from supervised methods on ScanNet25k for reference. We also report results with the smaller SUFLECA-S variant (from 4.1) and the SUFLECA-blend variant (from 3.3) of our method.

4.3 Datasets↩︎

ScanNet25k [1] are a dataset of images derived from the ScanNet [19] dataset with Scan2CAD annotations [39]. An alignment on this dataset is considered correct when translation, rotation, and scale errors are below \(20\,\mathrm{cm}\), \(20^\circ\), and \(20\%\), respectively. We report both category- and instance-averaged accuracy over the nine categories on the validation split. We follow the NMS-based evaluation protocol [35], ranking detections by the product of the ROCA semantic retrieval score [1] and \(\mathcal{S}_\mathrm{fit}\) defined in 3 . To ensure zero-shot evaluation, we exclude all ScanNet-derived images during weak supervision for this dataset. Following [2], all methods use ROCA bounding boxes and CAD retrievals [1], SAM2 masks [31], and depth from a monocular metric depth estimator [32] fine-tuned on the ScanNet25k training split.

The DiffCAD split [17] covers six of the ScanNet25k object categories, evaluated without NMS and using non-fine-tuned depth. This split was introduced by DiffCAD [17], whose priors are available only for six categories.

CO3D [38] is an object-centric dataset that we adapt for the inexact CAD fitting setting. We convert its COLMAP point clouds to metric scale by selecting the depth scale with the lowest error against monocular depth estimates. We also simulate occlusions by placing a rectangular occluder within the GroundedSAM [40] bounding box. For CAD retrieval, we use SAM3D [41] to generate category-level CAD pools and retrieve candidates with OSCAR [42]. We restrict the categories to two domains: seen categories (chair, couch), which appear during weak supervision of SUFLECA, and unseen categories (toaster, hairdryer, microwave, suitcase), which are used to evaluate generalization performance. We sample 100 frames per category across 208 objects. Since CO3D lacks ground-truth poses and SAM3D meshes have inconsistent canonical orientations, we report 3D-IoU, ICP-Rot, ADD-S, and ADD-S@0.1 [41] for comparison.

4.4 Results↩︎

2 reports alignment accuracy on ScanNet25k and the respective DiffCAD split. Among zero-shot methods, SUFLECA achieves \(33.4\%\) category-averaged and \(42.3\%\) instance-averaged accuracy using the NMS protocol, surpassing ZeroCAD [2] by \(10.3\) and \(12.2\) percentage points respectively, and outperforming FoundationPose [37] by a larger margin. This can be attributed to two factors: broader and more diverse NOC-supervised training data, which reduces the domain gap to real images, and our correspondence estimation, which produces geometrically consistent matches. Notably, SUFLECA also surpasses fully supervised methods, achieving the best or second-best accuracy across seven of the nine categories and outperforming MultiObj-SPARC [7] despite the latter being trained with 9D pose supervision on ScanNet. On the DiffCAD split, competing zero-shot methods perform poorly because alignment errors are not suppressed through NMS. In contrast, SUFLECA more than doubles the performance of ZeroCAD [2], achieving gains of \(20.2\) and \(23.9\) percentage points in category- and instance-averaged accuracy, respectively. Moreover, SUFLECA even outperforms the oracle variant of DiffCAD [17], which uses ground-truth pose to select the best result from eight hypotheses. Notably, the other SUFLECA variants also achieve competitive performance, outperforming competing zero-shot methods by a large margin and remaining comparable to supervised methods. SUFLECA-S attains lower overall accuracy than the base model, but still substantially outperforms other zero-shot methods, highlighting the effectiveness of its compact feature space. SUFLECA-blend performs on par with, or slightly below, SUFLECA, since the added DUNE-B features primarily improve generalization, which is less critical for the common object categories in ScanNet25k.

On CO3D, results are obtained under occluded masks and inexact retrieved CADs, posing significant front–back and left–right geometric ambiguities, as shown in 4. SUFLECA grounds objects in 3D substantially better than competing methods in this challenging setting, with its features more effectively able to disambiguate such cases. In seen categories, all variants of SUFLECA perform comparably and outperform competing methods, highlighting the effectiveness of our weak supervision. For unseen categories, SUFLECA-blend generally outperforms other methods, as it compensates for the limited supervision coverage for uncommon objects at the cost of a larger feature representation.

4.5 Ablation Study↩︎

We ablate the main design choices, and report alignment accuracy on ScanNet25k (NMS protocol) in 4.

Training data  This ablation evaluates the impact of including in-distribution training data. Training on the complete collected dataset, including ScanNet-derived images, improves performance from \(33.4/42.3\) to \(34.4/42.8\). This modest gain of only \(1.0/0.5\) percentage points suggests that SUFLECA benefits from additional data, but it already learns strong features without in-distribution images.

Correspondence estimation  We also ablate the correspondence estimation strategy. Using nearest-neighbor matching alone yields the largest drop, reducing accuracy to \(26.3/34.2\). Adding mutual consistency improves over nearest-neighbor matching to \(30.7/39.4\), but still remains below the full model. Using mutual \(k\)-nearest-neighbors matching further improves performance to \(32.0/41.0\), but still underperforms the full pipeline that applies geometric consistency. These results show that both mutual matching and geometric consensus are important for accurate alignment.

NMS scoring  We further evaluate the score used to rank alignments during NMS. Replacing our combined score based on \(\mathcal{S}_\mathrm{fit}\) with the ROCA retrieval score reduces accuracy from \(33.4/42.3\) to \(30.4/39.3\). This drop of \(3.0/3.0\) percentage points shows that semantic retrieval confidence alone is not a reliable proxy for alignment quality. We also note that the render-selection score from 2 performs even worse, reducing accuracy to \(29.2/38.8\), since it is coarse and hence sensitive to the choice of render viewpoints.

4.6 Robustness to Inexact CADs↩︎

We further analyze how alignment accuracy depends on the quality of the retrieved CAD model. As shown in  5, replacing ground-truth Scan2CAD annotations [39] with zero-shot retrieval using GroundedSAM [40] w/ OSCAR [42] leads to a gap of more than \(20\) percentage points in both category- and instance-averaged alignment accuracy. This is primarily due to the low instance retrieval accuracy of \(3.8\%\) (most retrieved CADs are inexact), resulting from heavy occlusion, appearance differences, and annotation bias. This indicates that zero-shot CAD retrieval from real-world observations remains an open problem and a key bottleneck for a fully zero-shot CAD fitting pipeline. Nevertheless, as we move from the ground-truth oracle to supervised ROCA [1] and then to fully zero-shot retrieval, the drop in instance retrieval accuracy is much steeper than the corresponding drop in alignment accuracy, suggesting that SUFLECA is moderately robust to CAD inexactness.

4.7 Efficiency Analysis↩︎

Setup  We compare the CAD alignment runtime and peak VRAM usage of different zero-shot methods, with the results reported in 6. We exclude detection, segmentation, depth estimation, and retrieval, as these stages are consistent across all methods. Moreover, since ZeroCAD [2] is not open-source, we implement a runtime-only proxy, termed as ZeroCAD*, to estimate a lower bound on its true runtime.

Discussion  SUFLECA achieves lower runtime and VRAM usage than competing methods owing to its compact feature representation (2.0\(\times\) to 5.3\(\times\) smaller than those of competing methods), and the absence of iterative pose refinement stages used in [2], [37]. The smaller SUFLECA-S further improves runtime and VRAM usage without a significant sacrifice in performance, while SUFLECA-blend improves generalization at the cost of increased runtime and VRAM usage. Notably, the majority of the runtime in SUFLECA is spent in RANSAC, enabling a straightforward accuracy-speed trade-off by adjusting the maximum iteration count.

5 Conclusion↩︎

We introduced SUFLECA, a weakly-supervised zero-shot method for 9D CAD alignment from images. Its two core contributions, scaled NOC-supervised feature learning across diverse real and synthetic data and a mutual \(k\)-nearest-neighbors matching algorithm with geometric consensus filtering, jointly address the domain gap and spatial inconsistencies that limit prior approaches. SUFLECA uses compact, geometrically discriminative features to enable sub-second CAD alignment without pose-annotated training data or iterative refinement, achieving state-of-the-art performance on ScanNet25K and, for the first time, surpassing supervised methods on this benchmark. Despite these advances, important limitations remain. First, alignment quality is still limited by CAD retrieval accuracy, making robust open-set retrieval from occluded real-world observations an important direction for future work. Second, SUFLECA is currently restricted to indoor scenes and common object categories, motivating the inclusion of outdoor scenes and less common categories to improve scalability and generalization.

References↩︎