1 Introduction↩︎

Shape abstraction, the representation of complex 3D geometry through a compact set of simple primitives, underpins applications from robotic grasping and collision detection to semantic scene understanding and shape editing. Good abstraction requires two distinct capabilities: semantic understanding of how an object decomposes into meaningful parts, and geometric fitting of a primitive to each part.

Two families of methods dominate the field of primitive shape abstraction. Learning-based approaches [1]–[4] train networks to predict primitive parameters and achieve strong results, but only within their training distribution and typically only for canonically oriented objects. Optimization-based methods [5]–[8] are category-agnostic but rely on purely geometric criteria to simultaneously segment and fit, producing decompositions that lack semantic coherence: a chair leg may be split arbitrarily rather than recognized as a single part (2). Existing methods thus either couple the semantic and geometric tasks or address only one.

Recently, generative image models trained at scale have emerged as generalist visual learners [9]: beyond synthesizing images, they carry broad visual knowledge that lets them identify and segment object parts directly in the image domain, across arbitrary categories and without task-specific training. This raises the question: can this pretrained 2D part understanding be harnessed for 3D shape abstraction, without any 3D supervision or retraining?

We answer affirmatively with a training-free harness (see 1). Given multi-view renders of any 3D object, we first prompt a vision-language model to analyze the object and identify its semantic parts; we then prompt a generative image model to paint a color-coded segmentation mask based on that analysis. We reproject these masks onto the geometry and fit each resulting part with a superquadric primitive via parameter optimization. The semantic decomposition is supplied entirely by the generative model, while the geometric fitting is handled by a classical optimizer. The result is category-agnostic, orientation-invariant, and contains no learned parameters.

Because the pipeline contains no learned parameters, it differs from prior work in kind: rather than training a specialized 3D model, it harnesses general-purpose 2D generative understanding for 3D primitive abstraction. While the individual ingredients are simple, making them work together is non-trivial: the system must obtain a globally consistent semantic part decomposition across views, preserve that decomposition during reprojection, and fit compact primitives without reintroducing category-specific training. This framing suggests a new paradigm for primitive shape abstraction, in which progress comes not from retraining a specialized 3D model, but from coupling improving foundation models with classical geometric optimization.

We validate this idea on HumanPrim and Toys4K, where the resulting method achieves the lowest Chamfer distance among all evaluated methods while using only 5–9 primitives per object on average. A ground-truth segmentation ablation (4.3) further shows that the fitter is not the current accuracy bottleneck: replacing generated masks with human-annotated part labels substantially improves results, confirming that abstraction quality scales with the generative model and can inherit future improvements without retraining.

2 Related Work↩︎

Vision-Language Models such as GPT-4V [10] and Gemini [11] have demonstrated broad visual understanding, including spatial reasoning about object structure. More recently, Gabeur et al. [9] showed that image generators trained at scale serve as strong generalist visual learners, with broad visual competencies, including part-level segmentation, that can be elicited purely through prompting. These capabilities sit at the intersection of two research threads we survey below: methods that lift 2D foundation-model predictions to 3D (2.1), and methods that abstract 3D geometry into compact primitive decompositions (Sections 2.2–2.3). Within the first thread, the key distinction is between discriminative feature-based approaches and our generative, output-based approach; within the second, the divide between learning-based and optimization-based methods motivates our training-free design.

2.1 Lifting 2D Segmentation to 3D↩︎

Foundation segmentation models such as SAM [12] enabled class-agnostic 2D segmentation, and a broad family of methods lifts 2D foundation-model predictions to 3D by reprojection or feature aggregation. PartSLIP [13], MeshSegmenter [14], and Qi et al.’s study of GPT-4V for zero-shot point clouds [15] all render or project the 3D input to 2D, apply a vision foundation model, and backproject the resulting labels or features. Our pipeline follows the same render–apply–backproject skeleton, but differs in what the 2D model is and what it returns: rather than reading features out of a discriminative backbone, we prompt a generative model to paint a part mask, and rather than stopping at backprojected labels we fit a primitive to each part. SAMPart3D [16] and COPS [17] remove the need for part-label supervision. You et al. [18] combine VLM-based component identification with a geometric parameter estimator to reverse-engineer CAD models from images. In contrast, native 3D segmenters such as PartField [19], P3-SAM [20], and Point-SAM [21] operate directly on point clouds or voxels, avoiding the render-and-reproject round trip; Point-SAM in particular offers a promptable interface that is a category-agnostic alternative to our generative decomposition stage. Most recently, SegviGen [22] steers a native 3D generative model to produce part segmentations by injecting 2D color guidance into its denoising process. Across all these methods, 3D part understanding is obtained either by reprojecting discriminative model predictions or by training native 3D networks. Our approach differs in kind: we prompt a generative image model to paint part masks directly, requiring no task-specific training, no 3D prior, and no learned lifting module.

2.2 Learning-Based Shape Abstraction↩︎

The earliest methods rely on annotated training data: Tulsiani et al. [1] pioneered learning-based abstraction by training a network to predict cuboid parameters, and Zou et al. [23] extended this with recurrent networks to generate primitive sequences. Without primitive-level supervision, Sun et al. [24] learn hierarchical cuboid abstractions in a self-supervised manner, Paschalidou et al. [2], [25] learn superquadric and deformable part decompositions, and Yang and Chen [26] jointly perform segmentation and cuboid fitting in an unsupervised setting. Zhao et al. [27] generalize the primitive vocabulary to sweep surfaces via unsupervised neural sweeping, Fedele et al. [3] proposed SuperDec, a self-supervised transformer that decomposes point clouds into superquadric primitives, and Kobsik et al. [28] introduced a self-supervised fine-to-coarse refinement strategy. Abstraction has also been reframed as a sequence generation task: Li et al. [29] proposed sequence-to-sequence cuboid generation, Ye et al. [4] trained an auto-regressive transformer on human-crafted primitive assemblies, and Tian et al. [30] fine-tune an LLM to predict mixed primitive parameters directly from point cloud input. These methods achieve strong within-distribution results but share two structural limitations: they require training data from specific shape categories, and they typically assume a canonical object orientation. Neither constraint holds for in-the-wild 3D data, and neither applies to our training-free harness.

2.3 Superquadric Recovery and Shape Primitives↩︎

Superquadrics were introduced as a flexible shape family by Barr [31] and first applied to parts-based object recognition by Pentland [32]; a small set of parameters interpolates between cuboids, ellipsoids, cylinders, and octahedra. Solina and Bajcsy [33] extended the family with global deformations such as tapering and bending and proposed a foundational recovery method from range images. Leonardis et al. [34] extended this to segmentation and multi-part recovery, and Jaklič et al. [35] provided a comprehensive treatment; we summarize the formalism as adapted in our methodology in the appendix. More recently, Liu et al. [5] introduced EMS, modeling superquadric fitting as Expectation-Maximization for multi-primitive decomposition, and Liu et al. [6] proposed Marching Primitives for fitting from signed distance functions. Monnier et al. [36] extend this line of work to multi-view images, fitting textured superquadrics via differentiable rendering without requiring a point cloud. Alaniz et al. [37] similarly fit superquadrics from multiple views, iteratively recomposing primitives via silhouette matching. Gao et al. [38] combine superquadrics with 2D Gaussian splatting for self-supervised part decomposition from multi-view images. Wang et al. [7] address structure-aware abstraction for generative meshes: Light-SQ partitions a watertight input mesh via SDF carving and structural decomposition, then fits superquadrics to each region. The partitioning is purely geometric, driven by volumetric structure rather than semantic labels, and the method averages around 60 primitives per object, a regime comparable to PrimAny [4] rather than to our 5–9 semantically grounded parts. Pursuing similar parsimony through a different route, Ganeshan et al. [8] introduce the SuperFrustum, a unified 8-parameter differentiable primitive combined with iterative residual fitting. These methods can decompose a single object geometrically, but lack the cross-instance semantic knowledge needed to identify parts that are consistent across a class. Our work unifies both capabilities: a generative image model supplies the semantic decomposition into meaningful parts, and superquadric fitting gives each part a compact, parameterized geometric representation.

3 Methodology↩︎

Our pipeline consists of three stages: (1) multi-view rendering and generative part segmentation, (2) reprojection and color-restricted clustering, and (3) superquadric primitive fitting via parallel multi-start optimization.

3.1 Multi-View Capture and Generative Segmentation↩︎

Given a 3D object, we render it from four opposing viewpoints and concatenate the views into a single image. Obtaining a consistent segmentation across all four views raises several challenges: the assigned colors must match across views, the number and granularity of parts must be agreed upon globally, and semantic assignments must be coherent (e.g., all four legs of a chair should map to the same part type). We address these through a two-stage pipeline that separates semantic analysis from mask generation.

The first call performs analysis: the model receives the multi-view image and a structured prompt asking it to identify the object, decompose it into semantic parts (e.g., legs, seat, back), and assign a distinct color to each. This establishes a consistent, object-level description before any pixel-level generation occurs; its output is a structured JSON mapping part names to colors that anchors the second stage. We assign one color per semantic part type rather than per instance (left and right legs share a color) for two reasons: current image generative models cannot reliably distinguish left from right across opposing views, and per-view instance identities tend to be inconsistent across viewpoints, so instance-colored masks cannot be reprojected into a coherent 3D labeling (8).

The second call performs mask generation: a generative image model receives the same multi-view image together with the class–color mapping established in the first stage and paints a color-coded segmentation mask. Because the part-color scheme is fixed upfront, the generator need only follow the established mapping, making cross-view color consistency achievable. This is the step that requires a capable image generator (7). Separating analysis from generation also gives explicit control over each step: the analysis answer is inspectable and can be verified before the more expensive generation proceeds.

3.2 Reprojection and Color-Restricted Clustering↩︎

The segmentation mask is reprojected onto the 3D geometry via a per-pixel voting mechanism that aggregates labels from all four views, making the labeling robust to imperfect coverage in any single view, and produces a colored point cloud labeled by semantic part. The resulting labels contain noise from imperfect generated masks and color bleeding at part boundaries.

To extract clean per-part point clouds, we apply color-restricted spatial clustering: points are grouped by quantized color, background points are discarded, and within each color group, spatially connected components are extracted via flood-fill with a configurable radius \(r\) using a KD-tree. Clusters below a minimum size are discarded as outliers, since small disconnected fragments typically arise from mask noise or color bleed rather than genuine object parts.

3.3 Superquadric Primitive Fitting↩︎

Given the pre-segmented part point clouds from the previous stage, this step is a straightforward parameter optimization: one superquadric is fitted independently to each point cloud without any further subdivision or segmentation of the points. Superquadric fitting is, however, a non-convex optimization problem: the loss landscape contains many local minima, and a single initialization frequently converges to a non-optimal solution. We address this through a parallel multi-start strategy.

Each point cloud segment is first normalized to a unit cube, then fitted with a superquadric parameterized by up to 15 values: size \((a_1, a_2, a_3)\), position \((p_x, p_y, p_z)\), rotation \((\alpha, \beta, \gamma)\) as Euler angles, shape \((\epsilon_1, \epsilon_2)\), and optional tapering \((k_x, k_y)\) and bending \((k_b, \alpha_b)\) deformations. We initialize from an approximate minimum-volume bounding box (ApproxMVBB, [39]) and generate nine candidate configurations by combining three primitive types (cuboid: \(\epsilon_1, \epsilon_2 \to 0.1\); ellipsoid: \(\epsilon_1, \epsilon_2 = 1.0\); cylinder: \(\epsilon_1 = 0.1, \epsilon_2 = 1.0\)) with three axis-aligned orientations obtained by rotating the bounding box axes. The orientation variation is necessary because deformations act along the principal axis.

All candidates are optimized in parallel using L-BFGS [40], minimizing the bidirectional Chamfer distance between the point cloud \(\mathcal{P}\) and points sampled from the superquadric surface \(\mathcal{S}\): \[\mathcal{L} = \frac{1}{|\mathcal{P}|} \sum_{\mathbf{p} \in \mathcal{P}} \min_{\mathbf{s} \in \mathcal{S}} \|\mathbf{p} - \mathbf{s}\|^2 + \lambda \sum_{\mathbf{s} \in \mathcal{S}} w_\mathbf{s} \min_{\mathbf{p} \in \mathcal{P}} \|\mathbf{s} - \mathbf{p}\|^2 \label{eq:chamfer}\tag{1}\] The forward term measures coverage of the target shape; the backward term, weighted by \(\lambda\), penalizes the primitive from extending beyond the target. Surface samples are weighted by their local area element \(w_\mathbf{s}\) to correct for the non-uniform parametric sampling density of superquadrics.

Parameters are mapped through a constrained latent space (e.g., \(\tanh\) for taper and bend) to maintain valid configurations. After convergence, the candidate with the lowest loss is selected and de-normalized to the original coordinate frame.

4 Evaluation↩︎

We evaluate using four metrics. Chamfer distance (CD) measures the mean squared distance between the original point cloud (or mesh) and points sampled from the fitted primitives (both sampled to 2048 points and normalized to the ground truth shape extent \([-1, 1]^3\)), capturing surface fidelity. Volumetric Intersection over Union (IoU) measures the overlap between the volume enclosed by the fitted primitives and the ground truth mesh (\(128^3\) grid); for datasets provided as point clouds only, we obtain a watertight reference mesh via Poisson surface reconstruction. Overlap Rate (OR), introduced by Wang et al. [7], quantifies how much the fitted primitives overlap each other: \[\text{OR} = \frac{\sum_{\mathbf{x}} \sum_{\theta} M_\theta(\mathbf{x})}{\sum_{\mathbf{x}} \bigcup_{\theta} M_\theta(\mathbf{x})} \label{eq:or}\tag{2}\] where \(M_\theta(\mathbf{x}) = 1\) if point \(\mathbf{x}\) lies inside primitive \(\theta\). OR \(= 1.0\) means no two primitives share any interior volume; higher values indicate redundant overlap. Primitive count (#P) reports the mean number of primitives per abstraction; fewer primitives indicate a more compact, interpretable decomposition.

4.1 Experimental Setup↩︎

We evaluate on two datasets: HumanPrim [4] (314 diverse objects as 10k-point clouds, testing multi-category generalization) and Toys4K [41] (315 everyday objects across diverse categories, with ground truth meshes for IoU).

We compare against four methods: Primitive Anything [4] (auto-regressive transformer, trained on human-crafted abstractions), Fine-to-Coarse (F2C) [28] (progressive cuboid refinement, trained on ShapeNet), SuperDec [3] (feed-forward superquadric decomposition, trained on ShapeNet), and EMS [5] (optimization-based EM superquadric fitting, category-agnostic). These methods collectively represent the main paradigms in compact shape abstraction: supervised feed-forward networks (Primitive Anything, F2C, SuperDec) and training-free optimization (EMS), together covering the current state of the art in superquadric and cuboid decomposition.

4.2 Results↩︎

1 presents quantitative results across all datasets.

Surface fidelity.

Our method achieves the lowest CD on both benchmarks across all evaluated methods. On HumanPrim we obtain CD = 0.079, the lowest of all evaluated methods, ahead of Primitive Anything (0.086), SuperDec (0.090), EMS (0.126), and F2C (0.127). As shown in 2, our primitives closely follow the surface of each part, with superquadric shapes conforming to elongated, curved, and boxy geometry as needed. On Toys4K our method achieves CD = 0.093, the lowest of all methods, including SuperDec (0.104) and Primitive Anything (0.145). 3 shows that this surface fidelity generalizes across a wide range of object types: from compact, near-primitive shapes (apple, bottle) to articulated structures (bicycle, animal limbs), where each part is tightly enclosed by a single fitted superquadric. The weaker performance of F2C (CD = 0.143) and Primitive Anything (CD = 0.145) on Toys4K can be attributed to limited generalization: F2C provides models only for chairs, tables, airplanes, and humans, and fails to produce meaningful abstractions for the broader object classes in Toys4K; qualitative results on its supported ShapeNet categories are shown in 9 in the appendix. Primitive Anything, trained on HumanPrim, performs well within its training distribution but does not generalize to the out-of-distribution objects in Toys4K.

Compactness.

Our method uses a compact set of 5–9 primitives per object across both benchmarks (5.2 on Toys4K, 8.6 on HumanPrim), comparable to F2C and SuperDec (4.9–6.6); the primitive count for each individual shape is annotated in the lower right corner of [fig:humanprim,fig:toys4k]. The slightly higher count on HumanPrim reflects the composition of the dataset: it skews toward complex mechanical and structural objects (vehicles, aircraft, multi-component furniture) that have more distinct semantic parts than the simpler everyday objects common in Toys4K; this is a feature of the representation, not a compactness failure. Primitive Anything uses \(3.4\times\) more primitives on HumanPrim (29.5) and \(14\times\) more on Toys4K (74.9), a scale at which the decomposition captures geometric detail rather than semantic structure, yet achieves higher CD than our method on both benchmarks.

Overlap rate.

Our method achieves near-perfect OR on both benchmarks (1.014 HumanPrim, 1.013 Toys4K) with no explicit overlap penalty in the objective. At the other extreme, PrimAny’s OR of 1.320–2.080 reflects the dense packing of its many primitives. Our low OR follows naturally from semantic segmentation: each primitive is fitted to a distinct, spatially separated part cluster, so inter-primitive overlap is structurally avoided.

Volumetric IoU.

On HumanPrim, our method achieves the highest IoU among compact methods (59.5%), ahead of SuperDec (58.3%), EMS (41.1%), and F2C (38.5%). On Toys4K, EMS leads IoU (59.0%) and our method reaches 55.6%, while achieving substantially lower CD (0.093 vs.). Our method optimizes surface Chamfer distance rather than volume, so a gap relative to methods with volumetric supervision is expected. This is a known property of surface-based fitting: primitives can closely approximate the target surface while leaving interior gaps. Part of this gap is attributable to view coverage: thin structures such as airplane wings, chair seats, or table tops are observed only from above in our four fixed viewpoints, so the underside points are never segmented and the fitted primitive captures only a partial surface, leading to primitives that are too thin or misaligned on flat, sheet-like parts. Introducing additional or more diverse viewpoints would alleviate this limitation. The remainder is attributable to segmentation rather than fitting: with ground-truth parts, IoU rises substantially without any change to the fitter (4.3).

4.3 Segmentation Upper Bound↩︎

To separate the contribution of segmentation from that of fitting, we replace the generative segmentation with ground-truth part labels and leave the fitter unchanged. We use PartNet [42], which provides human-annotated part hierarchies for ShapeNet [43] models across chair and table categories. PartNet labels are semantic rather than instance-level, so we split each part into spatially connected components (recovering, for instance, the four legs of a chair as separate primitives) using the same flood-fill clustering as our main pipeline. We evaluate at the three canonical PartNet granularity levels (L1–L3), fit superquadrics with the same configuration as our main pipeline, and score against the PartNet meshes.

2 reports the result. Ground-truth segmentation improves volumetric IoU at every granularity, by 11–17 points on chairs (37.8% to 49–55%) and 4–8 points on tables, and lowers Chamfer distance once the granularity is comparable to or finer than the generative model’s (e.g.chairs at a matched \(\sim\)​5-primitive budget: 0.105 vs.). GT L1 provides the cleanest isolation, as its primitive count (5.7 for chairs) matches ours (5.2), while L2 and L3 use roughly twice as many primitives (9–10), so their additional gains reflect both finer segmentation and increased representational capacity. Because the fitter and the flood-fill clustering are held fixed, these improvements are attributable entirely to cleaner part segmentation. This confirms that abstraction quality in our pipeline is tied to part segmentation rather than primitive fitting, and substantiates a central claim of this work: as generative image models improve, our training-free pipeline inherits the gains without retraining.

4.4 Run-to-Run Determinism↩︎

Because the generative segmentation model is non-deterministic, repeated calls on the same input may produce different masks, and thus different abstractions. To quantify the resulting variability, we run the full pipeline five times on a subset of 50 objects with identical inputs and report the mean and per-object standard deviation of each metric across runs (3). The variance is small relative to the performance gaps between methods, confirming that results are stable across runs.

4 shows qualitative examples for two objects across five runs. Color assignments differ between runs, as the VLM selects an independent palette each time; the quantitative metrics nevertheless remain stable. The insect example (bottom row) also reveals typical failure modes that vary across runs: wings are occasionally represented as blocky, volumetric structures rather than flat surfaces, and may be misclassified or merged with the body in some views.

4.5 Failure Cases and Limitations↩︎

Three failure modes are most prominent (5); further cases and limitations are discussed in the appendix. On the generation side, large backgrounds can cause the model to hallucinate additional phantom views not present in the input. Furthermore, we do not control the granularity of the decomposition explicitly: the number and coarseness of parts depends on the model’s interpretation of the input, and although the default behavior delivers reasonable outputs in most cases, reliably steering this through prompting remains an open research question. On the clustering side, clusters below a minimum size are discarded as outliers, which can inadvertently remove small but semantically important structures such as a thin chair leg. The main limitation is the runtime of roughly one minute per object, considerably slower than specialized end-to-end models.

5 Conclusion↩︎

We present a training-free pipeline for semantic shape abstraction and show that part segmentation is the critical factor: given clean, semantically coherent part boundaries, a compact and accurate primitive abstraction follows naturally from standard fitting. No task-specific training is required: the segmentation knowledge is entirely supplied by the generative model, and the fitting is handled by a classical optimizer. On HumanPrim and Toys4K, the method achieves the lowest Chamfer distance using only 5–9 primitives per object on average, while applying to new object categories without retraining. The pipeline’s accuracy is tied to the generative model rather than to any learned component: this is its main limitation, since it depends on a capable image-generation model for reliable segmentation, but it also means abstraction quality improves automatically as these models advance, as our ground-truth segmentation study confirms.

Nearer-term improvements include ensemble segmentation (aggregating masks from multiple runs via majority voting to reduce non-determinism) and adaptive view selection to improve coverage of thin or occluded geometry. A natural next step is an agentic setup, in which the foundation model does not merely segment but actively optimizes the abstraction, proposing primitive placements, evaluating fit quality, and iterating to close the loop between semantic understanding and geometric reasoning. More broadly, training-free pipelines that leverage foundation models as a source of semantic knowledge may represent a wider paradigm shift for 3D understanding, replacing category-specific supervision with general-purpose generative capabilities.

Superquadric Formalism↩︎

A superquadric surface is defined by the implicit inside-outside function: \[F(x, y, z) = \left( \left| \frac{x}{a_1} \right|^{\frac{2}{\epsilon_2}} + \left| \frac{y}{a_2} \right|^{\frac{2}{\epsilon_2}} \right)^{\frac{\epsilon_2}{\epsilon_1}} + \left| \frac{z}{a_3} \right|^{\frac{2}{\epsilon_1}} = 1 \label{eq:sq95implicit}\tag{3}\] where \(a_1, a_2, a_3\) control the size along each axis and the shape parameters \(\epsilon_1, \epsilon_2 \in (0, 2]\) continuously interpolate between cuboids (\(\epsilon \to 0\)), ellipsoids (\(\epsilon = 1\)), cylinders, and octahedra (\(\epsilon = 2\)). The corresponding parametric form is: \[\mathbf{s}(\eta, \omega) = \begin{pmatrix} a_1 \cos^{\epsilon_1}\!\eta \; \cos^{\epsilon_2}\!\omega \\ a_2 \cos^{\epsilon_1}\!\eta \; \sin^{\epsilon_2}\!\omega \\ a_3 \sin^{\epsilon_1}\!\eta \end{pmatrix} \label{eq:sq95parametric}\tag{4}\] with \(\eta \in [-\frac{\pi}{2}, \frac{\pi}{2}]\) and \(\omega \in [-\pi, \pi)\), where \(\cos^{\epsilon}\!\theta = \text{sign}(\cos\theta)|\cos\theta|^{\epsilon}\) denotes the signed power function.

We additionally use two global deformations. Tapering scales the cross-section linearly along the \(z\)-axis via parameters \(k_x, k_y\): \[\begin{pmatrix} x' \\ y' \\ z' \end{pmatrix} = \begin{pmatrix} x \cdot (k_x \frac{z}{a_3} + 1) \\ y \cdot (k_y \frac{z}{a_3} + 1) \\ z \end{pmatrix} \label{eq:taper}\tag{5}\] Bending curves the \(z\)-axis along an arc of radius \(R = a_3 / k_b\) in a direction specified by angle \(\alpha\), enabling representation of curved structures.

Segmentation Prompts and Model Comparison↩︎

The pipeline issues two sequential model calls per object; both receive the same four-view composite image as input.

Call 1: VLM Analysis. A vision-language model is asked to identify the depicted object and propose a decomposition into semantically meaningful parts (6). Each part is assigned a unique vibrant color (avoiding white, which is reserved for the background), and the model returns a structured JSON listing part names, short descriptions, and hex color codes. The prompt explicitly instructs the model not to distinguish left from right instances of symmetric parts: all chair legs, for example, share one color. This ensures that the subsequent reprojection step can produce a consistent 3D labeling across opposing views.

# TASK
Analyze the following multi-view images of an object:
- What object is depicted?

Segment it into distinct parts.
- In which parts can it be subdivided, e.g. legs, seat, wings, arms, ...
- Subdivide the object using simple geometric shapes
  (cuboids, cylinders, spheres, ...)
- Each part should be represented by a single geometric shape.
- If the object is composed of multiple instances of the same part, assign the same color to all instances.
- Do NOT differentiate between left and right instances.

Assign a single color to each unique part.
- Use vibrant colors like #FF0000 or #FF00FF.
- Do NOT use #FFFFFF as this is the background color.

# OUTPUT
Answer in a structured JSON format.
Answer only with the JSON string, nothing else.

{ "object": "", "parts": [{ "name": "", "description": "", "color": "" }] }

Call 2: Generative Image Model Mask Generation. The same four-view image is passed to a generative image model together with the JSON output of Call 1 as a structured prefix. The model is instructed to paint a color-coded segmentation mask over the input, using exactly the colors specified in the JSON and maintaining color consistency as the object rotates across all four views. It must not alter the object geometry, must use a white background, and must preserve the original aspect ratio.

{JSON Analysis Output}

# TASK
Generate a multi-view segmentation mask of the object:
- Color the distinct parts exactly as defined above.
- Maintain color consistency across all four perspective
  views (tracking the correct part as the object rotates).
- Each image consists of exactly four distinct views.
- Do NOT change the object in the image.
- Use a white background #FFFFFF.
- Keep the original aspect ratio.

The mask-generation step imposes the most demanding requirements: the model must follow a structured prompt, respect a specific color palette, and maintain cross-view consistency. We evaluated eight candidate models on this task using the identical two-call prompt (7). Several models succeed: GPT Image, NanoBanana, and HunyuanImage 3.0 preserve a single color per part across all four views and respect part boundaries. The remaining models fail in one or more ways: inconsistent colors across opposing views, hallucinated phantom views, color bleed across boundaries, or geometry alteration. Among the successful models, NanoBanana 2 [44] delivers the best mask quality at the lowest cost, and is used for all results in this paper.

Instance Segmentation Ablation↩︎

Our pipeline assigns one color per semantic part type, so all instances of the same part share a color. As an ablation, we test whether asking the model to assign a unique color per instance improves part separation. We replace the corresponding Call 1 prompt instructions as follows:

...
(*@{\scriptsize\ttfamily\sout{- If the object is composed of multiple instances of the same}}@*)
(*@{\scriptsize\ttfamily\sout{  part, assign the same color to all instances.}}@*)
(*@{\scriptsize\ttfamily\sout{- Do NOT differentiate between left and right instances.}}@*)
(*@{\scriptsize\ttfamily\textcolor{green!50!black}{- Differentiate multiple instances of the same category,}}@*)
(*@{\scriptsize\ttfamily\textcolor{green!50!black}{  like left and right or front and back.}}@*)
...

Call 1 reliably adopts the new instruction: the chair example in 8 receives 16 distinct colors, differentiating each leg, stretcher, and backrest slat individually. The generative image model (Call 2), however, cannot maintain these instance identities across viewpoints: left/right and front/back relationships are especially prone to being swapped or lost between views. The resulting masks are inconsistent across views and cannot be reprojected into a coherent 3D labeling, which is why our pipeline retains the original same-color-per-type prompt.

Failure Cases and Limitations↩︎

4.5 in the main paper summarises the three main failure modes; instance identity across views is discussed separately in ¿sec:sec:instance95seg?. Here we provide a more detailed account of each, together with additional structural limitations not discussed there.

Phantom view hallucination. When the multi-view composite image contains large areas of white background, the generative model occasionally fills that space with additional repetitive views that were not present in the input. The resulting segmentation mask is locally invalid, but this does not propagate to the 3D abstraction: votes from hallucinated pixels land outside the object silhouette and are silently discarded during reprojection. Cropping the composite more tightly around each object reduces the frequency of this failure.

Uncontrollable part granularity. The model determines how finely to segment on its own: the same object class may receive two parts in one run and six in another, though the total number of parts tends to remain within a reasonably compact range across object types. While the default behavior is usually semantically reasonable, controlling the exact number of parts is straightforward via prompting and remains an open direction for further exploration.

Instance identity across views. The model cannot reliably maintain left–right part identity across opposing front and back views. A chair’s left leg may receive one color in the front view and a different color in the back view, making true instance segmentation impossible to reproject into a consistent 3D labeling. Our prompt therefore instructs the model to assign one color per part type rather than per instance; the consequence is that instance-level distinctions (e.g., individual legs) collapse into a single semantic cluster (8).

Small-cluster loss and color bleed. The flood-fill step discards clusters below a minimum size, which can inadvertently remove small but semantically important structures. Minimal parameter tuning has been performed; systematic adjustment of the minimum cluster size and spatial radius could reduce such losses. A related issue arises when adjacent parts are assigned similar or bleeding colors: they merge into a single cluster fitted by one primitive. A joint segmentation-and-fitting algorithm, such as EMS [5], could recover multiple primitives from such merged regions.

Surface-only fitting. The optimizer minimizes surface Chamfer distance; since only the visible surface can be rendered and segmented, no volumetric signal is available during fitting. This contributes to the lower IoU values observed relative to methods that supervise volumetric overlap directly.

Fixed viewpoints. We use four fixed opposing viewpoints as a heuristic. No optimization of camera angles or view count is performed, so thin structures such as airplane wings or table tops may never be observed from below, leaving the underside unrepresented and yielding lower IoU for those parts.

Future Work and Improvements↩︎

The conclusion of the main paper identifies three near-term directions: ensemble segmentation to reduce non-determinism, adaptive view selection for better coverage, and an agentic setup in which a foundation model actively optimises the abstraction. Below we elaborate on these and describe additional directions not discussed there.

Ensemble segmentation. Running the full pipeline multiple times on the same object and aggregating the resulting per-pixel labels via majority voting before clustering would average out the stochastic color assignments and part-boundary variations observed across runs (4.4). The cost is \(k\) additional inference calls per object; even \(k = 3\) is likely to yield a substantially more stable abstraction.

Adaptive view selection. The four fixed opposing viewpoints used here are a simple heuristic. A more principled strategy would select views that maximise coverage of unseen surface area, for example by iteratively choosing the camera angle that minimises the fraction of unobserved points. This is particularly relevant for flat or strongly concave objects, where a fixed layout may leave large regions unobserved.

Sequential multi-view generation. The current single-pass composite reduces inference cost but introduces layout distractions that can trigger phantom-view hallucination. An alternative is to present each view individually and accumulate masks incrementally, allowing the model to focus on one viewpoint at a time at the expense of additional calls.

Qualitative Comparison on ShapeNet↩︎

9 shows results on three ShapeNet categories (Airplane, Chair, Table) for which all compared methods provide models. Our method recovers semantically meaningful part structure using a low number of primitives, comparable to the category-specific methods SuperDec and F2C, which perform equally well on their supported classes. EMS fails on several instances, producing geometrically arbitrary splits that do not align with semantic parts. PrimitiveAnything consistently oversegments, requiring 30–140 primitives per object to achieve similar surface coverage.

References↩︎