1 Introduction↩︎

3D shape editing is a classical problem in computer graphics and has recently attracted fresh attention in representational learning and neural geometry processing. The central challenge is often to build an appropriate neural shape representation that can support: (i) category-agnostic shape editing across different object classes; (ii) multiple operators for intuitive and fine-grained editing; (iii) awareness of shape semantics so that the edited shape responds naturally and accurately to the editing operation; and (iv) preservation of unedited shape regions while maintaining high-quality edit results.

Classical shape editing and manipulation usually rely on editing handles or proxies [1], e.g., points, curves, sketches, skeletons, or cages. These proxies provide a reduced representation, or abstraction, of the shape, making it easier to manipulate. With the emergence of many deep neural representations for 3D shapes, a common editing strategy is to carry out operations in a latent space, which functions as a proxy and is typically associated with a generative model such as autoencoders, adversarial networks (GANs) [2], or diffusion [3]. Latent-space editing has achieved notable success for images [4]–[6] and videos [7], where editing is mainly realized by dragging to deform objects or adjust scene layouts. As shown in Figure 1, standard 3D shape edit operations differ substantially in nature. Recent methods for latent-space 3D shape manipulations [8]–[11] either combine implicit functions with geometric primitives or link a shape latent space to CLIP space [12]. However, because geometric primitives and text prompts are often coarse, these methods still face difficulty in edit quality, particularly for fine-grained editing.

In this paper, we propose a new coupled neural shape (CNS) representation for 3D shapes. The representation is semantic-aware and supports a range of intuitive and fine-grained edit operations. Our CNS representation contains a latent code, which encodes high-level global semantics (e.g., symmetry) of the input shape, and a 3D neural feature volume, which supplies spatial context for relating local shape changes to a given edit.

Importantly, CNS construction is not restricted to a particular generative model. It can be instantiated flexibly on different backbones, from a category-specific 3D inversion model [13] to category-agnostic 3D foundation models with text- and image-conditioned generation [14], [15]. For the 3D inversion model, constructing CNS is straightforward: its encoder directly takes a 3D shape as input, allowing us to encode the input shape into the latent code z and extract the feature volume F from the generator. By contrast, constructing CNS on 3D foundation models is less direct, since these models are conditioned on a text or image embedding rather than on a shape. We therefore use the condition embedding as the global latent code. This choice is natural in the image-to-3D setting, where the image already contains rich shape information. In the text-to-3D setting, however, a text prompt may describe multiple plausible shapes, so we introduce a latent refinement design that optimizes the text embedding to faithfully reconstruct the input shape. To construct the coupled neural feature volume, we further present an inversion-guided neural volume construction mechanism, which uses a second-order inversion scheme to obtain a feature volume faithfully aligned with the input shape.

Based on the CNS representation, edits are performed implicitly in the latent space through coupled neural shape optimization. Specifically, given an editing operation, we co-optimize the two coupled components in the CNS representation. The editing operators currently supported by our framework include copy, resize, delete, mix, point-wise drag, and region-wise drag; see Figure 1. Each operator is converted into an objective that guides CNS optimization, so that the latent code and neural feature volume can be iteratively co-optimized to satisfy the editing operation.

Because the coupled optimization includes a global latent code, a local editing objective can back-propagate to this code and unintentionally change unedited regions. To address this issue, we introduce two complementary region-wise control mechanisms, i.e., KV-cache replacement and latent feature regularization, to better preserve the unedited regions.

We instantiate our coupled neural shape editing framework in two versions: CNS-Edit, a category-specific version built on a 3D inversion model, and CNS-Edit++, a category-agnostic version built on 3D foundation models. With our approach, we can produce a wide range of editing results that are aware of shape semantics and difficult to obtain with existing approaches. For instance, as shown in Figure 1, both the delete and copy operators can introduce topology changes to a 3D shape, which our method handles seamlessly. Quantitative and qualitative evaluations demonstrate the strong capabilities of CNS-Edit and CNS-Edit++ compared with state-of-the-art solutions.

Our main contributions are summarized as follows:

• We propose a new coupled neural shape (CNS) representation, composed of a latent code and a 3D neural feature volume, for 3D semantic shape editing.

• We construct the CNS representation on diverse 3D generative backbones, from a category-specific 3D inversion model to advanced category-agnostic 3D foundation models.

• We introduce a coupled neural shape optimization procedure that co-optimizes the two coupled components to satisfy the editing operation.

• We support multiple 3D shape editing operators, including copy, resize, delete, mix, point-wise drag, and region-wise drag, enabling flexible and fine-grained 3D shape editing.

• We develop two complementary region-wise control mechanisms, i.e., KV-cache replacement and latent feature regularization, to help preserve unedited regions during shape editing.

• Extensive qualitative and quantitative experiments show that our method outperforms existing state-of-the-art solutions.

2 Related Work↩︎

Shape editing has remained a long-standing challenge in graphics. To preserve geometric fidelity during shape editing, researchers have used a broad set of techniques, e.g., including but not limited to ARAP deformation [16], cage-based deformation [17]–[20], differential coordinates [21], slippage preservation [22], and Laplacian operator [23]. These shape editing techniques operate directly on mesh vertices and do not provide semantic understanding or semantic control.

The later development of structure-aware shape manipulation [24] commonly follows an analyze-and-edit paradigm. Such methods first discover structures in the edited shape, as represented by feature curves [25], part bounding boxes [26], and symmetries [27], and then perform edits while preserving these structures. However, before deep learning became widely adopted, structure discovery schemes were limited in capability and often brittle. Moreover, for segmented 3D shapes with separated parts, preserving proper part connections has often received insufficient attention [28]. Some works, e.g., [29], [30], also rely on CAD programs for shape manipulation. In this section, we focus mainly on learning-based methods for shape generation, manipulation, and editing, which are most closely related to our work.

Shape Editing with Learned Models. One line of methods [8]–[11], [31], [32] couples implicit functions with simple geometric primitives, so that the implicit functions can be edited by changing the parameters of the primitives. These methods improve the flexibility of shape generation, but usually trade off generative quality. Other methods [33]–[38] develop deformation-based shape editing. Nevertheless, an important limitation of these methods is that they cannot edit shape topologies. Recent works further incorporate additional modalities such as texts and sketches. Text-guided methods [39]–[50] are effective for high-level semantic modifications, but they often lack precise spatial control during editing. Sketch-guided methods [12], [51]–[55] allow users to alter a shape through sketch modifications, yet they often have difficulty handling non-expert, and thus low-quality, inputs or out-of-distribution inputs. CLIPXplore [12] is particularly relevant, as it also proposes a coupled representation and co-optimization. By connecting a CLIP space with a 3D shape space, however, its representation is designed for shape exploration rather than fine-grained editing.

Our framework proposes a new coupled neural shape representation for 3D shape editing, together with a set of operators for manipulating this representation. These operators are designed to support efficient and intuitive 3D shape editing, while producing high-fidelity results, fine-grained controllability, and topology modification.

Generative 3D Modeling. Many methods have studied 3D generation based on classical shape representations or recent neural representations. These representations include voxels [56], [57], point clouds [58]–[68], meshes [69]–[73], implicit functions [9], [74]–[81], and radiance fields (e.g., NeRF, 3D-GS) [82]–[90]. Alongside general 3D generation, specialized works have examined 3D modeling under challenging observations and interactions [91]–[99]. Recently, enabled by large-scale 3D datasets [100], [101], an increasing number of works [14], [15], [76], [102]–[105] have scaled up 3D generation and built 3D foundation models. These methods show strong generalizability across diverse object categories and input conditions, greatly improving the fidelity and diversity of 3D shape generation. Although these advanced works are strong at high-fidelity shape generation, they are not designed for fine-grained editing tasks, since they typically do not provide sufficient controllability. In this work, we construct the CNS representation on such 3D foundation models and repurpose their generative priors for category-agnostic shape editing.

Image Editing via Generative Models. Generative adversarial networks (GANs) [2] have played a foundational role in many later methods for image manipulation and editing [106]–[111]. However, accurately inverting real images back into latent codes remains challenging because of the limitations of GAN’s generative capabilities [112].

As a result, these limitations restrict the use of GANs in many real-world applications. Recently, with large-scale text-to-image diffusion-based models such as [113], many diffusion-based methods [114]–[118] have been proposed for text-based image editing. Nevertheless, editing images using the text remains an open problem, because texts usually lack precise, pixel-level spatial controllability. More recently, Pan et al. [4] proposed “Drag Your GAN” to further improve image editing controllability. It allows users to interactively drag arbitrary image points to target locations, enabling pixel-precise editing and producing impressive results. Following [4], later studies [5], [6] extend this manipulation framework to the stable-diffusion [113], which substantially improves the quality of image manipulation. Some recent methods attempt to exploit 2D image priors for 3D editing by applying drag on rendered views and lifting the edits back to 3D [119], [120]. However, these methods suffer from multi-view inconsistency and limited 3D spatial control, making faithful and fine-grained shape editing difficult.

3 Overview↩︎

Given a 3D shape, our goal is to modify it according to an editing operation. Specifically, we introduce the Coupled Neural Shape (CNS) representation, which allows us to exploit a pre-trained latent shape space and implicitly modify the shape through the CNS representation in the latent space so that it matches the editing operation. Figure 2 illustrates our framework, which contains the following components:

• First, we build the CNS representation, which contains a global latent code \(z\) for capturing overall shape semantics and a neural feature volume \(F\) for providing spatial context during editing. Importantly, CNS does not depend on a single backbone: it can be constructed on both a category-specific 3D inversion model and text- or image-conditioned 3D foundation models. We obtain \(z\) either by encoding the input shape (category-specific inversion model) or from the text or image embedding (3D foundation models), and we extract \(F\) from intermediate features of the generator so that it is faithfully aligned with the input shape. The two components are tightly coupled and jointly support semantic-aware, spatially controllable editing; see Figure 2 (a) for the overall procedure, Figure 3 for CNS construction on 3D foundation models, and Section 4.1 for details.

• Second, we design the coupled neural shape optimization procedure (Section 4.3) to update the CNS representation for the shape editing operation. Specifically, we first convert the editing operation into an optimization objective defined on the neural volume \(F\). Then, as shown in Figure 2 (c), we iteratively co-optimize code \(z\) and neural volume \(F\) to obtain a refined code \(z_N\). In each iteration, we search for code \(z_k\) by gradient descent in the shape space, so that its associated neural volume \(F_k\) follows the editing objective. This process helps preserve structural integrity in the edited shapes while accounting for shape semantics.

• Finally, we introduce a family of editing operators, i.e., copy, resize, delete, mix, point-wise drag, and region-wise drag (Section 4.2); see Figure 2 (b) for illustrations. We further derive procedures that translate each operator into an operator-specific objective (Section 4.4). With this objective, the co-optimization on the CNS representation can be properly guided to produce the final code \(z_N\), which is then decoded to generate the edited shape; see Figure 2 (d).

4 Method↩︎

4.1 Coupled Neural Shape Representation↩︎

We first present the Coupled Neural Shape (CNS) representation, which is designed to enable 3D shape editing. This representation is derived from the input shape and is composed of a pair of coupled neural tensors: (i) a global latent code \(z\), which captures overall shape semantics, and (ii) a 3D neural feature volume \(F\), which captures spatial context, allowing spatial locations in the neural volume to be associated with the input shape.

In particular, CNS is not bound to a specific 3D representation or generative backbone, so it can be constructed with different 3D generative backbones. Below, we describe how the CNS representation is built under three representative backbones: category-specific 3D inversion model [13], text-conditioned 3D generation model [14], and image-conditioned 3D generation models [14], [15].

Category-specific 3D Inversion Model. We first instantiate CNS using the category-specific 3D inversion framework of [13]. Given an input shape, we follow [74] to encode it into a compact wavelet coefficient volume. Specifically, the shape is sampled into a high-resolution truncated signed distance field volume, and the distance field is then converted into a wavelet coefficient volume. This wavelet representation keeps a spatial association with the original shape because of the local support property of the wavelet transform: a local modification in the wavelet volume affects only the associated local region of the shape, and vice versa.

Global latent code \(z\). We use the pre-trained diffusion-based autoencoder from [13] to construct the latent code \(z\). The encoder maps the wavelet coefficient volume to an initial latent code \(z'\), which is then refined through latent optimization following [13] to obtain a more faithful global latent code \(z\). This refined code forms the global latent component of CNS and provides a semantically meaningful latent space for shape editing.

Neural feature volume \(F\). To construct the neural feature volume \(F\), we deliberately run \(t\) iterations with the U-Net when denoising the noise volume, where \(t < T\). We then feed the partially-denoised volume and the global latent code \(z\) back into the U-Net and extract intermediate features from it. Specifically, we use the feature volume from the fourth last layer of the U-Net as the 3D neural feature volume \(F\) in our CNS representation. This choice is made because feature volumes from deeper layers lack sufficient spatial context, whereas those from shallower layers contain limited shape semantics. Importantly, before producing the final output, this volume is further processed only by convolution layers with a limited spatial receptive field. Therefore, local modifications in this volume typically correspond to targeted changes in the desired regions of the shape.

Category-agnostic 3D Foundation Models. We further instantiate CNS on category-agnostic 3D foundation backbones, including the text- and image-conditioned TRELLIS [14] and Direct3D-S2 [15]. Both backbones use a two-stage generation pipeline, where the first stage produces a dense voxel that captures the geometry structure, and the second stage adds geometric details. We construct CNS on the first stage of these backbones, as this stage provides the geometry structure needed for shape editing.

The first stage follows a rectified-flow latent generation formulation. Given an input shape \(S\), its voxelized representation is encoded into a latent \(x_0=\mathcal{E}(S)\), while an image or text encoder produces a condition embedding \(c\). Conditioned on \(c\), the generator \(\mathcal{G}\) learns to generate the target latent \(x_0\) from Gaussian noise \(x_1\). To instantiate CNS, we take \(c\) as the source of the global latent code \(z\) and construct the neural feature volume \(F\) from intermediate features of \(\mathcal{G}\). Figure 3 shows this construction for both the image- and text-conditioned settings. Next, we explain how \(z\) and \(F\) are obtained in the text- and image-conditioned settings.

Global latent code \(z\). For the image-conditioned setting, we first render the input shape \(S\) into an image and feed it into the DINOv2 image encoder [121]. We directly use the resulting image embedding \(c_{\mathrm{img}}\) as the global latent code, i.e., \(z=c_{\mathrm{img}}\). For the text-conditioned setting, we initialize the text prompt with the category of \(S\) (e.g., “a chair”) and encode it using the CLIP text encoder [122] to obtain the initial text embedding \(c_{\mathrm{text}}\). Since such a category-level prompt may correspond to multiple different shape instances, \(c_{\mathrm{text}}\) alone cannot represent the specific input shape \(S\).

To reduce this ambiguity, we introduce a latent refinement design that optimizes \(c_{\mathrm{text}}\) into an instance-specific latent code \(c\), i.e., a code specialized for reconstructing the input shape \(S\). Specifically, we freeze the generator \(\mathcal{G}\) and optimize the latent code with the same rectified-flow objective used when training \(\mathcal{G}\): \[\mathcal{L}_{\text{refine}}(c) = \mathbb{E}_{t,x_1} \left\| \mathcal{G}(x_t,c,t) - v_t \right\|_2^2, \quad\] where \(x_t\) denotes the noisy latent state at time \(t\), and \(v_t\) is the target velocity. Notice that the generator \(\mathcal{G}\) remains fixed during the refinement. After optimization, the refined text embedding \(c\) serves as an instance-specific latent code that captures the geometries of the input shape \(S\) and is used as the global latent code \(z\), i.e., \(z=c\), for subsequent editing.

Neural feature volume \(F\). To construct the neural feature volume \(F\) for both image- and text-conditioned foundation models, we introduce an inversion-guided neural volume construction mechanism. Given the encoded input shape \(x_0=\mathcal{E}(S)\), this mechanism first inverts \(x_0\) to an intermediate latent \(x_t\). We then feed \(x_t\) together with the obtained global latent code \(z\) into the generator \(\mathcal{G}(x_t,z,t)\) and extract \(F\) from its intermediate features.

A direct way to obtain \(x_t\) is to apply first-order inversion along the rectified-flow trajectory. Specifically, at each inversion step \(\tau\), the generator predicts the velocity \(v_\tau=\mathcal{G}(x_\tau,z,\tau),\) and the latent is updated by \(x_{\tau+\Delta \tau}=x_\tau+\Delta \tau \times v_\tau .\) However, this first-order update determines each inversion step using only one velocity evaluation at the current latent \(x_\tau\). Consequently, the update direction can be inaccurate, leading to reduced fidelity; see Figure 22 (c). Therefore, inspired by [123], we adopt a second-order inversion scheme to obtain a more accurate velocity estimation. This produces a more faithful intermediate latent \(x_t\) and thus a neural feature volume that is better aligned with the input shape.

Specifically, we split the interval \([0,t]\) into \(K\) steps. For each inversion step from \(t_1\) to \(t_2\), where \(\Delta=t_2-t_1\), we first predict the velocity at the current latent: \[v_1=\mathcal{G}(x_{t_1},z,t_1).\] We then update the latent using a second-order Taylor approximation: \[x_{t_2}\approx x_{t_1} +\Delta \cdot v_1 +\frac{1}{2}\Delta^2 \cdot v^{(1)}_1,\] where \(v^{(1)}_1\) denotes the derivative of the velocity \(v_1\). Since this derivative is not directly available from the generator, we estimate it through a half-step prediction: \[t_m=t_1+\frac{\Delta}{2},\quad x_{t_m}= x_{t_1}+\frac{\Delta}{2} \cdot v_1,\] and evaluate the velocity at the predicted midpoint \(x_{t_{m}}\): \[v_m=\mathcal{G}(x_{t_m},z,t_m).\] The velocity derivative is then approximated as \[v^{(1)}_1\approx \frac{v_m-v_1}{\Delta/2}.\] Substituting this estimate into the second-order Taylor update gives the practical inversion step: \[x_{t_2}\approx x_{t_1}+\Delta \cdot v_m.\]

Starting from \(x_0\), we repeat this procedure for \(K\) steps to obtain the intermediate latent \(x_{t}\). Compared with first-order inversion, this second-order approximation uses the midpoint velocity to estimate the rectified-flow trajectory more accurately, reducing inversion error and producing an intermediate latent that better preserves the geometry of the input shape \(S\). We then feed \(x_{t}\) and the condition code \(z\) into the generator \(\mathcal{G}\) and extract its intermediate features to form the neural feature volume \(F\).

Note: \(F\) and \(z\) are Coupled. It is worth emphasizing that the two neural tensors in our CNS representation are closely coupled. Changes in one component can be induced from modifications in the other. On the one hand, inducing changes in the neural volume \(F\) from a modified \(z\) is straightforward by following the CNS construction procedure described above, i.e., partial denoising for the category-specific inversion model, or inversion-guided neural volume construction for the 3D foundation models. On the other hand, when a modified neural volume \(F\) is given, the extraction of \(F\) is differentiable, so we can measure the difference between the original volume and the modified one with a loss. Thus, the changes can be back-propagated to the global code \(z\), and gradient descent can be used to obtain the updated code \(z_N\) that better matches the modified \(F\), as shown in Figure 2 (c). With these two coupling relations, shape editing can be formulated as deriving an objective from the editing operation and then co-optimizing the neural volume \(F\) and code \(z\) accordingly.

4.2 Shape Editing Operators↩︎

In this work, we introduce six shape editing operators:

• Copy Operator. This operator allows a user to copy part of the input shape and paste it to another location in the shape. As shown in Figure 4 (a), the user can mark a source region (in red) and specify a displacement (shown by the arrow) from the source region to the target region (in blue). The operator then copies the local geometry from the source region to the target region. It is important that the pasted geometry is seamlessly integrated with the original target region in the shape; see, e.g., the cloned stretcher connected with the chair legs in Figure 4 (a).

• Resize Operator. This operator allows a user to scale a selected region in a shape around a chosen anchor point along desired direction(s). As shown in Figure 4 (b), the selected region is marked in red, the desired direction is indicated by the black arrow, and the anchor point is marked by the blue dot. With resize, a local part can be scaled along a specified direction; see, e.g., the rectangular panel in the CAD model in Figure 4 (b). In this example, note that only the left panel is marked. The right panel in the CAD model is automatically resized together with the left panel.

• Delete Operator. This operator allows a user to remove a selected part from the input shape. For example, in Figure 4 (c), the user marks the cushion on the couch (in red) and removes it. Notably, after the cushion is removed, the newly exposed back of the couch remains semantically coherent with the surrounding regions of the shape.

• Mix Operator. This operator allows a user to replace a selected part of an input shape with a part from a reference shape, producing a mixed shape that combines geometry from both sources. As illustrated in Figure 4 (d), the user selects a target region on the input shape (in red) and specifies a semantically corresponding region on the reference shape (in green). The operator then transfers geometry from the reference region to the selected region of the input shape. Different from the copy operator, which reuses source geometry from the input shape itself, the mix operator obtains geometry from a separate reference shape and therefore enables cross-shape part transfer. For instance, as shown in Figure 4 (d), the legs of the input table can be replaced with the star-shaped pedestal base from the reference shape.

• Point-wise Drag Operator. Motivated by [4], this operator allows a user to drag the local geometry around a point toward a target location, so that the geometries along the path adapt to the changes caused by the drag. As shown in Figure 4 (e), by marking and dragging a source point (in purple) toward a target point (in red), we reduce the size of the hole. This operation only requires marking a point and dragging it.

• Region-wise Drag Operator. This operator allows a user to drag a selected region in the input shape by specifying several representative source-target point pairs. As illustrated in Figure 4 (f), the user marks a source region and specifies several representative source points inside the region, together with their target points. As shown in Figure 6, region-wise drag offers denser and more precise control over the selected region than point-wise drag.

Although some of the above operators can be partially achieved by traditional methods, e.g., resize can be done in [17]–[19], our method treats the editing operation in a more semantic manner. For example, when resizing one part of the shape, associated symmetric part(s) can be updated automatically; see, e.g., Figure 4 (b). This demonstrates that our method accounts for shape semantics, e.g., symmetry, during the editing process. In addition, the operators introduced above are atomic, so new operators can be formed by combining existing ones. For example, combining “copy” and “delete” yields the “cut-paste” operator. As shown in Figure 5, we first replicate the horizontal stretcher of a shape using the copy operator, and then remove the original stretcher using the delete operator.

4.3 Coupled Neural Shape Optimization↩︎

We next introduce the overall procedure, coupled neural shape optimization, for modifying the CNS representation under a given editing operation. This procedure has two main steps. First, we derive an operator-specific objective \(\mathcal{L}_{\text{op}}\) from the editing operation, where \(\mathcal{L}_{\text{op}}\) defines how the neural feature volume \(F\) of the input shape should be updated. Second, using objective \(\mathcal{L}_{\text{op}}\) as guidance, we co-optimize the two components in the CNS representation and decode the optimized CNS to produce the edited shape.

(i) Derive objective \(\mathcal{L}_{\text{op}}\) from editing operation. Objective \(\mathcal{L}_{\text{op}}\) consists of two parts. The first part is a list of spatial coordinates in the neural feature volume \(F\), specifying the target region in \(F\) where the editing operation should induce changes: coordinate list \(\Gamma = \{(x_i, y_i, z_i) | \forall i \in [1, \cdots,M]\}\), where M is the length of the list. The second part is a list of target feature values associated with the spatial coordinates in the first list: value list \(V = \{f_i | f_i \in \mathbb{R}^C, \forall i \in [1, \cdots, M] \}\), where \(C\) is the channel size. Using these two lists, the operator-specific objective \(\mathcal{L}_{\text{op}}\) is formulated as \[\label{eq:op95loss} \mathcal{L}_{\text{op}} = | F_k[\Gamma] - sg(V) |_{1},\tag{1}\] where \(F[\cdot]\) denotes the slicing operation on the neural feature volume \(F\); \(|\cdot|_1\) is the standard \(L_1\) loss; and \(k\) is the iteration round.

The formulation above applies to all operators introduced in Section 4.2. Section 4.4 gives details on how to derive objective \(\mathcal{L}_{\text{op}}\), i.e., \(\Gamma\) and \(V\), for each operator. If a location in \(\Gamma\) is not an integer coordinate in \(F_k\), the slicing output is obtained by trilinear interpolation over neighboring values. In addition, we apply the stop-gradient operator \(sg\) to the target values in \(V\) to encourage the values in \(F_k\) to approach the target values, instead of allowing the target values to move toward \(F_k\) during optimization.

(ii) Coupled Co-optimization. Once \(\mathcal{L}_{\text{op}}\) is defined, we co-optimize latent code \(z\) and neural volume \(F\) in the CNS representation for \(N\) iterations. At the start of co-optimization, we compute the initial neural volume \(F_0\) from the global latent code \(z_0\) of the input shape, following the procedure in Section 4.1. We then evaluate \(\mathcal{L}_{\text{op}}\) on the generated neural volume \(F_0\) to measure the difference between the values in the target region \(\Gamma\) of \(F_0\) and the desired target values \(V\). By computing the gradient of \(\mathcal{L}_{\text{op}}\) with respect to \(z_0\), we obtain an updated \(z_1\) through one gradient step with learning rate \(\alpha\). Repeating this procedure on each newly generated latent code yields a sequence of CNS representations, i.e., \(\{(z_0, F_0), (z_1, F_1), \cdots, (z_N, F_N)\}\), as illustrated in Figure 2 (c).

Overall, the CNS representation from later optimization steps should better satisfy the editing operation. Although we set a maximum number of optimization iterations, the optimization can stop early when a termination condition is met; see Section 5.1 for details on the termination condition of each operator. Finally, the optimized CNS representation is decoded to produce the edited shape by running the decoder over the remaining time steps, guided by the optimized latent code \(z_N\).

4.4 Deriving Objective for Each Operator↩︎

This section describes the operator-specific objectives, \(\mathcal{L}_{\text{copy}}\), \(\mathcal{L}_{\text{resize}}\), \(\mathcal{L}_{\text{delete}}\), \(\mathcal{L}_{\text{mix}}\), \(\mathcal{L}_{\text{pdrag}}\), and \(\mathcal{L}_{\text{rdrag}}\). For each operator, we define the target coordinates \(\Gamma\) and target feature values \(V\) used in Eq. 1 :

• \(\mathcal{L}_{\text{copy}}\). Given the selected region to be copied, we first identify the coordinates in the neural volume covered by the selection to form the coordinate list \(\Gamma_{\text{copy}}\). We then obtain the pasting region \(\Gamma_{\text{paste}}\) by shifting each coordinate in \(\Gamma_{\text{copy}}\) according to the user-provided displacement vector. The goal is to make the feature values in \(\Gamma_{\text{paste}}\) closely match the corresponding values in \(\Gamma_{\text{copy}}\). Thus, we set \(\Gamma=\Gamma_{\text{paste}}\) and \(V = F_0[\Gamma_{\text{copy}}]\) to define the objective \(\mathcal{L}_{\text{copy}}\).

• \(\mathcal{L}_{\text{resize}}\). Given the selected region to be resized, we denote its associated coordinate set as \(\Gamma_{\text{resize}}\). We then find the bounding box \(\mathcal{B}\) that contains the entire \(\Gamma_{\text{resize}}\) and resize \(\mathcal{B}\) into a new bounding box \(\mathcal{B}'\) according to the specified anchor point and resize direction. The target region \(\Gamma\) is therefore defined as the set of coordinates inside bounding box \(\mathcal{B}'\). For the target feature values \(V\), the values lie in \(\mathcal{B}'\) and should proportionally come from the values in \(\mathcal{B}\). Thus, we use trilinear interpolation to sample feature values in \(\mathcal{B}\) of \(F_0\) and obtain the target feature value at each coordinate in \(\Gamma\).

• \(\mathcal{L}_{\text{delete}}\). Given the selected region to be removed, we denote its coordinates as \(\Gamma_{\text{delete}}\). Our idea is to make the feature values in \(\Gamma_{\text{delete}}\) match those in an empty region of neural volume \(F_0\). Therefore, we search for an empty region, denoted as \(\Gamma_{\text{empty}}\), in the given shape and extract its local feature values. Then, we set \(\Gamma=\Gamma_{\text{delete}}\) and \(V=F_0[\Gamma_{\text{empty}}]\) to define the objective \(\mathcal{L}_{\text{delete}}\).

• \(\mathcal{L}_{\text{mix}}\). Given a reference shape and an input shape, we first construct their neural feature volumes, denoted as \(F_0\) and \(F_0^{\text{ref}}\), respectively. Based on the selected regions on the input shape and the reference shape, we denote their coordinates in the two feature volumes as \(\Gamma_{\text{target}}\) and \(\Gamma_{\text{ref}}\), respectively. The goal is to replace the feature values in the selected input region with the corresponding feature values from the reference neural volume. Therefore, we resize \(F_0^{\mathrm{ref}}[\Gamma_{\mathrm{ref}}]\) by interpolation to match the size of the selected input region. We then set \(\Gamma=\Gamma_{\mathrm{target}}\) and use the resized reference features as \(V\).

• \(\mathcal{L}_{\text{pdrag}}\). Given source point \(A\) and target point \(B\), we progressively copy-paste features around \(A\) along a linear path toward \(B\). Thus, unlike the other operators, drag is performed over multiple iterations. Inspired by [4]–[6], each iteration uses two steps to define \(\Gamma\) and \(V\), i.e., motion supervision and point tracking. In the motion supervision step, at the \(k\)-th iteration, we denote \(P_k\) as the source point (initially \(P_0=A\)) and \(\Gamma(P_k, r_1)\) as the local neighborhood around \(P_k\), where \(r_1\) is a radius parameter. Letting \(u\) denote the unit vector from \(P_k\) to \(B\), we translate the local neighborhood around \(P_k\) by vector \(u\) to locate the target region \(\Gamma=\{p+u|p\in \Gamma(P_k, r_1) \}\) and set \(V=F_{k}[\Gamma(P_k, r_1)]\) as the target values. With \(\Gamma\) and \(V\), we define objective \(\mathcal{L}_{\text{pdrag}}\) for performing the co-optimization at the \(k\)-th iteration. In the point tracking step, after the \(k\)-th iteration, we update position \(P_k\) to \(P_{k+1}\). If the source point is not tracked accurately, the next motion supervision step will be guided by an incorrect position, resulting in undesired outputs. Here, we search for \(P_{k+1}\) within a radius parameter \(r_2\) around \(P_k\), i.e., \(\Gamma(P_k, r_2)\), such that its features are most similar to those around source point \(A\) in the original neural volume \(F_0\): \[\label{eq:drag95track} P_{k+1} = \underset{q \in \Gamma(P_k, r_2)}{\text{argmin}} | F_{k+1}[q] - F_0[A] |_1.\tag{2}\]

  

• \(\mathcal{L}_{\text{rdrag}}\). Given a selected region \(\mathcal{R}\) and representative source-target point pairs \(\{(A_j,B_j)\}_{j=1}^{J}\) with \(A_j \in \mathcal{R}\), our goal is to drag the entire selected region according to these sparse representative point pairs. However, applying point-wise drag only to these representative pairs may under-constrain the larger region, while assigning drag to every spatial coordinate within \(\mathcal{R}\) would be redundant. Therefore, we construct a sparse set of support points for region-wise drag. Specifically, we uniformly downsample the dense coordinates within the selected region \(\mathcal{R}\) using a controllable stride and take the downsampled coordinates as support source points. We further include all user-specified source points, obtaining the support set \(\mathcal{H}=\{{S_l}|{l=1}, ..., {L}\}\). We then compute the displacement specified by each representative pair as \(\Delta_j=B_j-A_j\). For each support source point \(S_l\), we compute its distance to each representative source point: \[d_{lj}=\|S_l-A_j\|_2.\] The weights are then computed as follows: \[w_{lj} = \frac{\exp(-d_{lj})}{\sum_{m=1}^{J}\exp(-d_{lm})}.\] The displacement of \(S_l\) is computed as the weighted average of the representative displacements: \[\Delta_l=\sum_{j=1}^{J} w_{lj}\Delta_j,\] and its target point is set as \(T_l=S_l+\Delta_l\). If \(S_l\) coincides with a user-specified source point \(A_j\), we directly set \(T_l=B_j\). This gives the set of support source-target pairs \[\mathcal{D}_{\mathcal{R}} = \{(S_l,T_l)\}_{l=1}^{L}.\]

  Once these support point pairs are determined, each pair independently follows the full point-wise drag procedure defined in \(\mathcal{L}_{\text{pdrag}}\). Accordingly, the region-wise drag objective is formulated as \[\mathcal{L}_{\text{rdrag}} =\frac{1}{L}\sum_{l=1}^{L} \mathcal{L}_{\text{pdrag}}(S_l,T_l).\]

4.5 Region-wise Control↩︎

Although \(\mathcal{L}_{\text{op}}\) is defined locally, optimizing the global latent code \(z\) can unintentionally change unedited regions; see Figure 7 (c). To preserve the unedited regions, we assume a binary mask \(\mathcal{M}\) provided by the user to indicate the region to be preserved (\(\mathcal{M}=0\) for the preserved region and \(\mathcal{M}=1\) for the editable region), and introduce two complementary region-wise control mechanisms: a latent feature regularization that constrains the CNS optimization itself, and a KV-cache replacement strategy for transformer-based 3D foundation models that keeps the preserved region fixed while decoding the optimized CNS representation.

Latent Feature Regularization. To keep the unedited region unchanged, we introduce a feature regularization loss. Specifically, this loss penalizes the deviation between the current feature volume \(F_k\) and the initial feature volume \(F_0\) within the preserved region: \[\label{eq:region95reg} \mathcal{L}_{\text{reg}} = {\left\|(1-\mathcal{M})\odot(F_k-F_0)\right\|_1}\tag{3}\] The full objective for CNS optimization is \[\label{eq:region95total} \mathcal{L} = \mathcal{L}_{\text{op}} + \lambda_{\text{reg}}\,\mathcal{L}_{\text{reg}},\tag{4}\] where \(\lambda_{\text{reg}}\) balances the editing objective and region preservation.

KV-cache Replacement. For transformer-based 3D foundation models, motivated by [5], we introduce a KV-cache replacement strategy to preserve the unedited region. During the inversion-guided neural volume construction of Section 4.1, we invert the input shape with the second-order inversion scheme and cache the key and value tensors at every self-attention layer \(l\) and timestep \(\tau\) along the original trajectory, denoted as \(K_{\mathrm{ori},\tau}^{l}\) and \(V_{\mathrm{ori},\tau}^{l}\).

After CNS co-optimization, we decode the optimized CNS representation to obtain the edited shape. At each timestep \(\tau\) and self-attention layer \(l\), we replace the current key and value tensors outside the editable region with the cached ones from the original trajectory: \[\label{eq:kv95replace} \begin{align} \bar{K}_{\tau}^{l} &= \mathcal{M}\odot K_{\tau}^{l} + (1-\mathcal{M})\odot K_{\mathrm{ori},\tau}^{l}, \\ \bar{V}_{\tau}^{l} &= \mathcal{M}\odot V_{\tau}^{l} + (1-\mathcal{M})\odot V_{\mathrm{ori},\tau}^{l}, \end{align}\tag{5}\] where \(K_{\tau}^{l}\) and \(V_{\tau}^{l}\) are the key and value tensors decoded from the optimized CNS representation. The query tensor \(Q_{\tau}^{l}\) is left unchanged, and self-attention is computed using \(Q_{\tau}^{l}\) together with the replaced key-value tensors \(\bar{K}_{\tau}^{l}\) and \(\bar{V}_{\tau}^{l}\). In this way, the editable region uses the newly computed key-value tensors, while the preserved region reuses the cached ones from the original trajectory. This encourages the preserved tokens to remain consistent with the input shape during decoding, while still allowing the current queries to adapt to the edited CNS representation.

In summary, \(\mathcal{L}_{\text{reg}}\) directly preserves the spatial features of unedited regions at the neural feature volume level, whereas KV-cache replacement constrains feature propagation inside the generator during decoding; see Figure 7.

5 Results and Experiments↩︎

5.1 Dataset and Implementation Details↩︎

We evaluate our method on the following datasets: ShapeNet [127], ABC [128], SMPL [129], and Objaverse [100]. For the 3D inversion model [13], TRELLIS [14], and Direct3D-S2 [15], the neural feature volume \(F\) is constructed from the output features of their 12th, 21st, and 21st layers, respectively. We extract these features at \(t=200\) for the diffusion-based 3D inversion model, and at \(t=0.7\) for TRELLIS and Direct3D-S2. For CNS optimization, we use Adam with learning rates of \(3\times 10^{-2}\), \(2\times 10^{-3}\), and \(2\times 10^{-3}\) for the three backbones, respectively. Their inference processes take around 1 minute, 30 seconds, and 20 seconds, respectively, while CNS optimization takes less than 10 seconds per operator for all three backbones. For the point-wise drag operator, we set \((r_1,r_2)=(1,2)\) for all three backbones and stop the optimization once the source point reaches the target point. For the other four operators, optimization is stopped when the loss drops below one-third of its initial value. Please refer to supplementary material Section A for more details about the different backbones. We will release code and data upon the publication of this work.

3.2pt

5.2 Experiment Settings↩︎

We develop six editing operators: copy, delete, resize, mix, point-wise drag, and region-wise drag. However, existing methods have largely overlooked the last four operators. Therefore, our main evaluation focuses on comparing the point-wise drag operator and delete operator with existing methods. We conduct evaluations in two settings. First, in the category-specific CNS-Edit setting, we compare our method with four category-specific shape-editing methods. Among them, DualSDF [9], SPAGHETTI [124], and SLIDE [125] couple 3D shapes with corresponding coarse geometric primitives, so shape editings are performed by moving these primitives. In addition, we compare with DeepMetaHandles [35], which uses a deformation network to predict vertex offsets for shape editing. To facilitate comparison, we follow [12], [130] and construct a dataset of 50 editing cases, consisting of 25 chairs and 25 airplanes from ShapeNet [127]. Second, in the category-agnostic CNS-Edit++ setting, we build two benchmarks by sampling shapes from diverse object categories in Objaverse [100] to evaluate category-agnostic editing. For point-wise drag, we construct 50 editing cases from 20 different categories and compare with APAP [119], MVDrag3D [120], and DragFlow [126]. APAP and MVDrag3D use 2D diffusion priors for general 3D editing, whereas DragFlow is adapted by first editing the rendered image and then reconstructing the edited image into 3D with TRELLIS. For delete, we construct another 50 editing cases from 20 different categories and compare with Tailor3D [46], Instant3dit [47], and VoxHammer [45]. Please refer to the supplementary material Section B for details on dataset construction and comparison settings.

5.3 Quantitative Comparison↩︎

Evaluation metrics First, we employ the Frechet Inception Distance (FID) and Kernel Inception Distance (KID) to evaluate the visual quality of shapes generated by different methods. Assessing whether a method produces shapes that match the user’s operation is challenging. To this end, we conduct a user study to evaluate fulfillment of the user operation. Following [12], [130], we invite 10 participants to evaluate the edited shapes from two aspects: the Quality Score (QS), which measures the visual appeal of the shapes, and the Matching Score (MS), which measures how well they align with the user operation. For each edited shape, each participant is asked to assign QS and MS ratings from 1 (worst) to 5 (best).

Table 1 reports the results of our method built on the category-specific 3D inversion backbone [13], showing that it outperforms existing category-specific shape-editing methods. Moreover, Tables 2 and 3 further validate our method on different 3D foundation models, including Direct3D-S2-image, TRELLIS-text, and TRELLIS-image. Specifically, Table 2 reports the quantitative results for the point-wise drag operator, while Table 3 reports those for the delete operator. The consistent improvements over state-of-the-art baselines demonstrate the effectiveness and generality of our method.

5.4 Qualitative Comparison↩︎

Figure 8 presents visual comparisons in the category-specific CNS-Edit setting, where our method is built on the category-specific 3D inversion backbone [13] and compared with existing category-specific baselines. By leveraging the coupled neural shape representation, our method can perform shape editing with semantics, as shown by the airplane example in the first row of Figure 8. When the left wing of the airplanes is shortened (top two examples), the right wing is also shortened automatically, indicating that our method incorporates shape semantics, e.g., symmetry, into the editing process. In contrast, other baseline methods often produce implausible shapes or fail to accurately follow the specified edit.

Figure 9 shows visual comparisons for the point-wise drag operation in the CNS-Edit++ setting, with TRELLIS-image used as a representative backbone. Compared with state-of-the-art category-agnostic 3D drag-editing baselines, our method more faithfully follows the user-specified drag operation while preserving the fidelity and plausibility of the edited shapes. Figure 10 shows visual comparisons for the delete operation in the CNS-Edit++ setting, also using TRELLIS-image as a representative backbone. Compared with state-of-the-art 3D editing baselines that support part removal, our method removes the selected region more cleanly while better preserving the geometry of unedited regions.

Figure 24 further shows that our method can be built on different 3D foundation-model backbones, including TRELLIS-text, TRELLIS-image, and Direct3D-S2-image. The results indicate that our method remains effective across different backbone architectures and input conditions.

5.5 More Visual Results for Different Operators↩︎

Figures 18, 15, 13, and 11 provide additional visual results of our proposed operators in the category-specific CNS-Edit setting, where our method is built on the category-specific 3D inversion backbone [13]. We also show examples of the cut-paste operator, which combines the copy and delete operators, in Figure 17. Beyond this category-specific setting, Figures 12, 21, 16, 20, 14, and  19 present additional visual results in the category-agnostic CNS-Edit++ setting. These results show that the strong generative priors of 3D foundation models enable our method to edit a wider range of shape categories. For more visual results and failure cases, please refer to Supplementary Materials Section D, E, F, G, H, and I.

5.6 Ablation Study↩︎

We conduct ablation studies under two settings, targeting the construction of the neural feature volume and the instantiation of CNS on 3D foundation models, respectively.

Neural Feature Volume Construction. We first ablate the construction of the neural feature volume \(F\) using the 3D inversion backbone [13]. We analyze how features from different layers of the U-Net (16 layers in total) affect the neural volume \(F\). In our setting, we use the features output from the 12th layer (denoted as \(J=12\)) to construct \(F\). For ablation, we use features from shallower layers (\(J=15\)) and deeper layers (\(J=9\)). Features from shallow layers (\(J=15\)) provide rich spatial context, but they contain limited shape semantics; see Figure 23 (e). Using features from shallow layers allows the results to match the editing operation, but the quality of the edited shapes is reduced because of the lack of shape semantics. Features from deeper layers (\(J=9\)) encode more abstract semantics but do not provide sufficient spatial context, which reduces editing controllability. As shown in Figure 23 (c), deeper features (\(J=9\)) produce only limited edits, suggesting that deeper features lack enough spatial context for controllable editing.

Next, we perform another ablation by applying our operators directly in the spatial domain, i.e., wavelet volume. Specifically, we bypass CNS co-optimization and directly assign the target values in the wavelet volume. For the copy operator, we replace the values in the target pasted region with those from the corresponding source region. For the delete operator, we replace the values in the selected region with those from an empty region. As shown in Figures 23 (i) and (m), directly applying operators in the spatial domain introduces artifacts. Furthermore, as Figure 23 (m) shows, it also lacks an understanding of shape semantics during editing. In contrast, our full pipeline produces high-fidelity edits while preserving shape semantics; see Figures 23 (h) and (l).

CNS Instantiation on 3D Foundation Models. We then ablate the key components for instantiating CNS on 3D foundation models. Specifically, we evaluate three components: the second-order inversion scheme used for inversion-guided neural volume construction, KV-cache replacement, and latent feature regularization. Table 4 reports the quantitative ablation results. Removing any of the three components consistently degrades performance, indicating that second-order inversion, KV-cache replacement, and latent feature regularization are all important to our framework. Figure 22 further provides qualitative comparisons. Without second-order inversion, the edited shape becomes less faithful to the input shape; see Figure 22 (c). Removing either KV-cache replacement or latent feature regularization can cause the edit to affect unedited regions; see Figure 22 (d) and (e). In contrast, our full model achieves faithful edits while better preserving unedited regions.

5.7 Limitations and Future Work↩︎

Although our method enables high-fidelity 3D shape editing with various operators, it still has several limitations. First, our method currently works purely at the geometric level. A valuable future direction is to extend our framework to editing textured meshes, where texture is updated jointly with geometry so that the two remain consistent. For example, when a part is dragged or copied, its texture should be dragged and copied together with the geometry.

Second, our editing operators do not consider object functionality. An interesting direction is to incorporate functional and physical constraints, together with semantic priors, to ensure that edited shapes remain both geometrically plausible and functionally valid.

Third, our current framework mainly focuses on object-level shape editing. A promising direction is to extend it to larger-scale and dynamic settings, including scene-level [131], [132], city-level [133], and 4D mesh-sequence editing [134], while incorporating agent-based frameworks [135] to support more flexible and automated editing.

Lastly, although CNS optimization itself is efficient and takes less than 10 seconds per operator, the overall editing time is still limited by the decoding speed of the underlying generative backbone. In our experiments, generating the edited shape takes about 1 minute with the diffusion-based 3D inversion model [13], 30 seconds with TRELLIS [14], and 20 seconds with Direct3D-S2 [15]. A future direction is to develop acceleration strategies for the decoding process.

6 Conclusion↩︎

We presented CNS-Edit++, a category-agnostic 3D shape editing framework based on coupled neural shape optimization. Our method introduces a coupled neural shape (CNS) representation, consisting of a global latent code \(z\) and a neural feature volume \(F\). Given a specific operator, we convert it into an objective function defined on the neural feature volume and co-optimize \(z\) and \(F\) to obtain the edited shape. We further generalized CNS from a category-specific 3D inversion model to category-agnostic 3D foundation models. We also developed a family of editing operators, including copy, resize, delete, mix, point-wise drag, and region-wise drag, together with region-wise control mechanisms based on KV-cache replacement and latent feature regularization to preserve unedited regions. Extensive experiments across different 3D generative backbones demonstrate that our method produces high-fidelity editing results that better follow specified operators while preserving shape semantics and unedited regions, outperforming existing works.

References↩︎