1 Introduction↩︎

Articulated household objects are ubiquitous in daily environments and serve as fundamental building blocks for interactive virtual worlds, robotics, and embodied AI [1], [2]. Automatically creating high-quality articulated assets at scale remains challenging: expert authoring is costly, and data capture in real settings is often constrained. As a result, increasing attention has been devoted to developing automatic methods for articulated object modeling.

Existing methods fall into two camps: one line of work focuses on optimization-based reconstruction, assumes multi-image or video inputs [3]–[6] are available or synthesizes multi-images or video [7]–[9] from a single input image. Such approaches inject motion cues and connectivity information to recover articulated behavior, but they often rely on heavy optimization. They also require careful cross-view or cross-state alignment, or explicit capture of the object in motion, which is not always feasible in practice and limits throughput and scalability to diverse real-world objects. Other approaches [10], [11] regress kinematic parameters directly using trainable feedforward networks trained on existing datasets. While they can produce plausible geometry and kinematics from a resting-state image, they exhibit clear performance degradation when applied to structurally complex or visually ambiguous objects. Since a single static image provides no explicit articulation cues, the absence of dynamic information fundamentally limits identifiability.

Beyond methodological constraints, progress is bottlenecked by the scarcity of high-quality training data. Synthetic datasets like PartNet-Mobility (PM) [12], [13] lack photorealism, often leading to poor real-world generalization. Conversely, realistic datasets like Articulated Containers Dataset (ACD) [14], [15] are too small for large-scale training and are better suited for evaluation. Since acquiring kinematic annotations for real-world objects is prohibitively expensive, exploring methods that can leverage unannotated visual data for training is of great significance. For such unannotated data, we can harness off-the-shelf image and video foundation models to generate or enhance visual priors, easily scaling up diverse, high-quality datasets without relying on costly manual labels.

Motivated by the challenges of single-image articulated generation on real data, we argue that prior work [10], [11] largely neglects part-level appearance dynamics and instead focuses on kinematic parameters. Our key insight is that visual dynamics and kinematics are inherently inter-dependent: observed motions constrain feasible joint types and axes, while kinematic parameters explain motion patterns. We therefore introduce PWM-ArtGen, a generative model that learns the joint distribution of part-level visual dynamics and kinematic parameters. By decoupling diffusion timesteps, our approach facilitates visual-branch co-training on unannotated data, thereby capturing complex visual dynamics without the need for manual labels.

Also, we introduce a Visual Dynamics Regularizer (VDR) that explicitly promotes faithful part-level dynamics, thereby limiting dependence on static cues present in a single image. To support co-training, we curate PartNet-Mobility-Reality (PM-R), a photorealistic dataset of 19.7k paired images. Concretely, we apply a frozen image-to-image editing model to PartNet-Mobility renders to enhance materials, lighting, and backgrounds. In summary, our main contributions are as follows:

• PWM-ArtGen: We propose a single-image part world model that jointly learns visual dynamics and kinematics, enabling controllable 3D articulated generation and seamless co-training on unannotated data.

• Visual Dynamics Regularizer (VDR): We introduce a regularization mechanism that enforces consistency between visual motion cues and articulated behaviors, significantly improving kinematic reasoning.

• PM-R Dataset: We construct a photorealistic dataset of 19.7k image pairs with part-level dynamics, which unlocks scalable co-training and effectively bridges the synthetic-to-real gap.

2 Related Work↩︎

2.1 Articulated object creation↩︎

Reconstruction-based Methods. To advance robotic simulation and digital twins, the rapid generation of articulated object assets from images has garnered significant interest. Forming the foundation of this domain, initial works [12]–[19] largely depended on manual annotation to construct extensive datasets. One prominent line of work recovers articulated behavior through per-object optimization. Early approaches in this domain learn deformable signed distance fields [20] or reconstruct parts from point clouds captured across multiple object states [21]–[23]. To inject essential motion cues and connectivity information, subsequent methods rely heavily on multi-view imagery, stereo captures [24], or video inputs [3]–[6], [25], [26]. Recent advancements even attempt to synthesize these multi-view or video inputs from a single image to guide the optimization process [7]–[9]. However, these approaches consistently suffer from low optimization speed, fixed-category assumptions, and a strict dependence on dense cross-view or cross-state alignment. This limits their practical throughput and scalability to diverse, in-the-wild objects.

Generation-based Methods. Another line of work [10], [11], [27], [28] synthesizes articulated objects from a single image by assembling or adapting parts retrieved from pre-constructed asset collections. Although such retrieval-based paradigms benefit from realistic part priors, the absence of dynamic cues fundamentally limits their ability to infer complex real-world articulations. Recent work [29] begins to incorporate motion-related observations, but it relies on dual-image input and therefore cannot be directly applied to the more challenging single-image setting. Beyond retrieval-based generation, several methods [30], [31] attempt to synthesize articulated geometry through implicit representations such as generated SDFs followed by surface reconstruction. However, the reconstruction process can introduce artifacts and often struggles to preserve fine-grained geometric details. Recent advances in 3D foundation models have further enabled generative articulated object modeling [32], [33], yet existing solutions are still primarily demonstrated on relatively simple articulation patterns.

Unlike prior single-image methods that focus solely on static geometry or direct parameter regression and thereby overlook dynamic information, we argue that visual dynamics provide crucial physical constraints for kinematic reasoning. PWM-ArtGen jointly models visual dynamics and kinematic parameters. This coupled formulation not only improves parameter estimation but also enables effective co-training on large-scale unannotated data, making our method scalable and generalizable to complex real-world objects.

2.2 World Model↩︎

Recent advancements in world models [34]–[37] have demonstrated remarkable success in learning general physical dynamics from large-scale unannotated videos, which significantly enhances the reasoning and prediction of actions. Building on this, recent approaches [38]–[40] have effectively harnessed video representations to facilitate the prediction of complex robotic trajectories and control actions. Despite these successes, the introduction of world models into the domain of articulated object generation remains largely unexplored, yet it presents a highly promising avenue for capturing intricate structural and kinematic priors. Furthermore, while existing frameworks, UWM [38], are inherently designed for control policies and cannot be directly applied to this generation task, we adapt their core principles to seamlessly fit the unique requirements of articulated object generation.

3 Part World Model↩︎

3.1 Overview↩︎

We propose a diffusion-based Part World Model that learns the joint distribution of kinematic attributes and dynamic visual representations conditioned on the input image and target articulated part. Drawing on UWM [38], we utilize independent diffusion timesteps to ingest action-free observations. This strategy allows the visual branch to extract kinematically relevant features from unannotated data, significantly enhancing kinematic estimation accuracy. We adopt an articulation-centric parameterization and introduce a visual dynamics regularizer that explicitly promotes faithful part-level dynamics.

Formally, the model learns the conditional distribution \(p(o', a \mid o, \;p^{ref})\) through a coupled noise prediction network \(s_\theta\), where \(o\) denotes the static observation, \(o'\) the post-action dynamic observation, \(a\) the kinematic attributes, and \(p^{ref}\) the target articulated part serving as structural guidance. For each input image, the model generates coherent part-level dynamics and motion parameters that are spatially organized according to \(p^{ref}\), enabling consistent reconstruction of articulated objects. Based on the model, we construct a pipeline that assembles parts into an object generation by retrieval, which is illustrated in 1.

3.2 Data Representation↩︎

We represent each articulated part with a dynamic visual, a set of kinematic attributes, and corresponding part-level structural guidance.

Dynamic visual. The post-action dynamic observation \(o'\) is a \(256 \times 256\) RGB image depicting the scene after executing the action on the reference part.

Kinematic attributes. Compared to prior work [10], [11], we expand the kinematic attributes with \(a = \{t, r, \mathbf{l}, \mathbf{d}, \mathbf{b}, \mathbf{b}^{\mathit{base}}\}\), where \(t\) denotes the joint type, \(r\) the motion range, \(\mathbf{l} \in \mathbb{R}^3\) the joint axis location, \(\mathbf{d} \in \mathbb{R}^3\) the joint axis direction, \(\mathbf{b} \in \mathbb{R}^6\) the part bounding box, and \(\mathbf{b}^{\mathit{base}} \in \mathbb{R}^6\) the base bounding box. Considering dynamic visual behavior, the articulation types addressed in this work include revolute and prismatic joints. All articulated parts of the same object share the base bounding box \(\mathbf{b}^{\mathit{base}}\), which provides a consistent reference for post-assembly in our part-level modeling framework. This design eliminates the need to predefine a maximum number of parts, as required by prior works.

Part-level structural guidance. The target articulated part \(p^{ref}\) is derived from a part mask and comprises both the cropped part image and its 2D geometric cues: \[p^{\text{ref}} = \{o^{\text{part}},\, \text{2Dbbox}^{\text{part}},\, \text{2Dbbox}^{\text{base}}\}. \label{eq:pref}\tag{1}\] \({o}^{part}\) means the cropped part observation by the part mask, and \({2Dbbox}^{{part}}\), \({2Dbbox}^{{base}}\) encode the spatial extent of the region within the image using its center coordinates \((c_x, c_y)\), height \(h\), and width \(w\). We adapt DINOv2 [43] to encode the static observation \(o\) into global visual features \(f_o\), and the cropped part observation \({o}^{part}\) into local features \(f_{o^{part}}\). To ensure scale invariance, \({2Dbbox}^{{part}}\), \({2Dbbox}^{{base}}\) are normalized as \(f_{2Dbbox^{part}}\), \(f_{2Dbbox^{base}}\) by the corresponding resolution of the original image. The resulting reference part embedding \(f^{ref}\) is defined as: \[f^{\mathrm{ref}} = \{f_{o^{\mathrm{part}}},\, f_{\mathrm{2Dbbox}^{\mathrm{part}}},\, f_{\mathrm{2Dbbox}^{\mathrm{base}}}\}. \label{eq:fref}\tag{2}\] Consequently, a training sample is parameterized by the tuple \((o', a, o, p^{ref})\), which simplifies to \((o', \emptyset, o, p^{ref})\) during the co-training phase to incorporate unannotated visual dynamics.

3.3 Model Architecture↩︎

We instantiate PWM-ArtGen as a diffusion Transformer that jointly models dynamic visual tokens and kinematic attributes, as shown in 2. The model predicts the actions and observation noises \(\epsilon_a\) and \(\epsilon_{o'}\), conditioned on static observations \(o\) and the reference part \(p^{ref}\), where \(o\) is the input image and \(p^{ref}\) specifies the target articulated part. In accordance with the representation established in  3.2, the static observation \(o\) is encoded as global visual features \(f_o\), and the reference part \(p^{ref}\) is embedded as \(f^{ref}\). The concatenation of \(f^{ref}\) with independent diffusion timesteps \(t_a\) and \(t_{o'}\) serves as the conditioning signal for the transformer blocks via adaptive layer normalization (AdaLN). For the visual branch, we operate in a latent image space by a frozen SDXL VAE and patchify the latent into \(N\) tokens of width \(D\), yielding \(o'\in\mathbb{R}^{N\times D}\). Concurrently, for the kinematic branch, the kinematic attributes are projected into a tokenized space via a trainable MLP, denoted as \(a\in\mathbb{R}^{20\times D}\) and concatenated with the visual tokens to form the input.

3.4 Visual Dynamics Regularizer↩︎

Inspired by REPA [44], we introduce a visual dynamics regularizer (VDR) to promote faithful part-level dynamics and mitigate reliance on static single-image evidence. Under joint training of action and visual branches, VDR biases the visual representations toward dynamics-consistent features that reflect action-induced motion. During training, the diffusion model produces joint hidden states \(h_{t_{o'}}, h_{t_a} = f_\theta(z_t)\) at timestep \(t\), where \(z_t = (a_{t_a}, o'_{t_{o'}})\) and \(h_{t_{o'}}^{[n]}\) denotes the feature of the \(n\)-th spatial patch in the visual branch. To ensure semantic consistency between the generated visual patches and the pretrained visual encoder, we apply the visual dynamics regularizer loss defined as

\[\mathcal{L}_{\mathrm{VDR}}(\theta) := -\,\mathbb{E}_{o',\, \epsilon_{o'},\, t_{o'},\, t_a} \left[ \frac{1}{N} \sum_{n=1}^{N} \operatorname{sim}\big(y_{o'}^{[n]},\, h_{t_{o'}}^{[n]}\big) \right], \label{eq:vdr95loss}\tag{3}\] where \(y_{o'}^{[n]} = f(x_{o'})\) denotes the \(n\)-th patch feature of the clean image, and \(\operatorname{sim}(\cdot,\cdot)\) is the cosine similarity function. This aligns the denoising trajectory with the semantic manifold of clean data, promoting robustness to noise while preserving essential visual structure. The network in 2 consists of \(L\) layers in total, and only the visual hidden states from the first \(l\) layers are aligned with the DINOv2 patch-wise features of the post-action observation \(o'\) as shown in 3 .

3.5 Co-Training↩︎

PWM-ArtGen learns a joint diffusion process over kinematic parameters \(a\) and post-action dynamic observation \(o'\) conditioned on static observation \(o\) and target articulated part \(p^{\mathrm{ref}}\), enabling the model to incorporate part-level structural guidance during both training and inference. Specifically, we train a coupled noise prediction network [45] \(s_\theta(o, p^{\mathrm{ref}}, a_{t_a}, o'_{t_{o'}}, t_a, t_{o'})\) that approximates the conditional expectation of action and observation noise.

To leverage both action-annotated and action-free data, we adopt a co-training strategy. Let \(\mathcal{D}_{\mathrm{paired}}\) denote the dataset with ground-truth actions and \(\mathcal{D}_{\mathrm{free}}\) denote the dataset containing only observations. For action-free samples, we impute the missing actions with pure Gaussian noise by fixing the action diffusion timestep to \(T\) (, \(a_T = \epsilon_a \sim \mathcal{N}(0, 1)\)). The corresponding denoising objective is formulated as follows:

\[\begin{align} \mathcal{L}_{\epsilon_a,\, \epsilon_{o'}}(\theta) =\, &\mathbb{E}_{\substack{ (o,\, p^{\mathrm{ref}},\, a,\, o') \sim \mathcal{D}_{\mathrm{paired}} \\ t_a,\, t_{o'} \sim \mathcal{U}(0,T) \\ \epsilon_a,\, \epsilon_{o'} \sim \mathcal{N}(0,1) }} \Bigl[ w_a \Vert \epsilon_a^{\theta} - \epsilon_a \Vert_2^2 + w_{o'} \Vert \epsilon_{o'}^{\theta} - \epsilon_{o'} \Vert_2^2 \Bigr] \\ +\, \gamma &\mathbb{E}_{\substack{ (o,\, p^{\mathrm{ref}},\, \emptyset,\, o') \sim \mathcal{D}_{\mathrm{free}} \\ t_a = T,\, t_{o'} \sim \mathcal{U}(0,T) \\ \epsilon_a,\, \epsilon_{o'} \sim \mathcal{N}(0,1) }} \Bigl[ w_a \Vert \epsilon_a^{\theta} - \epsilon_a \Vert_2^2 + w_{o'} \Vert \epsilon_{o'}^{\theta} - \epsilon_{o'} \Vert_2^2 \Bigr], \end{align} \label{eq:joint95eps95loss}\tag{4}\]

where \[\begin{align} \epsilon_a^{\theta},\, \epsilon_{o'}^{\theta} &= s_{\theta}(o,\, p^{\mathrm{ref}},\, a_{t_a},\, o'_{t_{o'}},\, t_a,\, t_{o'}),\\ a_{t_a} &= \sqrt{\bar{\alpha}_{t_a}}\, a + \sqrt{1 - \bar{\alpha}_{t_a}}\, \epsilon_a,\\ o'_{t_{o'}} &= \sqrt{\bar{\alpha}_{t_{o'}}}\, o' + \sqrt{1 - \bar{\alpha}_{t_{o'}}}\, \epsilon_{o'}. \end{align}\]

Here, \(w_a\) and \(w_{o'}\) are weights chosen to trade off between the action prediction and image prediction objectives, and \(\gamma\) is a hyperparameter that balances the contribution of the action-free data. For any time step \(t \in \{1, \dots, T\}\), we define \(\alpha_t = 1 - \beta_t\) and \(\bar{\alpha}_t = \prod_{s=1}^t \alpha_s\), where \(\{\beta_s\}_{s=1}^T\) is a fixed variance schedule as in DDPM [45]. The timesteps \(t_a\) and \(t_{o'}\) are sampled independently, meaning \(\bar{\alpha}_{t_a}\) and \(\bar{\alpha}_{t_{o'}}\) correspond to distinct noise levels for the action and observation diffusion processes, respectively. This conditioning allows our model to align motion dynamics and visual appearance with part-level spatial context, providing a structured prior for generating coherent articulated behaviors.

The final training objective combines the co-training loss with the visual dynamics regularization (VDR): \[\mathcal{L} = \mathcal{L}_{\epsilon_a,\, \epsilon_{o'}} + \lambda \mathcal{L}_{\mathrm{VDR}}, \label{eq:final95loss}\tag{5}\] where \(\lambda\) balances VDR loss with diffusion reconstruction accuracy. This joint training encourages articulation-aware dynamics while maintaining semantic coherence in the generated visual space.

3.6 Synthetic-Real Mixed Training Strategy↩︎

To enhance the model’s generalization ability on real-world images, we construct the PM-R dataset as illustrated in 3. During co-training, we employ a synthetic–real mixed strategy using a length-proportional interleaving sampler that visits all samples once per epoch while mixing the two domains according to their relative sizes. At each epoch, the PartNet-Mobility-Reality and PartNet-Mobility subsets are independently shuffled, and a weighted round-robin procedure generates a global index order that maintains a consistent real–synthetic ratio. The sampler is DDP-compatible and partitions the sequence across workers without overlap, providing iteration-level balanced mixing. During training, the model jointly learns to predict both the action \(a\) and the post-action observation \(o'\), enabling a unified understanding of motion and visual dynamics.

4 Experiments↩︎

We evaluate reconstruction quality and part connectivity graph prediction on both the ACD and PartNet-Mobility test sets. Qualitative results on ACD are conditioned on the ground-truth part graph. Furthermore, we comprehensively evaluate the visual dynamics of the image branch through both quantitative metrics and qualitative visualizations on the PartNet-Mobility test set. Ablation studies validate the contributions of key components: the image branch (IB) for joint modeling, the Visual Dynamics Regularizer (VDR), and co-training (Co-T) on PartNet-Mobility-Reality (PM-R) dataset. Additional qualitative results and experiments are provided in the Supplementary Material.

4.1 Experimental Setup↩︎

Datasets. Following [11], we collect data from the PartNet-Mobility [12] dataset to train our model, and from the ACD dataset [14] for additional evaluation. Our training set consists of 473 articulated objects (985 movable parts in total) from PartNet-Mobility. Each object is rendered using BLENDER_EEVEE_NEXT under both the rest and open states, with 20 images per state sampled from random viewpoints within a \(90^{\circ}\) horizontal and \(90^{\circ}\) upward range relative to the object’s front. We extract part-level masks for the rest state and apply realistic augmentation strategies to construct PM-R. In total, we obtain about 19.7K rendered samples of PM for training and 19.7k PM-R image pairs without kinematic annotations for co-training. For testing, we use 77 unseen objects (155 parts) from PartNet-Mobility as the test split, each rendered from two random views in both states, yielding 310 test samples. To further evaluate the model’s generalization capability on realistic data, we include 135 objects (455 parts) from the ACD dataset in a zero-shot setting, producing 910 additional samples that better reflect real-world scenarios. To evaluate the visual branch, we render ten random views in both states from the PM test split, yielding 1550 test samples. Please refer to the Supplementary Material for more details on rendering configurations.

Compared methods. We select some representative methods: URDFormer [27], NAP [30], SINGAPO [11], and Articulate-Anything [28] for comparison of kinematic parameters; DragAPart [7], and Puppet-Master [8] for comparison of visual dynamics. Due to the limited complexity of parts in the PartNet-Mobility dataset, these methods would suffer significant performance degradation, particularly in zero-shot scenarios, if not trained on larger datasets with more diverse part configurations, as reported in SINGAPO [11]. In contrast, our method is specifically designed for part-level modeling and is thus largely unaffected by such dataset limitations. To ensure a fair comparison, we evaluated SINGAPO using its released checkpoint trained on 3,063 objects (approximately six times larger than PartNet-Mobility) and adopted the results of URDFormer and NAP-ICA reported in [11], where both methods were strictly adapted to the SINGAPO framework based on [27], [30]. Furthermore, we evaluate Articulate-Anything [28] using GPT-4o [42], and directly utilize the officially released checkpoints for the visual dynamics baselines, DragAPart [7] and Puppet-Master [8].

Metrics. To evaluate reconstruction quality and articulation correctness, we adopt four quantitative metrics: (1) \({d_{gIoU}~\downarrow}\), the generalized IoU between predicted and ground-truth part bounding boxes; (2) \({d_{cDist}~\downarrow}\), the Euclidean distance between the centers of corresponding parts; (3) \({d_{CD}~\downarrow}\), the Chamfer Distance between predicted and ground-truth meshes; and (4) \({Acc~\uparrow}\), the accuracy of the predicted articulation graph. In addition, we report the average overlapping ratio \({AOR~\downarrow}\) introduced by CAGE [10] to detect unrealistic collisions among sibling parts in the articulated graph. All metrics are computed over both resting and articulated states. For clarity, we prefix the metric names with RS-(resting state) and AS-(articulated state) in the tables to indicate the corresponding evaluation state. We evaluate all methods under two settings: with and without access to the ground-truth part connectivity graphs and masks. Following [7], we evaluate the generated visual dynamics using standard image quality metrics, including pixel-level PSNR, patch-level SSIM, and feature-level LPIPS. We utilize ground-truth drag or part mask as a conditioning signal.

4.2 Implementation Details↩︎

We train PWM-ArtGen for 200k steps on 4\(\times\)RTX 4090 GPUs with a global batch size of 128. The diffusion transformer features 12 layers, 12 attention heads, a hidden dimension of 768, an MLP ratio of 4, \(2\times2\) patch embeddings, and 512-dimensional timestep embeddings. Optimization uses AdamW (learning rate \(1\times10^{-4}\), weight decay \(1\times10^{-6}\), \(\beta=(0.9, 0.999)\), \(\epsilon=1\times10^{-8}\)). Inference relies on 10-step DDIM sampling [45], [47]. For the training objectives, we set the action and image prediction weights to \(w_a = w_{o'} = 1.0\), and the action-free data weight to \(\gamma = 1.0\). To facilitate early visual-kinematic alignment, we apply the Visual Dynamics Regularizer (VDR) only during the first 80k training steps, with \(\lambda = 0.5\), utilizing spatial features from a frozen DINOv2-Reg/14 encoder [43]. Additional details are in the Supplementary Material.

4.3 Comparison with the State-of-the-art Methods↩︎

Quantitative Results On Kinematic. We report quantitative results on the ACD and PartNet-Mobility datasets in ¿tbl:tab:acd? and ¿tbl:tab:pm?, respectively. For fair comparison, we use the same part connection graph predicted by GPT-4o [42]. All diffusion-based generative methods are evaluated five times per test sample, and the averaged results are reported to mitigate stochastic variations.

On the ACD test set, which contains more realistic and diverse articulated objects, our method achieves clearly superior performance across all metrics. Compared to existing baselines, PWM-ArtGen delivers the lowest reconstruction errors, demonstrating strong generalization capability to real-world-like data. Such consistent superiority on ACD highlights our model’s robustness under domain shift, which is a key challenge for articulated object generation.

8pt

On the PartNet-Mobility test set, when the ground-truth graph and masks are provided, PWM-ArtGen performs on par or better than the strongest baseline across most metrics, with only marginal differences in a few cases. In the setting without ground-truth graph and masks, our method still outperforms competing approaches on several measures, though it exhibits slightly higher \({d_{cDist}}\) and \({d_{CD}}\) values. This minor degradation is mainly attributed to the current limitations of our Part Mask Generator, especially on synthetic datasets such as PM, where rendering artifacts and imperfect geometry annotations often lead to suboptimal mask quality. In contrast, the ACD dataset provides relatively cleaner and more realistic object geometry, resulting in more reliable part-level supervision. It is worth noting that improving the segmentation quality of the Part Mask Generator is not the primary focus of this work, but this limitation can be effectively addressed in future research through more advanced segmentation pipelines. Overall, PWM-ArtGen exhibits strong reconstruction accuracy, reliable articulation prediction, and remarkable generalization to realistic data, which is one of the most challenging aspects of articulated generation.

Qualitative Comparison On Kinematic. As illustrated in 4, PWM-ArtGen generates more consistent and physically plausible part motions than others. Our method preserves structural integrity during articulation and better aligns part boundaries, demonstrating improved understanding of 3D part relationships. The improvement is especially evident on the ACD dataset, where the objects exhibit complex materials and realistic lighting conditions, highlighting the strong generalization ability of PWM-ArtGen to real-world scenarios.

Comparison On visual. As illustrated in 5, our model produces qualitative results comparable to Puppet-Master, while trailing slightly behind DragAPart. However, this gap is well-justified by the significant efficiency gains detailed in 1: our model achieves nearly 10\(\times\) faster inference speed with only \(0.25\times\) the parameters of DragAPart. These results indicate that our dynamics branch can generate visually plausible outputs. Furthermore, subsequent ablation studies demonstrate that explicit dynamics modeling significantly enhances the accuracy of kinematic estimation.

4.4 Ablation Study↩︎

We conduct detailed ablation studies to verify the effectiveness of each key component in PWM-ArtGen, including the Image Branch (IB) for joint modeling of visual dynamics and kinematics, the Visual Dynamics Regularizer (VDR), and the co-training (Co-T) on the PartNet-Mobility-Reality (PM-R) dataset. We construct several variants by selectively altering these components. The quantitative results are summarized in 2 and 3.

Effectiveness of Image Branch. We conduct ablation experiments to assess the contribution of jointly models visual dynamics and kinematic parameters. As shown in 2, modeling dynamics provides a strong inductive bias for kinematic estimation, leads to a noticeable degradation across all metrics rather than only action branch. Without the guidance of visual cues, the model struggles to infer motion directions and part boundaries accurately.

The Impact of co-training. Ablation results in 2 and 3 demonstrate that omitting co-training on PM-R severely degrades performance under complex articulated states. Since PM-R closely aligns with real-world data distributions, its inclusion during co-training effectively bridges the synthetic-to-real gap. It enhances the model’s ability to handle near-real inputs from ACD, leading to more precise kinematic estimation.

Analysis of Visual Dynamics Regularizer. As shown in 2 and 3, removing the VDR compromises part-wise coherence. This module imposes a critical inductive bias that encourages consistent motion and smoother surface deformations, thereby bridging the gap between static geometry and dynamic articulation.

5 Conclusion↩︎

In this work, we presented PWM-ArtGen, a unified Diffusion Transformer framework for generating articulated objects from a single image. By jointly modeling part-level visual dynamics and kinematic parameters through decoupled diffusion timesteps, our approach achieves a strong inductive bias for kinematic estimation without predefining part counts. Co-training on the PM-R dataset effectively leverages action-free visual priors and bridges the synthetic-to-real gap, while the Visual Dynamics Regularizer (VDR) ensures part-wise coherence and physically consistent motion. PWM-ArtGen achieves state-of-the-art performance on ACD and competitive results on PartNet-Mobility, demonstrating robust zero-shot generalization.

Acknowledgements↩︎

This work was supported by the National Key Research and Development Program of China (2023YFA1008501).

Our supplementary materials provide comprehensive implementation details and extended experimental validation for the proposed method, which can be summarized as follows:

• Implementation details of the Part Mask Generator pipeline, including SAM-based segmentation and GPT-4o connectivity graph prediction with visualized prompts.

• Mathematical formulation of the assembly strategy, including shared base bounding box initialization and 2D-to-3D positional guidance mechanism.

• Retrieval pipeline adaptation from SINGAPO with composite scoring metrics.

• Complete training configurations for both our model and the baseline, including hyperparameters, optimization settings, and computational resources.

• Extended quantitative and qualitative results, including hyperparameter analysis, baseline ablation studies on training data scale, robustness evaluations against imperfect structural priors, comparisons with the dual-state method, visual comparisons on PartNet-Mobility, and success/failure analyses on real-world ACD and in-the-wild or internet-sourced images.

6 Implementation Details↩︎

6.1 Part Mask Generator↩︎

Part Mask Segment using SAM.
We first apply the Segment Anything Model (SAM) to the input image \(o\) in automatic mask generation mode, producing a set of candidate masks that serve as initial part proposals. To ensure semantic coherence, we merge masks with area below a threshold of 1000 pixels into their nearest larger neighbor based on centroid distance and IoU overlap. This yields a refined set of \(N\) non-redundant part masks \({m_i}_{i=1}^N\), each representing a structural component of the object. An example output of this stage is shown in 6.
Part Connectivity Graph prediction.
We leverage GPT-4o (gpt-4o-2024-08-06) in a two-step reasoning process to construct an articulated connectivity graph from the segmented parts. In the first step, GPT-4o takes the original image \(o\) as input and generates a textual description of the object’s part-level structure, identifying movable components and their joint types (e.g., door, drawer). In the second step, we overlay the image with the SAM-generated part masks, each labeled with a unique ID, and prompt GPT-4o to map the previously described parts to these IDs. The model outputs a node-to-mask correspondence, which we parse into a graph \(G = (V, E)\). Unassigned graph nodes are discarded, while unassigned masks are merged into a single base node to preserve structural completeness. The prompt templates and model responses for both steps are visualized in 7 and 8, respectively.

6.2 Assembly↩︎

Given the output of the Part Mask Generator, a part connectivity graph \(G = (V, E)\) where each node \(v_i\) is associated with a mask ID and classified. We assemble a coherent 3D articulated object using the PWM-ArtGen model. The assembly is anchored to a shared canonical coordinate frame defined by a base bounding box \(\mathbf{b}^{base}\), ensuring spatial consistency across all parts.
Shared Base Bounding Box Initialization.
We compute a shared base bounding box \(\mathbf{b}^{base}= (\mathbf{c}^{{base}}, \mathbf{s}^{{base}})\) as follows:

Let \(\mathcal{B}\) denote the set of indices corresponding to parts assigned to the base node. Collect the 3D bounding boxes \(\{\boldsymbol{\mathbf{b}_i^{base}}\}_{i \in \mathcal{B}}\) of these parts, each defined by center \(\mathbf{c}_i^{base}\) and size \(\mathbf{s}_i^{base}\), compute the average centroid and average size: \[\mathbf{c}^{base} = \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \mathbf{c}_i^{base}, \quad \mathbf{s}^{base} = \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \mathbf{s}_i^{base}.\] All parts are then transformed into the coordinate system defined by \(\mathbf{b}^{base}\).
2D-to-3D Positional Guidance for Part Alignment.
To ensure spatial consistency between the input image and the reconstructed 3D object, we align each part’s 3D position according to its relative location within the base, using the base bounding box as the reference frame. Let the 2D base bounding box be defined by its center \(\mathbf{c}^{\text{base,2D}} = (c^{\text{base,2D}}_x, c^{\text{base,2D}}_y)\) and size \(\mathbf{s}^{\text{base,2D}} = (w^{\text{base,2D}}, h^{\text{base,2D}})\), where \(w^{\text{base,2D}}\) and \(h^{\text{base,2D}}\) are the width and height in pixel units. For part \(i\), with 2D centroid \(\mathbf{c}^{\text{part,2D}}_i = (c^{\text{part,2D}}_{i,x}, c^{\text{part,2D}}_{i,y})\), we compute its normalized coordinates within the base region: \[\begin{align} \hat{c}^{\text{2D}}_{i,x} &= \frac{c^{\text{part,2D}}_{i,x} - c^{\text{base,2D}}_x}{w^{\text{base,2D}}} + 0.5, \\ \hat{c}^{\text{2D}}_{i,y} &= \frac{c^{\text{part,2D}}_{i,y} - c^{\text{base,2D}}_y}{h^{\text{base,2D}}} + 0.5, \end{align}\] where \(\hat{\mathbf{c}}^{\text{2D}}_i = \big( \hat{c}^{\text{2D}}_{i,x},\, \hat{c}^{\text{2D}}_{i,y} \big) \in [0, 1]^2\).

In 3D space, the shared base bounding box is defined by its center \(\mathbf{c}^{\text{base}} = (c^{\text{base}}_x, c^{\text{base}}_y, c^{\text{base}}_z) \in [-1, 1]^3\) and size \(\mathbf{s}^{\text{base}} = (w^{\text{base}}, h^{\text{base}}, d^{\text{base}}) \in [0, 2]^3\), both expressed in the common normalized world coordinate system.. We interpret \(\hat{\mathbf{c}}^{\text{2D}}_i\) as the desired relative position of part \(i\) inside this base volume.

The 3D center \(\mathbf{c}_i = (c_{i,x}, c_{i,y}, c_{i,z})\) of part \(i\) is then updated to reflect this layout: \[\begin{align} c_{i,x} &\gets c^{\text{base}}_x + \left( \hat{c}^{\text{2D}}_{i,x} - 0.5 \right) w^{\text{base}}, \\ c_{i,y} &\gets c^{\text{base}}_y + \left( \hat{c}^{\text{2D}}_{i,y} - 0.5 \right) h^{\text{base}}, \end{align}\] while the Z-coordinate \(c_{i,z}\) remains unchanged.

This mapping ensures that a part located at the left edge of the base in the 2D image is placed at the left boundary of the 3D base volume, and similarly for all other relative positions. The alignment is applied only when the initial assembly exhibits significant inter-part collisions (collision ratio \(> 10\%\)); otherwise, the original positions are retained to avoid disrupting a physically plausible configuration. No rotation or scaling is applied—only the XY position is adjusted to reflect the 2D spatial layout within the 3D base reference frame.

6.3 Retrieval↩︎

We adapt the mesh retrieval pipeline from SINGAPO, by replacing its Abstract Instantiation Distance (AID) with a composite score combining AS/RS-\(d_{\text{gIoU}}\) and AS/RS-\(d_{\text{cDist}}\) to select the best-matching shape from the database.

6.4 Training Details↩︎

Our model is trained for 200k steps on 4\(\times\)RTX 4090 GPUs with a global batch size of 128. The diffusion process is defined over 100 timesteps using the DDIM framework. We explicitly set \(l=8\) for VDR. For evaluation, all inference experiments are conducted on a single RTX 4090 GPU. Furthermore, the construction of our PM-R dataset took approximately one week utilizing 4\(\times\)L40 GPUs.

We include the method from Singapo as a baseline, following the training configuration reported in their work. The model is initialized with CAGE-pretrained weights, which provides a strong prior over articulated object dynamics conditioned on scene graphs; the model then modulates this prior to better align with the input image. Training proceeds for 200 epochs with a batch size of 20, sampling 16 timesteps per iteration from a 1,000-step diffusion process. Optimization uses AdamW with differential learning rates: \(5\times10^{-4}\) for the ICA module and \(5\times10^{-5}\) for the base model, with \(\beta = (0.9, 0.99)\). The learning rate is warmed up over 3 epochs from \(1\times10^{-6}\) to its peak value, followed by cosine annealing down to \(1\times10^{-5}\). The network employs 6 attention blocks, each with 4 heads and 128 hidden units. Training is conducted on 4 RTX 4090 GPUs and takes approximately 12 hours.

7 Additional Experiments and Results↩︎

7.1 Hyperparameter analysis↩︎

We provide a sensitivity analysis of the co-training weight in 9. Taking RS-\(d_{\mathrm{gIoU}}\) and AS-\(d_{\mathrm{gIoU}}\) as representative metrics, the performance remains relatively stable across different weight values. In our experiments, we set the co-training weight to 1.0 as a balanced default configuration.

7.2 Justification for Baseline Selection and Fair Comparison↩︎

As shown in ¿tbl:tab:singapo95comparison?, the released checkpoint of Singapo trained on a much larger dataset of 3,063 objects significantly outperforms variants retrained on our limited PartNet-Mobility datasets. This performance gap confirms that Singapo, as originally proposed, is highly sensitive to training data scale and part configuration diversity. To ensure a fair comparison with our method designed for part-level modeling and robust to dataset scale, we adopt the released checkpoint as the canonical baseline. This avoids conflating architectural advantages with data-scale biases, aligning with the original evaluation protocol.

7.3 Robustness to Imperfect Part Masks and Graph Predictions↩︎

To evaluate the robustness of PWM-ArtGen against imperfect structural priors, we quantify performance degradation under varying levels of noise and incomplete data (see ¿tbl:tab:singapo95comparison95reduced?).

Experimental Setup.
We simulate incomplete graphs and masks by randomly dropping up to \(d\) movable parts (\(d \in \{1, 2, 3, 4\}\)) and their corresponding masks from the Ground Truth (GT) annotations, stopping when no movable parts remain. Note that \(d=0\) denotes the perfect GT baseline. The variable \(n\) indicates the average number of remaining parts (base + movable) per object. To assess real-world applicability, we also evaluate the performance of a fully automated pipeline utilizing predictions from SAM and GPT-4o.

Metric Selection for Robustness Analysis.
In ¿tbl:tab:singapo95comparison95reduced?, we exclusively report \(d_{\text{cDist}}\) and \(d_{\text{cD}}\) to ensure mathematically consistent evaluations across varying degrees of structural corruption \(d\). Other metrics are purposefully omitted due to severe confounding factors when comparing across different part counts: (1) The collision metric (AOR) becomes paradoxically skewed: dropping more parts inherently reduces collision probability, artificially improving the score. (2) Volume-dependent metrics (\(d_{\text{gIoU}}\)) suffer from baseline volume shifts when parts are removed, rendering cross-\(d\) degradation trends ill-posed. Conversely, point-wise distances (\(d_{\text{cDist}}\) and \(d_{\text{cD}}\)) provide stable, unbiased measurements of spatial localization and geometric fidelity for the remaining parts, independent of the overall graph size.

Tolerance to Missing Parts.
As anticipated, increasing the number of dropped parts (\(d=1 \to 4\)) results in a gradual decline in reconstruction quality across both the PM and ACD datasets. Crucially, this degradation is graceful rather than catastrophic. Even under extreme information loss, such as \(d=4\) on the PM dataset where the average part count \(n\) drops precipitously from 3.01 to 1.05, our model sustains reasonable reconstruction fidelity, evidencing strong fault tolerance to incomplete structural priors.

Robustness to Automated Pipeline Noise.
When evaluated on the inputs generated by SAM + GPT-4o, PWM-ArtGen demonstrates highly competitive practical robustness across all evaluated metrics. On the PM dataset, the automated pipeline inevitably introduces a performance drop from the pristine GT baseline (\(d=0\)). However, it consistently outperforms the scenario where a single GT part is omitted (\(d=1\)) across both \(d_{\text{cDist}}\) and \(d_{\text{cD}}\) metrics. For the structurally more complex ACD dataset, the automated pipeline’s performance generally falls within the degradation bounds of missing one to two parts (\(d=1\) and \(d=2\)). Specifically, it surpasses the \(d=1\) setting in Chamfer Distance metrics (\(\text{RS-}d_{\text{cD}}\): \(0.1033\) vs.\(0.1182\)), while aligning more closely with the \(d=2\) setting on center distances (\(\text{RS-}d_{\text{cDist}}\): \(0.1480\) vs.\(0.1477\)). These results validate that our model effectively tolerates real-world prediction noise, maintaining generation quality comparable to missing only 1–2 structural parts.

These findings confirm that PWM-ArtGen does not strictly overfit to perfect annotations. Instead, it effectively synthesizes articulated structures even when conditioned on severe part omissions and noisy masks from off-the-shelf vision-language models.

7.4 Comparison with Dual-State Method↩︎

To further contextualize our single-image approach, we compare it against DIPO, a dual-state method referenced in our related work. By utilizing both rest and open state images, DIPO accesses explicit visual cues of the articulation process, thus serving as an information-rich upper bound.

Due to the lack of released data processing scripts, fully reproducing DIPO’s exact experimental conditions is practically difficult. However, relying on their reported metrics, our single-image approach achieves a graph accuracy highly comparable to DIPO’s explicit dual-state framework. This comparable level of topological completeness provides a fair, mathematically sound, and highly referential basis for evaluating volume-dependent metrics like \(d_{\text{gIoU}}\).

Under this equitable baseline, our method demonstrates highly competitive bounding box localization on the PM dataset despite lacking the second observation state. Furthermore, on the challenging out-of-domain ACD dataset, our method significantly outperforms DIPO in bounding box metrics (RS/AS-\(d_{\text{gIoU}}\)). We attribute this to our co-training strategy and Visual Dynamics Regularizer (VDR), which endow our model with robust generalization capabilities, mitigating the domain gap issues inherent in acquiring perfectly consistent multi-state real-world pairs. While DIPO retains an expected advantage in fine-grained distance metrics (\(d_{\text{cDist}}\), \(d_{\text{cD}}\)) due to its dual-view geometric constraints, our findings confirm that single-image generation can robustly approximate, and in terms of real-world localization, even surpass multi-state methods.

7.5 Qualitative Evaluation Across Diverse Datasets↩︎

We have presented qualitative comparisons on the ACD dataset, demonstrating our method’s superior generalization under complex materials and realistic lighting. We now provide complementary results on the PartNet-Mobility test set in 10 to further validate our method’s robustness in controlled, synthetic environments, where part geometry and articulation are structured, yet occlusion and texture variation pose challenges.

 11 shows additional results across three categories. The first two rows highlight successful generations from ACD inputs, demonstrating robust recovery of articulated structures under occlusion and varied materials. The third row presents failures, which often arise when retrieval matches plausible motion to an incompatible 3D base, as seen in the second input of the third row, where a seven-drawer table’s motion is consistent but the geometry fails to accommodate it. The last row shows zero-shot results on in-the-wild or internet images, confirming our method’s capability on unconstrained inputs, underscoring both its strengths and current boundaries.

References↩︎

---

1. Corresponding author.↩︎