1 Introduction↩︎

Imitation learning and visuomotor policy learning have shown remarkable success in robot manipulation, enabling policies to conduct various tasks across diverse environments [1]–[9]. However, this progress has been driven in large part by access to large-scale, diverse demonstration data, whose collection remains expensive, time-consuming, and difficult to scale. To mitigate this bottleneck, a data augmentation paradigm has emerged: rather than collecting or synthesizing more demonstrations [10], these methods typically augment a single human demonstration to many training trajectories [11]–[14]. This idea is based on the simple but powerful equivariant assumption: rigid bodies satisfy the equivariant constraint that the Euclidean distance between any two material points is invariant under motion or contact forces. With the same rigid transformation applied to the end-effector, the relative pose between it and the object is preserved, leading to a valid trajectory.

This augmentation paradigm, however, is fundamentally mismatched to deformable object manipulation [15], [16]. As illustrated in Fig. 1, we identify two core challenges that break the rigid-object recipe. State-Space Challenge(Fig. 1 (a)): (i) High degrees of freedom. For rigid bodies, a 6-DoF pose provides a sufficient state abstraction. Deformable objects, in contrast, exhibit rich, high-dimensional shape and topology variations [17], [18]. As a result, rigid transformations alone cannot meaningfully expand the valid state distribution required for deformable manipulation. (ii) Dynamic constraint. For rigid and articulated objects, valid perturbations can be constructed directly in pose or joint space via kinematic constraints. For deformable objects, however, internal constraints are dynamic, which means the deformation depends on the interaction between internal particles; therefore, naive geometric perturbations typically yield implausible shapes and discontinuous structural changes.

Trajectory-Transfer Challenge(Fig. 1 (b)): The equivariant assumption no longer holds for deformable objects: material points are not equivariant [15]. Consequently, directly applying a global rigid isometry to augment the trajectories for topological variants of deformable objects introduces two problems: (i) the grasp pose becomes misaligned with the object’s local geometry, so the end-effector can no longer grip the object correctly; and (ii) a rigid-style trajectory transfer can only translate and rotate the trajectory as a whole, and cannot capture or compensate for the object’s local deformation. These challenges suggest that effective augmentation for deformable manipulation must jointly address two problems: synthesizing physically valid deformable states and transferring demonstrations in a deformation-aware manner.

To this end, we propose DeformGen, a dynamics-based topology data augmentation framework for deformable manipulation.

Unlike prior rigid augmentation methods that are confined to SE(3) perturbations and thus only produce spatial diversity, DeformGen achieves effective topological diversity for deformable objects by jointly synthesizing physically valid deformed states and transferring demonstrations in a deformation-aware manner. Specifically, for the state-space challenge, the key insight is that physically plausible states form a constrained manifold within the high-dimensional particle state space, and naive geometric perturbations almost always fall off this manifold. Therefore, we propose Dynamic Topological Transformation to augment the state distribution by applying randomized, spatially localized forces to the object and forward-simulating the resulting dynamics to prevent leaving the valid manifold.

These augmented assets can be used both to enrich the support of training demonstrations and to broaden policy evaluation beyond the narrow state distribution covered by the original data.

In response to the trajectory-transfer challenge, DeformGen transfers source demonstrations to each augmented state via Deformation-Field Warping. We compute per-particle displacements between the source and target object states and lift them, through \(K\)-nearest-neighbor inverse-distance interpolation, into a continuous deformation field \(D(\mathbf{x})\) over the workspace. Applying \(D\) to the demonstration trajectory simultaneously re-orients the gripper pose according to local geometric changes near the grasp region and compensates the global trajectory to remain aligned with the deformed object as a whole. A single demonstration then can be reused across an entire family of deformable states without breaking contact or violating the object’s physical structure(Fig. 1 (c)).

As shown in Fig. 1 (d), we train policies on the resulting augmented dataset to improve generalization and robustness. Experiments on high-fidelity deformable manipulation benchmark [19] show that DeformGen generally improves policy learning: compared with training on the original demonstrations alone, or with rigid-style augmentation baselines, policies trained with DeformGen achieve higher success rates in most settings. These results suggest that effective augmentation for deformable manipulation requires dynamics-consistent state synthesis coupled with deformation-aware trajectory transfer, rather than rigid pose perturbation alone. The contributions of this work are three-fold:

• Formulation of demonstration augmentation for deformable manipulation that identifies physically valid deformable state synthesis as the key missing ingredient beyond rigid-style augmentation.

• A dynamics-consistent pipeline that generates topology-coherent deformable assets through localized perturbation, physics rollout, and stabilization, and synthesizes corresponding manipulation trajectories via deformation-field warping;

• Extensive empirical evidence that the resulting synthetic demonstrations significantly improve policy learning, providing gains over both no augmentation and rigid-style augmentation baselines.

2 Related Works↩︎

2.1 Data augmentation for robot manipulation↩︎

Unlike pipelines that generate demonstrations from scratch using planners [20], generative models, or learned agents [21]–[23], data augmentation expands an existing dataset through task-agnostic transformations [11], [12]. Beyond appearance-only visual perturbations [24], [25], behavioral augmentation modifies object configurations and re-solves for task-successful trajectories, typically via physics-based planning [11]–[14], [26], image/video generation models [27]–[30] and Real2sim2real [31], [32]. Within the physics-based line, DemoGen [12] edits 3D point clouds directly and its extension R2E2R [33] renders consistent videos via a depth-conditioned generator. Simulation-based variants [11], [26], [32], [34] augment via geometric transformations in a digital twin, extending to clutter, bimanual embodiments, and photorealistic 3DGS [35] rendering. However, they rely on rigid SE(3) transformations that break down on deformables. Unlike DeformGen, SoftMimicGen [15] has tried to mitigate the trajectory-transfer challenge, but it still faces the limitation of state-space.

2.2 Deformable object manipulation↩︎

Early works use model-based planning built on physical simulators or learned dynamics models, using mass–spring [36], FEM [37], or particle-based representations [38]–[40] to predict deformation and plan actions [41]–[46]. A second line of work removes the need for explicit dynamics by directly learning visuomotor policies [47], [48] from demonstrations or interaction, covering tasks such as knot tying [49], cable insertion [50], cloth folding [51]–[55], and dough shaping [42]. More recently, large-scale vision–language–action models have been extended to deformable settings, showing promising generalization but requiring more data than their rigid-object counterparts [4], [5], [7]. However, this progress has been driven in large part by access to large-scale, diverse demonstration data, whose collection remains expensive, time-consuming, and difficult to scale.

3 Method↩︎

In this section, we propose DeformGen, which aims to synthesize a large volume of valid manipulation data for the same task but with varying object initial states, starting from sparse demonstration data. First, we present a novel object initial state augmentation approach in Sec. 3.1. Based on the above deformable object representation, we propose a manipulation trajectory augmentation method in Sec. 3.2. Furthermore, we describe the policy training framework employed to verify the effectiveness and efficiency of the augmented data in Sec. 3.3. For soft object modeling and simulation, we leverage PhysTwin [56] and Real2Sim-Eval [19] due to their high fidelity in both visual rendering and physical dynamics, with detailed descriptions provided in Appendix 8.1.

3.1 State Augmentation↩︎

The objective of this step is to generate diverse object configurations for the same task, serving both subsequent trajectory synthesis and policy evaluation. Fundamentally, synthesizing object states amounts to sampling the object’s configuration space. A practical method should produce states that are physically plausible under the simulator’s dynamics model and that are sufficiently diverse to improve downstream policy learning.

3.1.0.1 State space.

Following standard practice in deformable object simulation [36], [38], [39], [57]–[59], we consider a deformable object discretized into \(N\) particles with configuration space \(\mathcal{S} = \mathbb{R}^{3N}\), where each state \(\mathbf{s} = (\mathbf{p}_1, \dots, \mathbf{p}_N) \in \mathcal{S}\) specifies all particle positions. The physically plausible subspace \(\mathcal{S}_{\mathrm{real}} \subset \mathcal{S}\) contains all configurations consistent with real-world physical constraints. In general, \(\mathcal{S}_{\mathrm{real}} \subsetneq \mathcal{S}\): most points in \(\mathbb{R}^{3N}\) do not correspond to any physically realizable configuration.

3.1.0.2 Working assumption.

Our approach relies on the premise that a well-calibrated physics simulator \(\Phi_{\mathrm{sim}}(\mathbf{s}, \mathbf{f}, \Delta t)\) approximately preserves physical plausibility when evolving from a valid state, but cannot reliably restore it from an invalid one (Assumption 1 in Appendix 6). This asymmetry implies that any method which first perturbs the state out of \(\mathcal{S}_{\mathrm{real}}\) and then relies on simulation to “fix” it has no reliable path back to plausibility.

3.1.0.3 Why existing strategies fall short.

We identify three alternatives (details in Appendix 6.2): (i) Global rigid transformation preserves plausibility but is confined to a 6-DoF subspace of \(\mathbf{s}_0\), unable to capture shape or topological variation—confirmed by near-zero non-rigid residuals in Fig. 8. (ii) Per-particle perturbation faces a coverage–plausibility trade-off: large noise breaks connectivity; small noise yields only local wrinkles [60]. (iii) Kinematic deformation fields preserve topology but ignore material constraints, producing coherent yet dynamically inadmissible states. Both (ii) and (iii) leave \(\mathcal{S}_{\mathrm{real}}\) and rely on post-hoc repair that may fail.

3.1.0.4 Dynamics-based topological augmentation.

We instead augment states by applying localized external forces and forward-simulating the dynamics from a known valid state: \[\mathbf{s}_{\mathrm{aug}} = \Phi_{\mathrm{sim}}(\mathbf{s}_0,\, \mathbf{f},\, \Delta t), \quad \mathbf{s}_0 \in \mathcal{S}_{\mathrm{real}}, \label{eq:dynamics95aug}\tag{1}\] where \(\mathbf{f}\) is a localized force field. Because the method evolves the state through the simulator’s own dynamics, it never explicitly leaves \(\mathcal{S}_{\mathrm{real}}\), requiring no post-hoc repair from invalid states. Since localized forces can induce diverse non-rigid deformations (bending, twisting, folding, draping), the reachable set is not restricted to a low-dimensional submanifold. We do not claim full coverage of \(\mathcal{S}_{\mathrm{real}}\), but treat this as a practical sampling heuristic that explores a substantially broader region than rigid transformations—verified empirically in Fig. 8. Table 1 summarizes the comparison.

15pt

3.2 Trajectory Augmentation↩︎

The objective of this step is to synthesize valid manipulation trajectories for unseen object configurations. Our synthesized trajectories consist of three phases: approach, grasp, and manipulation. The grasp poses and manipulation trajectories are synthesized using Deformation Field Warping, while the approach trajectory is generated via interpolation from the robot’s reset pose to the grasp pose.

Rigid trajectory transfer methods [12], [32] assume uniform transformation across the object, neglecting distinct deformations across different parts of a deformable object. Inspired by [61], we construct a deformation field from per-particle displacements, yielding a closed-form spatial mapping without iterative optimization.

3.2.0.1 Position warping.

Let \(p_{\mathrm{orig}}, p_{\mathrm{def}} \in \mathbb{R}^{N \times 3}\) be the source and deformed point clouds. The per-point displacement is \(\delta_i = p_{\mathrm{def},i} - p_{\mathrm{orig},i}\). For each end-effector position \(x_t\) at timestep \(t\), we retrieve its \(k\) nearest neighbors from \(p_{\mathrm{orig}}\) and interpolate via inverse distance weighting: \[w_{t,j} = \frac{1}{\left\|x_t - p_{\mathrm{orig},\,\mathrm{nn}_j(x_t)}\right\| + \varepsilon}, \;\; \tilde{w}_{t,j} = \frac{w_{t,j}}{\sum_j w_{t,j}}, \;\; d(x_t) = \sum_j \tilde{w}_{t,j}\,\delta_{\mathrm{nn}_j(x_t)},\] where \(\varepsilon > 0\) ensures numerical stability. The warped position incorporates a time-dependent decay: \[x_t^{\mathrm{warp}} = x_t + \alpha_t \cdot d(x_t),\] where \(\alpha_t = \mathrm{decay}(t)\) allows the trajectory to follow local deformations initially while gradually reverting to the original path.

3.2.0.2 Orientation adaptation.

For the end-effector orientation, we construct local relative coordinates within the KNN neighborhood of \(x_t\): \[\ell^{\mathrm{orig}}_{t,j} = p_{\mathrm{orig},\,\mathrm{nn}_j(x_t)} - x_t, \quad \ell^{\mathrm{def}}_{t,j} = \ell^{\mathrm{orig}}_{t,j} + \delta_{\mathrm{nn}_j(x_t)}.\] A local Jacobian matrix \(J_t\) is estimated via least squares fitting to map the original local vectors to the deformed ones: \[J_t = \arg\min_J \sum_j \left\| \ell^{\mathrm{def}}_{t,j} - J\,\ell^{\mathrm{orig}}_{t,j} \right\|^2.\] Letting \(X_{\mathrm{orig}}\) and \(X_{\mathrm{def}}\) denote the matrices of stacked local vectors, the closed-form solution is: \[J_t = X_{\mathrm{def}} X_{\mathrm{orig}}^{\top} \left( X_{\mathrm{orig}} X_{\mathrm{orig}}^{\top} \right)^{+},\] where \((\cdot)^{+}\) denotes the Moore-Penrose pseudoinverse. The induced rotation \(R_t'\) is obtained by projecting \(J_t R_t\) onto the \(SO(3)\) manifold via SVD. The final warped orientation is computed via: \[R_t^{\mathrm{warp}} = \mathrm{SLERP}\left(R_t,\; R_t',\; \alpha_t\right).\]

In practice, the grasp pose correlates more strongly with nearby object points, so we use a small \(K\) for warping the grasp pose. The manipulation phase depends on the overall object state, so we set \(K\) to the total number of object points to capture global deformation. Given the tabletop scenario, we constrain rotations to the Z-axis perpendicular to the table surface. Details are in Appendix 7.

3.3 Policy Training↩︎

To evaluate the effectiveness and efficiency of our augmentation approach, policies are trained via imitation learning and validated within a simulation environment. For three tasks—rope routing, toy packing, and cloth folding—we collect one teleoperation demonstration per task. Using the state augmentation method described in Sec. 3.1, we synthesize more than \(1200\) distinct object states for each task to facilitate trajectory synthesis and evaluation scenarios. Subsequently, we employ the Deformation Field Warping method and the local rigid-transfer ablation for trajectory augmentation.

The augmented trajectories are executed in simulation to verify their success, with task-specific success criteria detailed in Appendix 8.2. We record third-person and wrist-mounted RGB images along with corresponding actions during execution. The successful episodes are split into training and held-out test sets (details in Sec. 4.1).

Following the protocol of Real2Sim-Eval [19], we train four policy architectures: ACT [62], Diffusion Policy [63], SmolVLA [64], and \(\pi_0\) [1] (fine-tuned via LoRA). The trained policies are evaluated on held-out object states unseen during training, including configurations where the warping method failed to generate successful trajectories.

4 Experiments↩︎

4.1 Implementation Details↩︎

All experiments are conducted in Real2Sim-Eval [19] with PhysTwin [56] for soft-body dynamics and rendering. The robot is an xArm7 with two RGB cameras (third-person and wrist, \(848\times480\), 30 Hz). We evaluate on three tasks: rope routing (thread a rope through a clip), toy packing (place a stuffed toy into a container), and cloth folding (fold cloth into a triangle). Success criteria, augmentation parameters, and training hyperparameters are detailed in the Appendix.

For state augmentation, the gripper executes randomized Cartesian perturbations while in contact with the object (180 steps for rope/toy, 260 for cloth), followed by stabilization. For each task, we generate augmented states and attempt trajectory synthesis to obtain 1,000 successful trajectories for training and 200 successful states for testing.

4.1.0.1 Compared methods.

To disentangle the contributions of state augmentation and trajectory synthesis, we compare four training regimes:

• 1 Src.: a single source demonstration without any augmentation.

• SoftMimicGen* (SMG*): SoftMimicGen [15] shares a similar philosophy to ours in trajectory synthesis—adapting demonstrations to deformed object geometry—but its state augmentation remains rigid: its state distribution is “typically one with a larger set of possible placements for objects in the scene” [15], i.e., SE(3) perturbations of object pose. Since SoftMimicGen is not open source, we reimplement its core design following the descriptions in the original paper.

• DeformGen* (DG*): topological state augmentation paired with local rigid trajectory transfer. This ablation uses the same distribution of augmented states as DG but replaces deformation-field warping with a rigid transform estimated from the \(K\) nearest material points around the grasp point and applied to the entire trajectory, isolating the effect of trajectory synthesis.

• DeformGen (DG): our full method, which pairs dynamics-based topological state augmentation with deformation-field warping. Compared with DG*, DG adapts the trajectory through a continuous deformation field rather than a single local rigid transform, enabling better alignment with the deformed object throughout manipulation.

The comparison between SMG* and DG reveals the effect of state augmentation contrasting rigid with topological diversity, while the comparison between DG* and DG reveals the effect of trajectory synthesis contrasting local rigid transfer with deformation-field warping.

4.2 Experiment Results↩︎

Results are in Table 2. DeformGen achieves the highest average success rate across three out of four policy architectures. The result reveals two insights:

2pt

(1) Topological state diversity contributes to generalization. Comparing SMG* with DG highlights the effect of state augmentation: both methods use deformation-field warping for trajectory transfer, but SMG* relies on rigid state perturbations whereas DG uses dynamics-based topological state augmentation. DG achieves higher average success rates in most architectures, suggesting that broader coverage of deformable-object configurations is important for policy generalization.

(2) Deformation-field warping provides complementary gains. Since DG* and DG use the same topologically augmented states, their comparison isolates trajectory transfer. DG often improves over DG*, suggesting that deformation-aware warping can further benefit policy learning.

Together, these findings suggest that both broader deformable-state coverage and deformation-aware trajectory transfer contribute to policy generalization, with their relative effects varying across architectures and tasks. Figure 6 qualitatively illustrates this trend: policies trained with DeformGen successfully complete tasks while comparison methods often exhibit grasp misalignment or incomplete manipulation.

4.3 State Coverage Analysis↩︎

We decompose each augmented state into a rigid SE(3) component and a non-rigid residual via Procrustes alignment (Figure 8). Rigid augmentation clusters near the source with negligible residual, while DeformGen spreads broadly with large residuals—confirming that the performance gains in Table 2 stem from genuine topological diversity rather than merely more data at similar configurations.

Figure 3 shows representative augmented object states generated by DeformGen for each task. Starting from a single source configuration, our dynamics-based augmentation produces a diverse set of topologically distinct states—including different rope curvatures, varied stuffed toy orientations and compressions, and diverse cloth folds and drapes—all of which are physically plausible under the simulator’s dynamics model.

4.4 Ablation Studies↩︎

Q: Can DeformGen generalize to rigid-only scenarios? Table 3 evaluates whether topological augmentation hurts performance when the test states only involve rigid transformations. SMG* achieves the highest average success rate for ACT, DP, and SmolVLA, which is expected because its rigid-state training distribution closely matches the rigid-only test set. In contrast, DG* and DG are trained on broader topological variations. Nevertheless, DG remains competitive and even performs best for \(\pi_0\), suggesting that training on topologically diverse data does not substantially compromise performance on simpler rigid scenarios.

6pt

Q: Impact of synthetic data quantity. Table 4 shows that average performance improves with scale. The average success rates increase monotonically from \(N{=}100\) to \(N{=}750\) for both ACT (19.50% \(\to\) 61.50%) and SmolVLA (36.83% \(\to\) 63.17%). This suggests that dynamics-based augmentation can benefit from increased data scale, with larger synthetic datasets providing average gains.

3pt

Q: Can the policy generalize to synthesis failure cases? This ablation tests whether the policy merely memorizes the augmented trajectories or learns transferable manipulation strategies. We evaluate on hard samples—states where trajectory synthesis itself failed to produce a valid demonstration, meaning the policy has never seen a successful trajectory for these configurations. Since rope achieves nearly 100% synthesis success, this study is conducted on toy and cloth only. As shown in Table 5, policies trained with augmentation achieve non-trivial success on these out-of-distribution states, though performance varies across tasks and architectures. Augmentation helps policies learn generalizable manipulation strategies rather than memorizing individual demonstrations, enabling some degree of extrapolation to unseen configurations.

4.5 Failure Analysis↩︎

Figure 9 shows representative failure cases. Common failure modes include imprecise grasp on highly deformed configurations where the visual appearance deviates significantly from training data, and premature release due to unstable contact under large deformations.

5 Conclusion↩︎

In this work, we proposed DeformGen, a dynamics-based augmentation framework that expands the valid state distribution through localized physical disturbances, forward simulation, and stabilization, and transfers source manipulation trajectories via deformation-field warping. In this way, DeformGen augments both deformable object states and their associated manipulation behaviors. Experiments on high-fidelity deformable manipulation benchmarks showed that DeformGen generally improves policy learning over both training on the original demonstrations alone and rigid-style augmentation baselines. More broadly, our results suggest that effective augmentation for deformable manipulation requires dynamics-consistent state synthesis and deformation-aware trajectory transfer.

Appendix↩︎

6 State Augmentation Details↩︎

6.1 Formal Assumption↩︎

Our approach relies on the premise that a well-calibrated physics simulator approximately preserves physical plausibility when evolving from a valid initial state. We formalize this as a working assumption on the simulator \(\Phi_{\mathrm{sim}}(\mathbf{s}, \mathbf{f}, \Delta t)\), which evolves state \(\mathbf{s}\) under external forces \(\mathbf{f}\) over time interval \(\Delta t\):

We note that this assumption is an idealization: real simulators introduce numerical integration errors and may not perfectly model all material properties, so the generated states are plausible with respect to the simulator’s dynamics model rather than guaranteed to match real-world physics exactly. The use of high-fidelity simulators (PhysTwin [56]) narrows this gap in practice.

6.2 Detailed Analysis of Existing Strategies↩︎

We provide a detailed analysis of three representative augmentation strategies and their limitations when applied to deformable objects.

6.2.0.1 (i) Global rigid transformation.

Applying a uniform \(\mathbf{T} \in SE(3)\) to all particles preserves all inter-particle relations, so the augmented state remains in \(\mathcal{S}_{\mathrm{real}}\). However, the reachable set is confined to a 6-dimensional subspace spanned by rigid pose variations of \(\mathbf{s}_0\), which cannot capture any shape or topological variation of deformable objects. This is confirmed empirically in Fig. 8: rigid augmentation produces near-zero non-rigid residuals across all three tasks.

6.2.0.2 (ii) Per-particle independent perturbation.

Adding independent noise \(\boldsymbol{\epsilon}_i \sim \mathcal{P}(\sigma)\) to each particle can in principle reach any \(\mathbf{s} \in \mathcal{S}\), but faces a practical coverage–plausibility trade-off. Large \(\sigma\) produces disordered point clouds that break inter-particle connectivity and introduce topology artifacts (e.g., self-intersections, disconnected segments), pushing the state far outside \(\mathcal{S}_{\mathrm{real}}\) in ways that subsequent stabilization steps typically cannot recover. Small \(\sigma\) preserves local topology but induces only surface wrinkles [60], confining the resulting states to a local neighborhood of \(\mathbf{s}_0\) with insufficient diversity for policy learning. Crucially, even when a stabilization step is applied, there is no mechanism to verify whether the result has returned to \(\mathcal{S}_{\mathrm{real}}\), making this approach unreliable in practice.

6.2.0.3 (iii) Kinematic topological transformation.

Applying a continuous deformation field \(\boldsymbol{\phi}: \mathbb{R}^3 \to \mathbb{R}^3\) improves upon (ii) by preserving topological coherence—the connectivity structure is maintained by construction. However, \(\boldsymbol{\phi}\) is constructed without reference to the object’s material model, so the deformed state may violate internal dynamic constraints (e.g., producing rest-shape configurations with unrealistic internal stress or interpenetration with the environment). These configurations are structurally coherent but dynamically inadmissible, and stabilization cannot reliably project them back onto \(\mathcal{S}_{\mathrm{real}}\) because the simulator’s corrective dynamics may converge to a different basin or fail to converge at all.

6.3 Advantages of Dynamics-Based Augmentation over (ii) and (iii)↩︎

Beyond broader coverage relative to rigid transformations, our dynamics-based approach offers two practical advantages over per-particle perturbation and kinematic deformation:

• No plausibility–diversity trade-off. Both strategies (ii) and (iii) face a fundamental tension: increasing perturbation magnitude increases diversity but also the likelihood of producing implausible configurations. Our method sidesteps this trade-off because diversity is achieved through the simulator’s own dynamics—larger or longer-duration forces naturally produce more diverse states, while the simulation’s internal constraints (collision handling, material constitutive laws, boundary conditions) continuously enforce plausibility throughout the trajectory. There is no separate perturbation-then-repair pipeline that could fail.

• Implicit enforcement of coupled constraints. Deformable objects are subject to multiple interacting constraints simultaneously: material elasticity, self-collision avoidance, environmental contact, and gravitational settling. Strategies (ii) and (iii) perturb geometry without awareness of these coupled constraints, and a subsequent stabilization step can at best enforce them approximately and sequentially. In contrast, forward simulation enforces all constraints jointly at each time step through the simulator’s integrated solver, producing states where internal stresses, contact forces, and boundary conditions are mutually consistent. This is particularly important for objects with complex rest-state interactions (e.g., a rope draped over a fixture, or cloth resting on a surface with folds), where violating one constraint easily cascades into violations of others.

6.4 Reachable Set Discussion↩︎

The set of states reachable via dynamics simulation from \(\mathbf{s}_0\) is: \[\mathcal{R}(\mathbf{s}_0) \;=\; \left\{ \Phi_{\mathrm{sim}}(\mathbf{s}_0,\, \mathbf{f},\, \Delta t) \;\middle|\; \mathbf{f} \in \mathbb{R}^{3N},\; \Delta t > 0 \right\}. \label{eq:reachable}\tag{2}\] In principle, \(\mathcal{R}(\mathbf{s}_0)\) could be large, since any physically plausible configuration is connected to \(\mathbf{s}_0\) through some physical process. However, we do not claim that our randomized sampling of \((\mathbf{f}, \Delta t)\) achieves full coverage of \(\mathcal{S}_{\mathrm{real}}\) or matches the true distribution of real-world configurations. We treat dynamics-based augmentation as a practical sampling heuristic that explores a substantially broader and more physically grounded region of the state space than rigid transformations. This is verified empirically in Sec. 4.3.

7 Trajectory Augmentation Details↩︎

7.1 Decay Function↩︎

The decay function \(\alpha_t = \mathrm{decay}(t)\) in the position and orientation warping controls how strongly the deformation field influences the trajectory over time. We support three configurations:

• None: \(\alpha_t = 1\) for all \(t\). The deformation field is applied uniformly throughout the trajectory.

• Linear: \(\alpha_t = \max(0,\; 1 - t/T)\), where \(T\) is the total trajectory length. The influence decreases linearly to zero.

• Exponential: \(\alpha_t = e^{-\lambda t}\), where \(\lambda > 0\) controls the decay rate. The influence decreases exponentially.

The decay allows the trajectory to closely follow local deformations near the grasp phase while gradually reverting to the original trajectory towards the end of the manipulation phase. The choice of decay function is task-dependent and specified in Appendix 8.5.

7.2 KNN Scope for Grasp vs.Manipulation Phases↩︎

In practice, we observe that the grasp pose correlates more strongly with object points in the vicinity of the grasp point. Therefore, we employ a small \(K\) (e.g., \(K = 5\)–\(10\)) for warping the grasp pose, so that only nearby particle displacements influence the grasp alignment.

Conversely, the manipulation phase depends on the overall object state; the end-effector must compensate not only for local geometry changes but also for global shape shifts. We therefore set \(K\) equal to the total number of object points \(N\) for the manipulation trajectory, effectively using a globally weighted deformation field.

7.3 Orientation Constraints↩︎

Given the tabletop manipulation scenario in our experiments, significant orientation changes occur primarily around the Z-axis (perpendicular to the table surface). We therefore constrain the orientation warping to the Z-axis component: the original rotation matrix \(R_t\) and the induced rotation \(R_t'\) are first projected onto their Z-axis rotational components before SLERP interpolation is applied. This prevents spurious tilting or flipping of the end-effector that could arise from noisy Jacobian estimates in the other axes.

8 Implementation Details↩︎

8.1 Simulation and Robot Setup↩︎

All experiments are conducted in Real2Sim-Eval [19], which provides physically accurate soft-body dynamics and photorealistic rendering via PhysTwin [56]. The robot is an xArm7 manipulator equipped with two RGB cameras: a fixed third-person camera and a wrist-mounted camera, both at \(848\times480\) resolution and 30 Hz frame rate. The policy outputs 8-dimensional actions consisting of end-effector position \((x, y, z)\), quaternion orientation \((q_w, q_x, q_y, q_z)\), and gripper opening at 30 Hz control frequency. Internally, the simulation converts the policy output to a 13D command (xyz \(+\) \(3\times3\) rotation matrix \(+\) gripper) before execution.

8.2 Task Descriptions and Success Criteria↩︎

8.2.0.1 Rope routing.

The robot must thread a deformable rope through a clip. Success is evaluated over the final 100 frames of each episode. The episode is considered successful if at least 30 frames satisfy the condition that the rope forms sufficient intersections (at least 100 spring-segment crossings) with both the upper and lower planes of the clip, indicating that the rope has been threaded through.

8.2.0.2 Toy packing.

The robot must place a stuffed toy into a container. Success is evaluated at the final frame. We construct a minimum oriented bounding box (OBB) from the initial reference mesh and scale it by a factor of \(1.05\). The episode succeeds if at least 3,050 object points fall within this scaled OBB.

8.2.0.3 Cloth folding.

The robot must fold a cloth into a triangular shape. Success is evaluated at the final frame. The point cloud is projected onto the table plane to form a binary mask. We extract the largest connected component, fit a minimum bounding triangle, and verify three conditions: (i) the contour has 3–4 approximate vertices, (ii) the IoU between the mask and the fitted triangle \(\geq 0.72\), and (iii) the mask coverage of the triangle \(\geq 0.80\).

8.3 State Augmentation Parameters↩︎

Since the simulation environment does not expose a direct external-force API, we implement localized physical disturbances by commanding the gripper to execute randomized Cartesian perturbations while in contact with the object, transmitting forces through contact dynamics. Each augmentation episode consists of a sequence of random steps; each step applies either a planar translation (sampled from discrete \(\pm x\), \(\pm y\) directions) or a \(z\)-axis rotation (with probability \(p_{\mathrm{rot}}\)). Task-specific configurations are as follows:

• Rope / Toy: 180 random steps, translation magnitudes sampled from \(\{0.012, 0.006, 0.003\}\) m, rotation steps of \(\pm6^{\circ}\), rotation probability \(p_{\mathrm{rot}} = 0.45\).

• Cloth: 260 random steps, translation magnitudes sampled from \(\{0.018, 0.009, 0.0045\}\) m, rotation steps of \(\pm8^{\circ}\), rotation probability \(p_{\mathrm{rot}} = 0.55\).

After perturbation, the object is stabilized for 30–40 simulation steps to reach quasi-static equilibrium.

8.4 Data Splits↩︎

For each task, we generate augmented states via dynamics-based topological transformation and attempt trajectory synthesis until obtaining sufficient successful demonstrations. The generation statistics are:

• Rope: 1,294 successful trajectories out of 1,300 generated states. Trajectory synthesis success rate: 99.5%. Split into 1,000 training / 200 test. Remaining successful trajectories are unused.

• Toy: 1,327 successful trajectories out of 2,200 generated states. Trajectory synthesis success rate: 60.3%. Split into 1,000 training / 200 test, with 200 failed states sampled from 873 available. Remaining successful trajectories are unused.

• Cloth: 1,778 successful trajectories out of 4,500 generated states. Trajectory synthesis success rate: 39.5%. Split into 1,000 training / 200 test, with 200 failed states sampled from 2,722 available. Remaining successful trajectories are unused.

Failed states from toy and cloth tasks serve as hard samples for the generalization ablation in Sec. 4.4. Rope is excluded from this ablation due to insufficient failed samples.

8.5 Trajectory Augmentation Hyperparameters↩︎

Task-specific trajectory warping configurations:

• Rope: Grasp KNN \(K=5\), manipulation KNN \(K=N\) (all points), decay function: linear.

• Toy: Grasp KNN \(K=5\), manipulation KNN \(K=N\), decay function: none.

• Cloth: Grasp KNN \(K=10\), manipulation KNN \(K=N\), decay function: exponential (\(\lambda=0.02\)).

8.6 Policy Training Hyperparameters↩︎

All policies are trained on a single NVIDIA A100 GPU. Hyperparameters are tuned per algorithm:

8.6.0.1 ACT [62].

Learning rate: \(1\times10^{-5}\). Batch size: 512. Training epochs: 10.

8.6.0.2 Diffusion Policy (DP) [63].

Learning rate: \(1\times10^{-4}\). Batch size: 512. Training epochs: 10. Scheduler: cosine with 500-step warmup.

8.6.0.3 SmolVLA [64].

Learning rate: \(1\times10^{-4}\). Batch size: 128. Training epochs: 10. Scheduler: warmup 1000.

8.6.0.4 \(\pi_0\) [1].

Fine-tuned via LoRA using the OpenPI framework. Peak learning rate: \(2.5\times10^{-5}\), decay learning rate: \(2.5\times10^{-6}\). Batch size: 8. Training epochs: 10. Optimizer: AdamW (\(\beta_1{=}0.9\), \(\beta_2{=}0.95\), weight decay \(1\times10^{-10}\)). Scheduler: cosine decay.

9 Limitations↩︎

We acknowledge several limitations of the current work:

• Single-arm manipulation only. All experiments are conducted with a single xArm7 manipulator. Extending DeformGen to bimanual or multi-robot settings—where coordination between arms introduces additional constraints on trajectory synthesis—remains future work.

• Limited task diversity. We validate on three deformable manipulation tasks (rope, stuffed toy, cloth), which cover a range of material properties (1D, quasi-rigid 3D, 2D sheet). However, other important categories such as dough/clay shaping, surgical tissue manipulation, or cable routing in cluttered environments have not been evaluated. The generality of our dynamics-based augmentation to these domains remains to be demonstrated.

• Sim-to-real gap. All experiments are conducted entirely in simulation. While Real2Sim-Eval and PhysTwin provide high-fidelity physics and rendering, transferring the augmented policies to real hardware may require additional domain adaptation or fine-tuning to handle discrepancies in material properties, contact dynamics, and visual appearance.

• Trajectory synthesis is not universally successful. Guaranteeing successful trajectory transfer for arbitrary initial states is inherently difficult. Following the core philosophy of [61], our warping assumes that geometric correspondence preserves task semantics, but this only holds approximately—complex contact dynamics, large topological changes, and kinematic constraints can cause warped trajectories to fail. This is reflected in our varying success rates. Encouragingly, policies trained on successful trajectories still generalize to some failure states (Sec. 4.4). Future work could mitigate this via iterative human-in-the-loop demonstration collection, closed-loop trajectory refinement, or multi-source warping that selects the most compatible demonstration for each target state.

10 Broader Impacts↩︎

This work aims to reduce the cost of collecting manipulation demonstrations for deformable objects by providing an automated data augmentation pipeline. The positive societal impact includes enabling more accessible and scalable robot learning for tasks involving soft materials (e.g., household assistance, garment handling, food preparation), potentially benefiting applications in elder care and manufacturing.

As the method operates entirely in simulation for data synthesis and does not involve real-world data collection, human subjects, or generation of potentially harmful content, we do not foresee direct negative societal impacts beyond standard safety considerations for robotic manipulation systems. The trained policies are task-specific manipulation controllers without broader capabilities that could be misused.

11 Licenses↩︎

We list the licenses of all external assets used in this work:

• Real2Sim-Eval [19]: MIT License

• PhysTwin [56]: MIT License

• ACT [62]: MIT License

• Diffusion Policy [63]: MIT License

• SmolVLA [64]: Apache License 2.0

• \(\pi_0\) [1]: Apache License 2.0

Our use of these assets complies with their respective license terms.

References↩︎