1 Introduction↩︎

Accurate simulation of deformable objects is a persistent challenge in robotics and computer vision. Such objects can change shape significantly under external forces, undergoing stretching, bending, crumpling, and twisting. Simulation difficulty stems from material properties, such as heterogeneous elasticity and damping, but is also complicated by factors, such as self-collisions and friction, that come into play during contact-rich interactions.

The literature contains two dominant paradigms for deformable object simulation: physics-based methods [1]–[3] and learning-based methods [4]–[6] (see Section 2 for a more detailed survey of related works). Physics-based methods use equations to describe how materials deform and respond to forces, yielding physically plausible and interpretable simulations. However, these methods require precise material parameters that are difficult to obtain in practice [7]–[9]. The complex mathematics of these simulators typically restricts practitioners to gradient-free optimization [7], which can tune only a handful of parameters before becoming intractable [8], [10]. Even with optimal parameters, physics-based models are inherently limited by discretization (coarse meshes miss fine-scale deformations [11], [12]) and modeling assumptions (linear elasticity fails under large strains [13], [14]). Learning-based methods sidestep these limitations by fitting observations that are challenging to model analytically. However, purely data-driven models are data-hungry [15] and suffer from poor generalization to unseen scenarios [16], [17]. They often lack the physical consistency required for real-world deployment [16] and, without the inductive bias provided by physical priors, they may learn spurious correlations [18].

To combine the interpretability and generalizability of physics-based models with the flexibility of learning-based models, we propose a hybrid simulation framework of Physics-Guided Residual Dynamics(PGRD). As shown in Fig. 1 and explained in Section 3, PGRD uses a spring-mass simulator as its physics backbone and a neural network to predict residual corrections to that model. PGRD follows a two-stage training procedure: first, we optimize the simulator’s parameters to match real-world observations; and second, we train the neural network to compensate for the discrepancies between the physics model and the data. Note that making this two-stage approach work is nontrivial, as directly injecting learned position corrections into simulated states can destabilize the dynamics and cause error accumulation over time. Motivated by impulse-based simulation that evolves states through velocity updates rather than position [19], [20], [20], we predict residual velocities and integrate them forward in time, yielding position corrections that respect the underlying dynamics. This velocity-based formulation enables smooth, stable training and simulation. To model temporal dependencies, we incorporate a sliding-window transformer that refines velocity corrections across multiple timesteps. This temporal history enables the network to capture dynamic phenomena such as momentum, while a gating mechanism ensures stable training.

Compared to purely physics-based or learning-based simulation methods, PGRD offers several advantages. First, the physics backbone provides priors that can improve generalization to new scenarios. Second, it is expected to require less training data than purely learning-based approaches, as the network only needs to predict corrections rather than the entire dynamics from scratch. Third, it maintains computational efficiency while achieving higher accuracy than physics-based methods alone. To further reduce compounding errors over long horizons, we train PGRD with multi-step rollouts: during training, the model is recursively applied to its own predicted states rather than being reset to ground truth at every step. This exposes the residual model to the same distribution shift encountered at test time and is important for stable long-horizon simulation.

In Section 4, we validate PGRD on a diverse set of real-world deformable objects, including rope, paper, soft toys, flag, and duster. Across these objects, we evaluate simulation quality against tracked 3D point trajectories from real-world executions, covering both prehensile and non-prehensile manipulation. Our experiments demonstrate that PGRD consistently outperforms both purely physics-based and purely learning-based approaches in tracking accuracy. Notably, only PGRD performs well on the duster, which is a heterogeneous object with a rigid stem and soft feathers. Here, uniform material assumptions in most physics-based deformable simulators fail and learning-based methods struggle with the complexity. Besides standard tracking evaluation, we also show the advantage of PGRD for action-conditioned video prediction, which we perform by attaching 3D Gaussians to the simulated particles.

Beyond evaluation, we demonstrate two applications of PGRD in Section 5. First, PGRD can serve as a forward model for manipulation planning via Model Predictive Path Integral (MPPI) control, where candidate action sequences are rolled out and ranked by Chamfer Distance to a target configuration. We demonstrate this on challenging tasks such as cable rerouting through a narrow slot, where PGRD succeeds in 8 out of 10 trials compared to 2 out of 10 for the tuned spring-mass baseline. We further extend planning to a language-conditioned setting, where a goal image generated from a language command replaces the need for a pre-collected target point cloud. Second, PGRD enables interactive photorealistic simulation in which users issue manipulation commands and the predicted particle states drive 3D Gaussians to render photorealistic views of the resulting deformation.

In summary, our contributions are as follows. (1) We propose Physics-Guided Residual Dynamics, a hybrid simulation framework that combines an optimizable spring-mass model with learned residual corrections to accurately model deformable object dynamics. (2) We introduce a two-stage training procedure that first optimizes physics parameters using black-box optimization and then trains a neural network to predict residual corrections. (3) We conduct extensive experiments on diverse real-world deformable objects, demonstrating that our approach achieves superior performance compared to existing physics-based and learning-based methods.

2 Related Works↩︎

Physics-Based Simulation for Deformable Objects. Traditional physics-based simulation methods rely on analytical models to discretize deformable objects and numerical solvers for the equations of motion of the models. For deformable objects, spring-mass models [1], [21], [22] represent one of the most intuitive and efficient approaches, where objects are modeled as networks of point masses interconnected by springs. Consequently, this representation has been extensively employed to model the dynamics of diverse deformable objects in both computer graphics and robotics [1], [7], [21]–[25]. Our work leverages the spring-mass model as a backbone precisely because of its efficiency and interpretability. While more complex approaches like Finite Element Methods (FEM) [26], [27], Position-Based Dynamics (PBD) [3], [28], and Material Point Methods (MPM) [2], [29] offer higher physical fidelity, they incur high computational costs and complexity. We include MPM as a baseline in our experiments to validate this tradeoff.

Despite their physical foundations, the above-mentioned methods require precise material parameters that are difficult to obtain in practice. A recent line of work addresses this challenge by reformulating physics simulators to be compatible with automatic differentiation [22], [25], [30]–[41]. These methods identify optimal physics parameters by backpropagating through the simulation process, solving inverse problems to find material properties that best match observed data. While such parameter identification can improve simulation accuracy, it imposes a strong requirement the simulator must be fully differentiable. This is often impractical in contact-rich robotics tasks where discontinuities from collisions [23], [42], [43], friction mode transitions [43]–[45], and non-smooth contact geometry [43], [46], [47] lead to exploding or vanishing gradients. In practice, even state-of-the-art differentiable simulation methods suffer from these fundamental limitations. We compare against PhysTwin [25], a recent state-of-the-art differentiable spring-mass approach, across all of our tasks, and show that it additionally struggles on heterogeneous deformable objects. Our method does not require the simulator to be differentiable, enabling broader applicability to realistic manipulation scenarios.

Learning-Based Simulation for Deformable Objects. Learning has emerged as a powerful alternative to traditional physics-based simulation, particularly excelling in scenarios requiring real-time performance with complex nonlinear dynamics [4]–[6], [48]–[59]. These approaches leverage data-driven models, typically neural networks, to learn deformation patterns directly from observation data, bypassing the need for explicit material parameters or complex numerical solvers [60], [61].

Graph Neural Networks (GNNs) have become prominent in this domain due to their natural ability to represent meshes as graphs. GNNs excel at capturing long-range interactions and complex dependencies through message passing [60], [62]–[67], allowing them to effectively simulate internal and external forces causing deformation [68], [69]. Notably, GBND [63] establishes a topology over sparse particles, enabling the GNN to efficiently learn and propagate the dynamics across the object. We compare against GBND in our experiments. However, GNN-based methods face significant challenges: the effectiveness of message passing is sensitive to graph structure and is vulnerable to partial observations, while insufficient steps fail to capture global information and excessive steps cause oversmoothing [70], [71].

Recent learning-based methods have explored architectural improvements, including transformers for handling dense particles [72], [73], recurrent neural networks for temporal consistency [74], and attention mechanisms for improved long-range dependency modeling [64]. These methods have demonstrated success across diverse materials, including cloth [22], [65], [75], fluids [60], [64], plasticine [76]–[78], and granular materials [60]. A recent state-of-the-art method is Particle-Grid Neural Dynamics [79] (PGND), which combines particle representations with spatial grids to learn dynamics while maintaining spatial continuity, and serves as a strong baseline in our experiments. On the down side, approaches like PGND require substantial training data and may struggle with generalization to unseen scenarios or material properties that are significantly different from the training distributions. Our work addresses these limitations by grounding the neural network in physical priors, allowing it to focus on learning only residual corrections rather than the full dynamics from scratch.

Residual Dynamics. The idea of residual learning has been previously employed to model the dynamics of robots [80]–[83]. In these works, the robot’s model is known, and the residuals primarily reflect simple modeling errors, such as PID gains or backlash. These discrepancies are relatively easy to capture because they often manifest as consistent deviations from the nominal model. In contrast, we model the dynamics of deformable objects, where the residual must compensate for complex, high-dimensional phenomena, including nonlinear stiffness, spatially varying material properties, and contact-rich interactions. The most closely related work in spirit is [84], which models the environment’s residual dynamics rather than the robot’s. However, it is limited to low-dimensional toy problems such as planar pushing, and its simple architecture does not scale to the high-dimensional particle cloud representations required for deformable object manipulation, making a direct comparison infeasible. To the best of our knowledge, PGRD is the first work to extend the residual paradigm to modeling high-dimensional, complex states of deformable objects.

3 Method↩︎

This section presents the details of our Physics-Guided Residual Dynamics method, which is illustrated in Fig. 1. We first explain our physics backbone, followed by the residual dynamics framework.

3.1 Physics Backbone↩︎

Given RGBD observations, we extract surface points to instantiate the simulation. We represent deformable objects using a spring-mass model, wherein the object is discretized into a graph structure consisting of point masses (nodes) interconnected by springs (edges). Each node \(i\) has a position \(\mathbf{x}_i \in \mathbb{R}^3\) and velocity \(\mathbf{v}_i \in \mathbb{R}^3\) that evolve over time according to Newtonian mechanics with a semi-implicit Euler solver. This representation provides computational efficiency and straightforward implementation while capturing essential elastic deformation behaviors. To model manipulation, we apply constraints to particles near the gripper: for prehensile manipulation, they move rigidly with the gripper, whereas for non-prehensile manipulation, the gripper acts as a spherical collision shape that pushes penetrating particles to its surface.

The fidelity of the spring-mass simulator depends critically on five physical parameters collectively denoted as \(\theta\): 1) Stiffness, which defines the elastic resistance of springs; 2) Damping, which controls energy dissipation during motion; 3) Threshold, which determines the maximum distance for creating springs between nodes; 4) Max springs per node, which limits the connectivity of each mass point for simulation stability; and 5) Ground friction, which governs contact interactions with the environment. These parameters collectively encode the material properties and environmental conditions that govern the dynamics of the simulated object.

Given a batch of trajectories sampled from the dataset, each with initial state \((\mathbf{X}_0,\mathbf{V}_0)\) and action sequences \(\mathcal{A} = \{a_t\}_{t=0}^{T-1}\), we roll out the simulator for \(T\) time steps with actions \(\{a\}_{t=0}^{T-1}\) under candidate parameters \(\theta\) to obtain predicted point cloud positions \(\{\hat{\mathbf{X}}_t\}_{t=1}^T\). At each time step \(t\), we compute a pointwise mean squared error between predicted and ground truth configurations. We average this error over the trajectory horizon to obtain the optimization objective. We minimize this objective using Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [85], which is particularly effective for our problem as it optimizes parameters without requiring gradients. We use CMA-ES because contacts and collisions cause sudden changes in forces, preventing reliable gradient computation in the simulator.

For volumetric objects such as soft toys, modeling only the surface shell often leads to unrealistic collapse, as the lack of internal support points causes the object to flatten under gravity or compression. To address this, we augment the spring-mass model with internal points that provide structural volume. Starting from our multi-view observations, we first reconstruct the complete surface point cloud using RaySt3R [86] and extract a watertight mesh via marching cubes. We then uniformly sample particles within this mesh to populate the interior, ensuring the spring-mass model preserves the object’s volume and structural integrity during simulation.

3.2 Residual Dynamics Framework↩︎

While our optimized spring-mass backbone provides a physically grounded baseline, it inherently struggles to capture complex behaviors such as nonlinear stiffness, non-uniform contact friction, and heterogeneous material properties. Instead of attempting to model these intricacies analytically, we introduce a learned residual module to predict the discrepancy between the physics model and observations.

Formally, at each time step \(t\) and step size \(dt\), we use the physics backbone to generate an immediate prediction \((X^{\text{sim}}_{t+1}, V^{\text{sim}}_{t+1})\). We then employ a neural network to predict a per-particle residual velocity \(\Delta V^{\text{residual}}_{t+1}\) based on the current simulator state and history. An alternative design would be to predict residual positions directly. However, we found this formulation to be unstable in practice: directly correcting positions led to occasional simulation blow-ups, where object state becomes NaNs. Intuitively, directly predicting positions can create a mismatch between the corrected positions and the velocities, destabilizing the simulation. We therefore adopt a velocity-based residual formulation, which ensures smooth integration and avoids the instability often associated with direct position corrections. The final velocities are obtained by adding the learned residuals to the simulator’s predictions: \[\hat{V}_{t+1} = V^{\text{sim}}_{t+1} + \Delta V^{\text{residual}}_{t+1}.\] These corrected velocities are then time-integrated to obtain the final position updates: \[\hat{X}_{t+1} = X^{\text{sim}}_{t+1} + \Delta V^{\text{residual}}_{t+1} \cdot dt.\] For particles that are rigidly held by grippers, we enforce a boundary condition by zeroing out their residuals. This formulation effectively bridges the gap between the optimized spring-mass model and the observed real-world dynamics.

We train the residual network with a supervised objective that minimizes the mean squared error between predicted and ground truth particle positions over a rollout: \[\mathcal{L}_{\text{res}} = \frac{1}{T} \sum_{t=1}^{T} \frac{1}{N} \left\| \hat{X}_t - X_t^{*} \right\|_2^2,\] where \(\hat{X}_t\) denotes the predicted particle positions at rollout step \(t\), \(X_t^{*}\) denotes the corresponding ground truth positions, and \(N\) is the number of particles. To improve robustness to error accumulation over long horizons, we apply this loss over multi-step rollouts rather than a single step. During training, we do not reset the simulator to the ground truth state at every step. Instead, we feed the hybrid simulator’s predicted state at time \(t\) back as the input for time \(t+1\). This exposes the network to its own past predictions and allows it to learn how to recover from drift.

3.3 Network Architecture↩︎

To parameterize the residual velocity, we design a network consisting of three primary components: encoder, decoder, and temporal aggregator. The network architecture is shown in Fig. 2. While individual components draw on established design choices, our contribution lies in their integration into a single residual-dynamics architecture. We ablate the architecture components in App. 7 and provide the complete hyperparameters for the network components in App. 8.

Encoder. We employ a Point Transformer V3 (PTv3) [87] architecture to extract per-particle features. At time step \(t\), we construct input features by concatenating the current state \((X_{t}, V_{t})\), history from previous time steps \((X^{\text{his}}_{t}, V^{\text{his}}_{t})\), and the simulator’s immediate predictions \((X^{\text{sim}}_{t+1}, V^{\text{sim}}_{t+1})\). The encoder processes these inputs through multiple layers of patch-based self-attention, where each particle attends to others within its local neighborhood. We demonstrate experimentally that the attention mechanism computes position-dependent features for each particle based on its local geometric configuration.

Decoder. To decode per-particle residuals, we utilize a Neural Radiance Field (NeRF) style architecture. We apply Fourier positional encoding to the particle positions and concatenate the resulting embeddings with the features extracted by the encoder. These concatenated features are passed through an MLP decoder to produce an initial velocity correction estimate \(\Delta V^{\text{initial}}_{t+1}\). Empirically, we found that including the decoder improves performance compared with directly using PTv3 outputs as velocity estimates, as discussed in the decoder ablation in App. 7.2.

Temporal Aggregator. To incorporate dynamics from the previous predictions, we refine the base predictions using a sliding-window transformer. At each step \(t\), we update a temporal buffer with the current initial correction \(\Delta V^{\text{initial}}_{t+1}\). If the buffer contains fewer than \(W\) frames, we pad it by replicating the most recent frame. The sequence is projected to a latent embedding, augmented with sinusoidal positional encodings, and processed by the transformer. We extract the output feature vector \(\mathbf{h}_t\) corresponding to the final time step and apply two parallel learned projections: \[\boldsymbol{\delta}_t = \tanh(\text{Proj}_\delta(\mathbf{h}_t)), \quad g_t = \sigma(\text{Proj}_g(\mathbf{h}_t)),\] where \(\text{Proj}\) denotes a linear layer, \(\tanh\) produces a bounded temporal offset \(\boldsymbol{\delta}_t\), and the sigmoid function \(\sigma\) generates gating weights \(g_t\). The final residual velocity is computed as: \[\Delta V^{\text{residual}}_{t+1} = 0.1 \cdot [(1 - g_t) \odot \Delta V^{\text{initial}}_{t+1} + g_t \odot \boldsymbol{\delta}_t],\] where \(\odot\) denotes element-wise multiplication. This gating mechanism allows the network to adaptively blend the local base prediction with the temporally refined correction based on the transformer’s context. This design is similar in spirit to gated recurrent models [88], [89], in that it uses temporal context to decide how much of the current prediction to preserve and how much to replace with a history-based update.

4 Experiments↩︎

4.1 Data Collection↩︎

We evaluate PGRD on six objects shown in Fig. 3 spanning a range of shapes, material properties, and manipulation types.

1. Rope. A mostly one-dimensional object grasped at one end and manipulated by dragging across the table or lifting and lowering motions.

2. Paper. A thin two-dimensional sheet grasped at the top and manipulated by suspending it in the air while performing waving motions, exhibiting bending and twisting.

3. Flag. A cloth flag, demonstrating large area deformation dynamics that are challenging to model due to the waving motion. We chose it because its top stick prevents the cloth’s surface from collapsing. We model the stick as a rigid constraint, assuming that all points along the stick follow the gripper trajectory.

4. Sloth Toy. A three-dimensional volumetric object with long, highly flexible limbs that are challenging to model.

5. Duster. A heterogeneous object with a rigid stem and highly deformable feathers. This is especially challenging because it requires different dynamics predictions for the rigid stem compared to the flexible feathers.

6. Teddy Toy. A volumetric object with higher rigidity than the soft toy. We deform it by poking rather than holding it, representing non-prehensile manipulation.

We collect data using UFACTORY xArm 7 following the procedure from [79]. Four Intel RealSense D455 cameras capture synchronized multi-view observations while the object is manipulated. We segment the object with Grounded SAM 2 [90] and track image points with CoTracker [91]. These 2D tracks are lifted to 3D using depth from all camera views, and an iterative rollout procedure is used to extract persistent correspondences across frames. The resulting dataset consists of temporally consistent 3D point cloud trajectories of the object together with the robot gripper positions over time, obtained from robot proprioception and used as the action sequence. Additional details are provided in App. 10.

For training, we use 100 episodes, and for validation, we use 20 episodes. Each episode spans 4 seconds, but consecutive episodes are generated with a temporal stride of 2 seconds, so neighboring episodes overlap by 2 seconds. As a result, the 100 training episodes correspond to 202 seconds of unique data, or approximately 3.37 minutes, while the 20 validation episodes correspond to 42 seconds, or 0.70 minutes. In total, the dataset contains 120 episodes extracted from 244 seconds of interaction, which is approximately 4.07 minutes of recorded data. The entire data processing is offline and happens before training. We train PGRD with 5-step rollout, but evaluate it over the full evaluation episode length of 37 steps. Hence, this evaluation paradigm tests generalization capability over horizons not seen during training.

4.2 Evaluation Tasks and Metrics↩︎

We evaluate PGRD in two settings. First, we measure how accurately it predicts object motion in 3D using tracking metrics. Second, we evaluate whether the same predicted trajectories produce visually accurate future observations in image space. In both settings, performance is measured over complete validation episode rollouts rather than isolated timesteps. For the visual-space evaluation, we convert the predicted 3D trajectories into 2D video frames using 3D Gaussian Splatting. We initialize a 3DGS representation from the first frame and then update the Gaussians using the predicted particle motion while keeping appearance parameters fixed. This yields action-conditioned video predictions that can be compared directly against ground-truth frames using visual metrics.

Tracking Metrics. We assess rollout accuracy in 3D using three metrics:

• Mean Distance Error (MDE) measures the average distance (in cm) between corresponding points in predicted and ground truth configurations, providing a direct assessment of positional accuracy.

• Chamfer Distance (CD) computes the bidirectional nearest neighbor distance (in cm) between point clouds, capturing coverage and precision without requiring correspondences.

• Earth Mover’s Distance (EMD) quantifies the minimum cost (in cm) of transforming one point cloud into another, providing a holistic measure of distributional similarity.

Visual Metrics. To evaluate action-conditioned video prediction, we render future frames. We then compare these rendered frames against the ground truth using three metrics:

• J-Score (IoU) measures intersection over union of predicted and ground truth masks, quantifying spatial overlap.

• F Score evaluates contour accuracy, determining how well the boundary of the predicted object mask matches the boundary of the ground truth.

• Learned Perceptual Image Patch Similarity (LPIPS) assesses perceptual similarity using features extracted from predicted and ground truth images, capturing visual differences closer to human perception.

4.3 Baselines↩︎

We compare PGRD against five baselines: two analytical physics-based simulations and two learning-based approaches.

Spring-Mass Model [25]: This is the same as the physics backbone we use for our model. Improvements over this baseline demonstrate that the learned residual model effectively captures the dynamics missed by the physics simulator alone.

Material Point Method (MPM) [2] : MPM combines Eulerian and Lagrangian representations to handle material behaviors and topological changes. This hybrid approach discretizes materials into particles while using a background grid for computing spatial derivatives and enforcing conservation laws. MPM is particularly effective for materials exhibiting both solid and fluid-like behaviors but requires careful parameter tuning and can be computationally expensive.

Graph Based Neural Dynamics (GBND) [63] : This is a learned approach that employs Graph Neural Networks to learn object dynamics directly from data without explicit material parameters. The object is represented as a graph where message passing mechanisms propagate information between nodes to capture long range interactions.

Particle-Grid Neural Dynamics (PGND) [79] : This represents the state-of-the-art in learning-based methods, employing a hybrid representation of particles and spatial grids inspired by MPM to model deformable object dynamics. In this method, particles capture the object geometry while the spatial grid discretizes the 3D domain to ensure spatial continuity and improve learning efficiency.

Differentiable (Diff.) Spring-Mass [25]: This extends the spring-mass baseline with a two-stage optimization. It first estimates global parameters as spring-mass baseline, then refines them through gradient-based optimization using a differentiable simulator to learn spatially varying physical parameters. This baseline is motivated by PhysTwin [25], but we implement a variant that builds one canonical spring graph and registers it to each episode’s initial observations before fitting the physical parameters across episodes. While this yields a more expressive spring-mass model, the dynamics remain constrained by the underlying spring-mass formulation.

4.4 Results↩︎

As explained in Section 4.2, we evaluate PGRD in two ways: 3D tracking accuracy and action-conditioned video prediction.

Dynamics and Tracking Accuracy. We first analyze our model’s ability to track and simulate object states over time, with results summarized in Tab. 1. PGRD consistently achieves the lowest error across all objects and metrics, demonstrating that learned residual corrections on physics backbones improve accuracy beyond either pure physics or pure learning approaches. Among physics methods, the optimized spring-mass model generally outperforms MPM, particularly for objects with coherent structure like the rope and sloth toy. Among learning-based methods, PGND substantially outperforms GBND across all objects, indicating that particle representations with physics priors are more effective than graph neural approaches for deformable object simulation. Building on the optimized spring-mass model, Diff. Spring-Mass improves over the optimized spring-mass baseline on most objects, showing that differentiable parameter refinement provides a stronger physics-based model. However, it still falls short of PGRD, whose learned residual corrections can capture dynamics beyond the spring-mass formulation.

Qualitatively, Fig. 4 reveals consistent patterns in how different approaches handle deformation. The optimized spring-mass model captures overall motion but lacks the flexibility to represent local variations in material properties, often appearing too stiff. In our implementation, the MPM baseline often struggles to maintain structural cohesion, with object particles separating rather than behaving as connected solids. Learning-based methods show complementary strengths: PGND generates smooth predictions but accumulates drift over time, while GBND struggles with graph topology, producing unphysical artifacts. Diff. Spring-Mass consistently matches the observed deformation better than the spring-mass model itself, suggesting that differentiable parameter optimization improves the fitted physical response. However, because it is still constrained by the spring-mass formulation, it remains less flexible than our residual model. PGRD leverages its physics backbone for stability while using learned residuals to correct local dynamics, maintaining both plausibility and accuracy throughout the rollout.

The qualitative results in Fig. 4 provide further insights into the limitations of different approaches. In our implementation, the MPM baseline performs poorly on thin objects like paper and flag, causing them to break apart and lose structure. We hypothesize that this behavior arises from the interaction between the grid discretization and the sparse particle representation used for these thin objects. The duster, with its rigid handle and soft feathers, reveals distinct failure modes across all baselines: the optimized spring-mass model cannot assign spatially varying stiffness and causes the rigid stem to bend unnaturally; MPM fails to maintain volume with material points collapsing toward the floor; GBND predicts erroneous upward motion for all feather particles; and PGND incorrectly compresses the rigid stem. Only PGRD simulates the duster correctly. For the teddy, under non-prehensile manipulation, the object slips out of the optimized spring-mass model. The differentiable spring-mass model cannot accurately model the spatially varying parameters for the duster. We do not report MPM results for the Teddy Toy because the evaluated MPM baseline does not model the non-prehensile manipulation required for this task. GBND predicts minimal particle displacement, producing an overly stiff response. PGND shows the gripper’s entire surface in our visualization, but the gripper fingers themselves are not visible because they have pushed into the teddy’s interior. Only our PGRD successfully captures the deformation, properly maintaining contact while allowing realistic compression.

Action Conditioned Video Prediction. Given an initial observation and a sequence of actions, we evaluate whether accurate dynamics predictions translate to realistic future frame generation. We use 3D Gaussian Splatting as the rendering module, where Gaussians are fit to the observed point cloud at the initial timestep and parameterized by position, covariance, opacity, and color. As PGRD predicts future point cloud configurations, we update the Gaussian positions and covariances while keeping opacity and color fixed, allowing the rendered frames to reflect the predicted deformations while preserving appearance consistency with the initial observation. At each timestep, we project the updated Gaussians into the camera view and alpha-blend them in depth order to produce the output image. Fig. 5 shows an example pair consisting of a frame rendered from the dynamics predicted by PGRD and the corresponding ground truth frame, which are used to compute the visual metrics. Among the physics-based methods, Diff. Spring-Mass generally improves over the optimized spring-mass baseline, indicating that differentiable parameter refinement translates to more accurate rendered masks and appearances. Compared with the learning-based method PGND, its performance is mixed: it is better on some objects and metrics, but worse on others. Overall, PGRD consistently achieves the best \(\mathcal{J}\)-Score and \(\mathcal{F}\)-Score across all objects, demonstrating better spatial overlap and foreground detection compared to both physics-based and learning-based baselines. PGRD also maintains the lowest LPIPS scores, indicating that the physics backbone enables the residual model to preserve object appearance during rendering.

5 Applications↩︎

5.1 Planning using PGRD↩︎

In our planning experiments, we use PGRD as a forward model for model-based control: given a state \(s_t\) and an action \(a_t\), it predicts the next state \(s_{t+1}\), where actions specify gripper positions that interact with the object. To plan over this forward model, we integrate PGRD with Model Predictive Path Integral (MPPI) control [92]. At each timestep, MPPI samples a batch of candidate action sequences \(\{\tau_i\}\) spanning a planning horizon \(H\), rolls out PGRD to predict future states, and updates the action distribution toward lower-cost regions. Following standard MPC practice, only the first action is executed before replanning with the newly observed state. We use the Chamfer Distance (CD) between the predicted and a pre-collected target point cloud as the planning objective. The parameters for planning experiments is provided in App. 11.

Standard Manipulation Tasks. We evaluate planning on the rope with four start and goal configurations, as shown in Fig. 6 (top): (1) lifting the rope up, (2) lowering it, (3) rotating it such that one end remains roughly stationary while the other moves, and (4) deforming it into a curved shape. We compare PGRD against the spring-mass backbone, providing both methods with the same MPPI planning budget. We run 10 trials for each task using the same start and end configurations. As shown in Fig. 6, PGRD’s CD decreases smoothly and saturates at a value close to the target, whereas the spring-mass model converges more slowly and sometimes fails to reach a comparable final error. Additional results on the sloth toy are provided in App. 9.

Cable Rerouting. We further test PGRD on a more challenging task: cable rerouting through a narrow slot. This requires the robot to move the rope and thread it through the slot. For these experiments, the residual dynamics model is not trained on examples containing rope-slot interaction data. Hence, this also shows the generalization capability of PGRD. We run 10 trials with different start and end configurations. The target configuration is provided to the planner.

As demonstrated in Fig. 7, PGRD successfully navigates the rope through the slot. The learned residuals compensate for inaccuracies in the physics backbone, enabling precise trajectory execution that avoids collisions with the edges. In contrast, the spring-mass backbone struggles with these fine dynamics. Small errors cause the rope to collide with the edges, forcing the planner to retry the motion multiple times before completing the task.

Quantitatively, Fig. 8 shows the distributions of start and end configurations overlaid (left) and comparisons of chamfer distances (right). PGRD achieves significantly lower CD than the optimized spring-mass. We also evaluate success rates for this task. A trial is considered successful if the rope passes entirely through the slot and the gripper remains below the slot height in the final configuration. Under this criterion, PGRD succeeds in 8/10 trials, whereas the spring-mass model succeeds in only 2/10 trials. These results demonstrate that the improved model enables more reliable execution of challenging manipulation tasks.

Language-Conditioned Planning. The planning experiments above require a target point cloud, so it must be collected beforehand. To relax this requirement, we integrate PGRD with a language-conditioned goal-generation pipeline, enabling the planner to operate solely on language commands.

Given an initial RGB image and a language command (e.g., “pass rope through red slot”), we use Nano Banana Pro [93] to generate a goal image. Note that here we use a single camera rather than the four in the standard planning experiments, since the goal image can be generated more reliably for a single viewpoint. To estimate depth from the generated image, we use Depth Anything V2 [94]. Following Patel et al. [95], we resolve the scale and shift ambiguity inherent to monocular depth estimation by fitting an affine transform to align the predicted depth to the initial depth map from the depth camera. We restrict the alignment to background table points, which remain consistent in both images. Since the object appears in a different configuration in the goal image than in the initial image, including object points for this alignment would degrade the quality. The resulting depth map is unprojected into 3D with known camera parameters to obtain the target point cloud, which is passed directly to the MPPI planner.

To determine where to grasp the object, we use AnyGrasp [96] to generate grasp candidates on the initial point cloud. For the rope, we select the candidate closest to the right end. For the sloth toy, the language command implies a specific region to manipulate, which we identify by extracting DINOv2 [97] features from both the initial and goal images to establish dense correspondences. We locate the object point with the largest displacement between the two images and select the AnyGrasp candidate closest to that point.

Fig. 9 shows real-world execution for multiple tasks. Beyond removing the need to physically collect the goal configuration, this pipeline opens up a richer space of goals, including spatial references such as “pass rope through closer slot” and “lift right arm” of the sloth toy.

5.2 Interactive Photo-Realistic Simulation.↩︎

As shown in Fig. 10, PGRD supports interactive, photo-realistic simulation by combining its predicted dynamics with 3D Gaussian Splatting. Users issue manipulation commands via keyboard input; at each step, the physics backbone propagates the object state, the learned residual network applies per-particle corrections, and the updated Gaussians are rendered to produce a photo-realistic view of the resulting deformation. Because only Gaussian positions are updated while appearance attributes remain fixed, the rendering maintains visual consistency with the initial observation throughout the interaction.

6 Conclusions↩︎

This paper presented Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that bridges physics-based and learning-based approaches for deformable object simulation. By combining an optimized spring-mass backbone with learned velocity residuals, PGRD achieves superior accuracy while maintaining physical plausibility. Our velocity-based residuals, combined with a sliding-window transformer for temporal aggregation, yield stable predictions, avoiding the instability issues plaguing naive residual learning. Extensive experiments across real-world objects demonstrate that PGRD consistently outperforms a variety of SOTA methods. PGRD handles challenging scenarios involving heterogeneous materials, volumetric deformations, and contact-rich interactions where existing methods struggle. Beyond tracking, PGRD also shows promise for applications like action-conditioned video prediction, planning, and interactive simulation.

We structure the supplement into the following sections:

• Ablation study of the main components of PGRD, including the encoder, decoder, temporal aggregator, gating, and internal particles for volumetric objects.

• Network architecture details and hyperparameter settings for the encoder, decoder, and temporal aggregator.

• Planning experiments on the sloth toy, showing PGRD’s behavior on a volumetric object with long flexible limbs.

• Data collection and processing pipeline for extracting temporally consistent particle trajectories from multi-view RGBD observations.

• MPPI planning hyperparameters used for the simple tasks and the cable rerouting experiments.

7 Ablation Study↩︎

5pt

We provide a detailed ablation study of the main components of PGRD. We consider both replacement ablations, which test natural alternatives to our design, and removal ablations, which isolate the role of individual components. All quantitative results are summarized in Tab. 3. We report only MDE, and the other metrics follow the same trends.

7.1 Encoder choice↩︎

We use PTv3 as our encoder. We compare this design to a PointNet++ [98], which is a natural baseline because it is a standard point cloud architecture and also used in PGND. This comparison tests whether the stronger local feature extraction of PTv3 is important for residual prediction. The trend in Tab. 3 indicates that it is.

7.2 Decoder↩︎

In Sec. 3.3, we describe a NeRF-style decoder that combines encoder features with positional encoding to predict per-particle residual velocities. A natural simplification is to remove this decoder and directly project the encoder features to residual velocities. This tests whether explicit position-conditioned decoding remains useful once the encoder features are already available. The results in Tab. 3 suggest that it does.

7.3 Temporal aggregator↩︎

The temporal aggregator refines the current residual estimate using a sliding window of previous predictions. This design is motivated by the fact that many deformation effects depend on recent motion history rather than on the current configuration alone. The corresponding ablation removes temporal refinement while keeping the encoder and decoder unchanged. The degradation in Tab. 3 supports the need for temporal context in predicting residual corrections.

7.4 Gated temporal refinement↩︎

Our temporal module uses gating to blend the decoder prediction with the temporally refined correction. A natural alternative is direct addition. This ablation isolates whether the gate is important beyond the temporal model itself. As shown in Tab. 3, the gate is useful for turning temporal information into a stable refinement rather than an uncontrolled update.

7.5 Internal particles for volumetric objects↩︎

For volumetric objects, the spring mass backbone includes internal particles so the particles don’t collapse. This component is relevant only to the sloth and the teddy toy. The corresponding ablation removes the internal particles while keeping the rest of the system unchanged. The resulting drop in Tab. 3 is consistent with the role of internal particles in preventing unrealistic collapse for volumetric objects.

8 Network Architecture Details↩︎

Our network network consists of three primary components: an encoder, a decoder, and a temporal aggregator. The hyperparameter values for each component are provided in Table 4. The encoder uses a Point Transformer V3 (PTv3) [87] architecture to extract per-particle features through patch-based self-attention, where the feature dimension determines the size of the latent representation. The input channels correspond to the concatenated position and velocity information from the current state, history timesteps, and the physics simulator predictions. The decoder employs a conditional NeRF-style MLP to map spatial locations to residual velocities, where hidden layers refers to the number of fully connected layers, hidden dimension specifies the width of each layer, and output channels indicates the dimensionality of the predicted residual velocity (3 for xyz components). The decoder takes as input the per-particle features from the encoder, concatenated with Fourier positional encodings of particle positions. The temporal aggregator employs a transformer encoder to refine predictions using temporal context. Here, the embedding dimension defines the size of the latent space for temporal features, attention heads specifies the number of parallel attention mechanisms, transformer layers indicates the depth of the transformer stack, and feedforward dimension determines the width of the intermediate feedforward network within each transformer layer. Finally, the gating scale controls the magnitude of temporal corrections applied to the base velocity predictions, preventing large adjustments that could destabilize the simulation.

9 Planning on Sloth Toy↩︎

We extend our planning experiments to the sloth toy, a volumetric object with long, flexible limbs. It requires accurate prediction of both volumetric deformation and limb dynamics, testing the framework’s ability to handle three-dimensional objects. Similar to rope, we design four manipulation objectives with varying complexity, as visualized in Fig. 11 (top): (1) lifting the object, (2) lowering it, (3) rotating it, and (4) bending it into a curved configuration. Following the same experimental setup, we conduct 10 trials per objective with identical initial and target states, allocating equal computational budget to both PGRD and the spring-mass baseline. The results in Fig. 11 demonstrate that PGRD exhibits consistent convergence behavior, reaching target configurations with reasonable residual error. The spring-mass baseline shows slower error reduction and occasionally plateaus at suboptimal solutions, particularly for tasks involving complex limb deformation, where the learned residuals provide crucial corrections to the physics backbone predictions.

10 Data Collection and Processing↩︎

We mount four Intel RealSense D455 cameras around the workspace to capture synchronized RGBD observations at 30 Hz. The cameras are calibrated to a shared world coordinate frame using a checkerboard calibration procedure, enabling the fusion of observations across views. For each camera view, we apply Grounded SAM 2 to extract object masks. Using text prompts containing object descriptions, it detects and segments the object in the first frame of a sequence and propagates the mask across subsequent frames. We employ CoTracker [91] as the tracking model due to its stable tracking performance. It predicts a set of 2D trajectories, initialized from uniformly sampled grid locations within the first-frame object mask. For each pixel in the segmented region at time \(t\), we compute its 2D displacement to time \(t+1\) from the predicted trajectories, and convert it to a 2D velocity by dividing by the time step.

For each camera view, we lift the 2D pixel velocities to 3D using the corresponding depth measurements. Specifically, for a pixel at position \((u, v)\) with depth \(d\) and 2D velocity \((v_u, v_v)\), we compute the 3D velocity by back-projecting both the current and next pixel positions to 3D coordinates using camera intrinsics, then computing the difference. This yields per-pixel 3D velocities in the camera frame, which we transform to the world frame using the calibration parameters. We aggregate velocities across all camera views by averaging the 3D velocity estimates for overlapping spatial regions, thereby improving robustness to noise and partial observations from individual views.

Given the 3D point cloud with per-point velocities at each timestep, we extract temporally consistent particle trajectories using an iterative rollout procedure. Starting from frame 0, we initialize particles at the point cloud positions \(\{x_i^0\}\). For each subsequent timestep \(t\), we propagate particles forward using their current velocities: \(x_i^{t+1} = x_i^t + v_i^t \cdot \Delta t\). To update velocities at the new positions, we perform k-nearest neighbor search (with \(k=5\)) between the propagated particle positions and the observed point cloud at time \(t+1\). Each particle’s velocity is updated to the mean velocity of its k nearest neighbors in the observed cloud. This iterative process maintains correspondence across frames while being robust to tracking noise, yielding persistent point tracks suitable for training our dynamics model. The entire processing pipeline runs at approximately 2 Hz for sequences with 1000 particles.

11 MPPI Planning Parameters↩︎

We use different set of parameters for simple and cable rerouting experiments. They are provided in Tab. 5.

References↩︎