1 Introduction↩︎

Recovering physical properties of objects from video, such as material stiffness and deformation behavior, is important for robotic manipulation of deformable objects, as well as building digital twins and virtual environments [1]. Recent advances in differentiable physics simulation [2], [3] and neural 3D representations [4], [5] have enabled inverse physics: recovering physical parameters by optimizing through differentiable simulations to match observed motion. Existing inverse physics methods [6]–[10] typically rely on synchronized multi-view captures, where geometric constraints across views resolve 3D structure and absolute scale. However, multi-view rigs are unavailable for the most common sources of dynamic video, including most in-the-wild captures. Enabling physical system identification from monocular video would unlock these abundant data sources for downstream applications without requiring specialized hardware.

These multi-view methods [6]–[10] rely on triangulation to resolve 3D structure and absolute scale prior to physical parameter estimation. The physics optimization therefore operates on geometry that visual evidence has already constrained.

The monocular setting is significantly more challenging, as geometry must be inferred without multi-view constraints and is inherently ambiguous in scale. While recent 3D foundation models [11] can recover plausible geometry from a single image and provide a strong starting point, their outputs remain unreliable in occluded regions and lack absolute scale, both critical for physical simulation. Prior inverse physics methods typically decouple geometry reconstruction and physical parameter estimation, assuming a reliable 3D representation can be obtained from multi-view inputs [6]–[9]. In the monocular setting, this decoupling breaks down, often requiring joint optimization of geometry, appearance, and physical parameters.

We present MonoPhysics, a framework for monocular inverse physics of general deformable objects based on differentiable MPM simulation and 3D Gaussian Splatting, initialized from a 3D foundation model [11] and jointly refined under physical and visual supervision. We use 3D Gaussians for both rendering and simulation, where each Gaussian serves as a visual rendering primitive explaining pixel observations and a physical MPM particle with 3D position, volume, velocity, and deformation. The inverse physics pipeline has a visual side and a physical side, coupled by differentiable rendering and simulation. In the monocular setting we identify several connections between them that remain weak or missing, preventing image cues from fully constraining the underlying physical parameters in the absence of multi-view constraints.

We propose three bridges that fill in paths missing from the standard pipeline. First, because per-particle position updates produce only local corrections, a single learnable scalar aggregates a coherent global scale signal across all particles, coupling the 3D visual representation to physical-space dynamics so that observed motion can resolve absolute scale. Second, physics-aware geometry refinement computes per-particle volumes from the current particle distribution and incorporates both visual and physical importance during Gaussian relocation, allowing the geometry to adapt to simulation feedback rather than remain frozen at initialization. Third, we propose a differentiable position map that gives Gaussians a direct gradient path from image-space pixel-location losses such as silhouette alignment; standard 3DGS rendering only carries gradients through colors and opacities, so position-based supervision (which is essential when predicted and target shapes do not yet overlap) would otherwise be unavailable. We optimize geometry, appearance, and physical parameters using a combination of rendering loss, optical flow loss, silhouette loss, and particle distribution regularization. We evaluate MonoPhysics on the Vid2Sim benchmark [10] and compare against existing inverse physics methods [6]–[8], [10]. Existing methods degrade in the monocular setting, while our approach consistently outperforms baselines in both future prediction and physical parameter estimation, achieving physical parameter accuracy on Vid2Sim comparable to multi-view setups, using only a single camera. We further test on a new dataset with both elastic and plastic objects under diverse shapes, viewpoints, and motion conditions, where our method demonstrates strong performance.

2 Related Works↩︎

2.1 Inverse Physics from Video↩︎

Estimating physics from video sequences has been studied extensively [6], [12]. Differentiable simulation frameworks [2], [3] enable gradient-based optimization of physical parameters by differentiating through the simulation. This allows observed object behavior to directly supervise material properties, initial velocities, and other physical parameters. PAC-NeRF [6] combined differentiable MPM simulation with NeRF to optimize geometry and physical parameters from multi-view video. SpringGaus [7] integrates a spring-mass model into 3D Gaussian Splatting for reconstruction and simulation of elastic objects. GIC [8] applies point-wise losses on particles from a learned 4D representation to estimate physical parameters. Vid2Sim [10] uses feed-forward prediction for initialization followed by lightweight optimization with a mesh-free differentiable simulator [13]. MASIV [9] introduces learnable neural constitutive models [14] that remove the need for hand-crafted constitutive laws, enabling material-agnostic system identification. These methods benefit from multi-view constraints or direct 4D guidance, which reduces the problem to estimating physical parameters from an already-resolved geometry. In the monocular setting, these constraints are absent.

2.2 Monocular and Sparse-View Inverse Physics Estimation↩︎

Monocular video poses unique challenges for inverse physics. Without multi-view constraints, scale is ambiguous and geometry inaccurate, with limited appearance signals for supervision. Prior monocular or limited-view methods partially address these issues. ProJo4D [15] improves parameter estimation and prediction in sparse-view settings, but still relies on multi-view input. NeuPhysics [16] learns a 4D representation from monocular video and estimates physical parameters from reconstructed trajectories, but separates representation learning from parameter estimation, assuming accurate 4D reconstruction. PPR [17] resolves scale for articulated bodies, but relies on priors not applicable to general deformable objects. Gao et al. [18] estimate physical parameters of thin deformable objects from monocular video, assuming full surface visibility in the first frame. FluidNexus [19] reconstructs and predicts fluid dynamics from a single video, but is limited to fluids.

2.3 Physics-Informed Scene Understanding↩︎

Physical constraints and priors have been used to improve static scene reconstruction or infer structure. Physics-Informed Deformable Gaussian Splatting [20] uses physics as regularization for monocular 4D reconstruction and learns time-varying material parameters, but does not perform full inverse estimation with forward prediction. Our framework directly addresses monocular-specific challenges, including ambiguous scale and inaccurate geometry, for general deformable objects using physics-derived signals without object-specific priors or multi-view constraints. PhyRecon [21] integrates differentiable rendering with particle-based physics simulation to produce physically stable static scene reconstructions. TopoGaussian [22] infers internal topology from multi-view images. Structure from Collision [23] estimates interior structure from appearance changes during collision. PhySIC [24] uses physical contact constraints for human-scene alignment from a single image. Guo et al. [25] optimize for physical compatibility to obtain plausible static objects from a single image. BrickGPT [26] uses physical stability constraints to produce buildable brick structures. However, these methods either operate on static scenes or require multi-view input; our work uses physics as a corrective signal specifically for monocular inverse estimation of dynamic deformable objects.

3 Method↩︎

We first formulate the problem overview (3.1). Our three contributions are bridges across the three layers of the inverse-physics pipeline introduced in 1: a global scale scalar (3.2), physics-aware geometry refinement (3.3), and a differentiable position map (3.4). Loss functions and the full optimization procedure follow in 3.5.

3.1 Problem Formulation and Overview↩︎

The input is a monocular RGB video of a deformable object interacting with its environment, e.g., an object falling onto a table. We assume known camera parameters (intrinsics and extrinsics), a known ground plane, and a known constitutive model (e.g., Neo-Hookean elasticity), consistent with prior inverse physics methods [6]–[8]. We also require a foreground alpha mask for each frame; when ground-truth masks are unavailable, an off-the-shelf segmentation method can be used. Our goal is to jointly recover the object’s geometry, appearance, and material parameters from these inputs. For 3D representation, we use 3D Gaussian Splatting (3DGS) [5], which integrates with particle-based simulations such as the Material Point Method (MPM). Each Gaussian has position \(\mathbf{x}_i\), color \(\mathbf{c}_i\), opacity \(o_i\), and covariance \(\Sigma_i\) for appearance, and physical volume \(V_i\). The initial physical state is represented by the initial velocity \(\mathbf{v}_0\).

We use an off-the-shelf monocular 3D reconstruction method [11] to produce an initial set of 3D Gaussians from the first frame, then sample MPM particles from them. We use a differentiable MPM simulator [2] to advance them through time under the current physical parameters, producing deformed representations at each frame. At each frame, the color of every pixel \(C\) is rendered via 3D Gaussian Splatting using alpha compositing [5]: \[\label{eq:3dgs95render} C = c_{bg}\prod_{i}\bigl(1 - \alpha_i\bigr) + \sum_{i} \alpha_i\,\mathbf{c}_i \prod_{j < i}\bigl(1 - \alpha_j\bigr),\tag{1}\] where \(\mathbf{c}_i\) is the color of Gaussian \(i\), \(\alpha_i\) is the product of the learned opacity and the Gaussian response of particle \(i\) at the pixel, and \(c_{bg}\) is the background color of that pixel. Losses are backpropagated through the differentiable simulation to update all parameters jointly, as detailed in 3.5.

3.2 Scale Estimation through Reparameterization↩︎

Monocular 3D reconstruction recovers scene geometry only up to an unknown scale factor, as points along the same projection ray are indistinguishable from the camera’s perspective. For physics, however, absolute scale governs world-space positions, velocities, contact timing, volumes, and stress magnitudes, so all downstream physical quantities depend on it. Scale error therefore propagates into every estimated parameter.

Physical dynamics can resolve this ambiguity. However, naively optimizing per-particle positions cannot recover the global scale. Each particle’s gradient mixes local position corrections with the global scale signal, so independent per-particle updates produce local adjustments rather than a coherent change in the object’s distance from the camera (1).

Existing inverse-physics methods store particles directly in world space [6]–[9]. We instead maintain particles in camera space and introduce a single learnable scalar \(s\) that controls the global scale. Scaling in camera space moves particles along their projection rays, leaving 2D projections unchanged but changing all physical quantities. Since physics simulation requires world-space coordinates, we transform using the known camera extrinsics: \[\begin{align} \label{eq:c2w} \mathbf{x}_i^w &= \mathbf{R}\,(s \cdot \mathbf{x}_i) + \mathbf{t}, \end{align}\tag{2}\] where \(\mathbf{R}, \mathbf{t}\) are the camera-to-world rotation and translation, \(\mathbf{x}_i\) is the camera-space position. Because every particle’s physics depends on \(s\), image-space losses backpropagated through the differentiable simulation aggregate a coherent global signal for scale from all particles and all frames. This gives \(s\) a clean optimization path that per-particle degrees of freedom cannot provide (4.3 and 1).

3.3 Physics-Aware Geometry Refinement↩︎

Accurately estimating geometry from a single camera is difficult: unobserved regions remain inherently ambiguous even with 3D geometric priors from foundation models. Our key idea is that a differentiable physics simulation (MPM) provides complementary signals to refine geometry, since physical behavior constrains regions that visual observation alone cannot resolve. In neural field-based representations [6], [23], [27], geometry can be refined by optimizing continuous density or occupancy fields. Particle-based representations lack this continuous structure, so geometry is instead captured through per-particle volumes.

Per-particle physical volume. Existing works that incorporate physics into 3DGS representations [8], [9] assign fixed per-particle volumes, computed once at initialization. In standard MPM initialization, these volumes are chosen to satisfy a local partition-of-unity over the simulation grid: the total volume contributed by particles to each grid node matches the cell volume \(\Delta x^3\). More specifically, the total volume contributed to each grid node equals the cell volume \(\Delta x^3\): \[\label{eq:partition95of95unity} \sum_i V_i w_{i\ell} = \Delta x^3,\tag{3}\] where \(\Delta x\) denotes cell edge size, and \(w_{i\ell}\) is the contribution of particle \(i\) to grid node \(\ell\). During initialization, these per-particle physical volumes are determined. However, keeping volumes fixed throughout optimization prevents the representation from adapting its geometry and total volume.

We instead make volumes adaptive by maintaining the partition-of-unity (3 ), inducing volumes from the current particle distribution at each step. We compute them by iteratively refining per-particle volume to meet the partition of unity for \(T\) iterations. 6 in supplementary shows the algorithm.

Physics-informed particle management. Adaptive volumes alone cannot add particles where geometry is under-resolved, nor remove particles wasted on negligible regions. We therefore complement adaptive volumes with a particle management scheme that relocates particles during optimization. Prior visual-only methods select particles using visual importance only [5], [28], [29]; we propose using physical importance in addition to visual importance. We define a visual importance \(p_i^{vis}\) from the rendering footprint, a physical importance \(p_i^{phys}\) from the physical volume, and the overall importance \(p_i\) as the average of the two: \[p_i \propto \tfrac{1}{2}p_i^{vis} + \tfrac{1}{2}p_i^{phys}, \quad p_i^{vis} \propto \det(\Sigma_i)^{1/2} \cdot o_i, \quad p_i^{phys} \propto V_i,\] where \(\Sigma_i\) is the 3D covariance matrix and \(o_i\) is the learned opacity of particle \(i\). We periodically remove particles with the lowest visual importance \(p_i^{vis}\) or physical importance \(p_i^{phys}\), and replace them by splitting those with the highest overall importance \(p_i\), similar to 3DGS-MCMC [29]. This steers particle redistribution toward regions that are under-resolved either visually or physically.

3.4 Differentiable Position Map↩︎

Aligning a rendered shape to its target requires gradients that pull misplaced particles toward the correct silhouette. Standard image-space losses provide such gradients only indirectly and only where predicted and observed shapes already overlap: rendering losses act through colors and opacities at overlapping pixels, and optical flow supplies inter-frame motion cues that still rely on a roughly aligned reference shape. Particles that are globally misaligned therefore receive no directional gradient from either loss, a limitation that becomes pronounced under large deformations, where global cues are essential.

We bridge this gap by making each pixel position a differentiable function of the Gaussians that affect that pixel’s values, either color or opacity. By writing each position relative to the covering Gaussian’s image-space mean, we introduce the missing gradient path that enables shape supervision. Let \(\mathbf{x}_i^{2D}\) and \(\boldsymbol{\Sigma}_i^{2D}\) denote the 2D image-space mean and covariance of Gaussian \(i\). For pixel \(\mathbf{p}\) covered by Gaussian \(i\), we define the reparameterized position: \[\label{eq:reparam} \tilde{\mathbf{p}}_i = \mathbf{x}_i^{2D} + (\boldsymbol{\Sigma}_i^{2D})^{1/2}z, \quad z = \mathrm{sg}\left((\boldsymbol{\Sigma}_i^{2D})^{-1/2}(\mathbf{p} - \mathbf{x}_i^{2D})\right),\tag{4}\] where \(\mathrm{sg}(\cdot)\) is stop-gradient, treating its argument as a constant during differentiation. We render a position map using the same alpha-compositing formula as in 1 , substituting \(\tilde{\mathbf{p}}_i\) for per-Gaussian color: \[\label{eq:posmap} \tilde{\mathbf{p}} = \mathbf{p}\prod_{i}\bigl(1 - \alpha_i\bigr) + \sum_{i} \alpha_i\,\tilde{\mathbf{p}}_i \prod_{j < i}\bigl(1 - \alpha_j\bigr),\tag{5}\] where \(\tilde{\mathbf{p}}\) is the output pixel coordinates. Numerically, \(\tilde{\mathbf{p}}\) always equals the actual pixel coordinate \(\mathbf{p}\). This formulation lets pixel-location losses propagate positional gradients to Gaussian parameters.

3.5 Loss Functions and Optimization↩︎

We introduce the loss terms used across the whole optimization stages.

Image losses (\(\mathcal{L}_{\text{color}}\), \(\mathcal{L}_{\alpha}\)). We use standard image losses (L1 + SSIM) for \(\mathcal{L}_{\text{color}}\), and L1 loss between the rendered opacity map and the ground-truth alpha mask for \(\mathcal{L}_{\alpha}\).

Optical flow loss (\(\mathcal{L}_{\text{flow}}\)). Optical flows can provide which parts should move and in which direction. Several prior works have similarly used optical flow to supervise motion estimation in differentiable rendering and inverse physics pipelines [20], [30], [31]. We render predicted flow by splatting per-particle world-space velocities into the image plane using the same Gaussian rasterizer. Optical flow estimation can be noisy and uncertain, especially when physical dynamics change what is visible between frames. We therefore use a probability-based optical flow loss [32]; the detailed parameterization is given in the supplementary. To capture both global displacement and instantaneous motion, we supervise with two complementary optical flow signals: flow from the first frame to the current frame, and flow from the previous frame to the current frame: \[\label{eq:flow} \mathcal{L}_{\text{flow}} = - \frac{1}{|\Omega_t|} \sum_{j \in \Omega_t}\Big[ \log p\left(\mathbf{f}_{0 \to t}(\mathbf{p}_j) \mid \hat{\mathbf{f}}_{0 \to t}(\mathbf{p}_j)\right) + \log p\left(\mathbf{f}_{t\text{-}1 \to t}(\mathbf{p}_j) \mid \hat{\mathbf{f}}_{t\text{-}1 \to t}(\mathbf{p}_j)\right)\Big],\tag{6}\] where \(\mathbf{p}_j\) is the position of pixel \(j\), and \(\Omega_t\) is the set of ground-truth foreground pixels at frame \(t\). \(\hat{\mathbf{f}}_{i \to j}(\cdot)\) and \(\mathbf{f}_{i \to j}(\cdot)\) denote the rendered and estimated optical flow respectively from frame \(i\) to frame \(j\) at pixel \(\mathbf{p}\). The first term anchors global displacement to the reference frame (\(t=0\)), while the second term supervises instantaneous velocity.

Silhouette loss (\(\mathcal{L}_{\text{sil}}\)). Using the differentiable position map (3.4), we obtain rendered pixel positions \(\tilde{\mathbf{p}}_k\) that are differentiable with respect to Gaussian parameters. We then treat each foreground region as a weighted point cloud of these positions and align it to the target using optimal transport, which provides global directional gradients even when the predicted and target silhouettes do not overlap. Let \(\Omega_t\) denote the set of target foreground pixels at frame \(t\), with per-pixel alpha values \(\{a_j\}_{j \in \Omega_t}\), and let \(\hat{\Omega}_t\) denote the set of active rendered foreground pixels, with rendered opacity values \(\{\tilde{a}_k\}_{k \in \hat{\Omega}_t}\). Since the number of pixels and their alpha values might differ, we use unbalanced debiased Sinkhorn divergence [33] as a loss: \[\label{eq:sil95meas} \mathcal{L}_{\text{sil}}^t = \mathrm{SD}(\{\tilde{\mathbf{p}}_k,\, \tilde{a}_k\}_{k \in \hat{\Omega}_t},\, \{\mathbf{p}_j,\, a_j\}_{j \in \Omega_t}),\tag{7}\] where \(\mathrm{SD}(\cdot, \cdot)\) is the debiased Sinkhorn divergence and \(\mathbf{p}_j\) is the GT pixel position. The resulting gradients indicate how each rendered pixel \(\tilde{\mathbf{p}}_k\) should shift spatially and adjust its opacity \(\tilde{a}_k\) to match the target silhouette.

Particle distribution regularizer (\(\mathcal{L}_{\text{distr}}\)). With trainable particle positions, the local distribution directly affects inverse physics estimation. Particles may detach from the body and drift independently, or cluster together and leave voids in their vicinity, both of which degrade the estimation of physical parameters. To mitigate these failure modes, we regularize the spatial distribution via a K-nearest-neighbor penalty: \[\label{eq:connectivity} \mathcal{L}_{\text{distr}} = \sum_{i=1}^{N}\sum_{j \in \mathcal{N}_{K}(i)} \max\bigl(0,\, \|\mathbf{x}_i - \mathbf{x}_j\| - \tau_{\max} \Delta x\bigr) + \max\bigl(0,\, \tau_{\min} \Delta x - \|\mathbf{x}_i - \mathbf{x}_j\|\bigr),\tag{8}\] where \(\mathcal{N}_{K}(i)\) are the \(K\) nearest neighbors of particle \(i\), \(\tau_{\min}\) and \(\tau_{\max}\) are lower and upper bounds in the unit of simulation cell size \(\Delta x\). The first term pulls a particle toward its neighbors when they drift too far apart, while the second pushes neighbors apart when they become overly clustered. Both terms vanish in well-distributed regions, avoiding over-regularization of the particle distribution and the underlying geometry. Since each Gaussian can undergo a large deformation during simulation, we apply this regularizer only at the initial configuration (frame \(0\)), before any deformation occurs.

Optimization strategy. Starting from initial 3D geometry obtained from an image-to-3D model [11], our optimization proceeds in two stages. In the first stage, we jointly optimize all physics parameters (material parameters and initial velocity), Gaussian parameters (positions \(\mathbf{x}_i\), opacity \(\mathbf{o}_i\), and covariance \(\Sigma_i\)), and global scale \(s\) (3.2) via differentiable MPM simulation, while not optimizing colors \(c_i\). Following GIC [8], we focus on physical dynamics in this stage and refine appearance later. Within each iteration, after backpropagating through the simulation, we take a small number of alpha-map loss steps without re-running the simulation. This keeps the visual representation consistent with the evolving geometry and prevents appearance and geometry from drifting apart. We use all four losses as follows: \[\label{eq:total95loss951} \mathcal{L} = \sum_{t=0}^{T}\left[ \mathcal{L}_{\alpha}^t + \mathcal{L}_{\text{sil}}^t\right] + \sum_{t=1}^{T}\mathcal{L}_{\text{flow}}^t + \mathcal{L}_{\text{distr}}.\tag{9}\] In the second stage, we only refine appearance, using both image \(\mathcal{L}_{\text{color}}\) and alpha \(\mathcal{L}_{\alpha}\) losses. Details are in the supplementary.

4 Experiments↩︎

4.1 Experimental Settings↩︎

Baselines. We compare our results against existing inverse physics methods: PAC-NeRF [6], SpringGaus [7], and GIC [8]. For comparison, we also included Vid2Sim [10] and use it as a baseline. Since Vid2Sim requires multiple views and can not operate with a monocular video input, we only report Vid2Sim’s performance with multiple cameras, excerpted from their paper.

Datasets. We evaluate on two datasets. We first evaluate on the Vid2Sim [10] dataset (elastic objects with fixed camera placement, no initial velocity), using only the first camera to simulate a monocular setting. We introduce a new synthetic dataset built from Google Scanned Objects (GSO, CC-BY 4.0) [34], comprising 5 elastic (Neo-Hookean) and 5 plasticine objects with ground-truth point clouds, randomly sampled material parameters, initial velocities, object sizes, and camera distances. Full dataset construction details, including the scene list and constitutive models, are provided in 9. Including two material models demonstrates that our improvements are not specific to elastic objects.

Metrics. We report two groups of metrics. For future prediction, we use PSNR, SSIM, and LPIPS evaluated from the same camera viewpoint (to avoid conflating geometric misalignment with missing texture in novel views), and Chamfer Distance (CD) and Earth Mover’s Distance (EMD) for geometric accuracy. CD is computed as the bidirectional Chamfer distance between predicted MPM particles and the ground-truth point cloud, averaged over future-prediction frames. For physical parameter estimation, we follow the standard evaluation protocol in this domain [6]–[8]: MAE of initial velocity, log Young’s modulus \(E\), Poisson’s ratio \(\nu\), and yield stress \(\sigma_y\) (for plasticine).

4.2 Comparison↩︎

Vid2Sim benchmark. 1 and 2 show results on Vid2Sim. Among single-camera baselines, our method achieves the best rendering quality (PSNR) and the lowest material parameter error (MAE \(\log E\)). Notably, our single-camera MAE \(\log E\) and MAE \(\nu\) are both on par with multi-view Vid2Sim using 12 cameras, indicating that our visual-physical bridges allow physical parameter recovery from a single view at a level comparable to dense multi-view setups.

Our synthetic benchmark. 2 and 3 provide results on our synthetic dataset, which contains both elastic and plasticine objects with diverse material properties, sizes, and initial conditions. Our method achieves the lowest CD for both material types, indicating substantially more accurate geometry recovery. For material parameters, it reduces the MAE for Poisson’s ratio on elastic objects, and reduces the MAE for both Young’s modulus and yield stress on plasticine. Rendering quality is competitive: PSNR is comparable to or higher than baselines on elastic, while on plasticine, GIC achieves slightly higher PSNR but at substantially higher geometric error, suggesting appearance fits the observed view without recovering accurate 3D structure. Initial-velocity MAE is on par with baselines. SpringGaus material parameters are omitted as it uses a spring-mass simulator with parameters not directly comparable to the elastic and plasticine constitutive models.

These improvements stem from resolving scale ambiguity and jointly optimizing geometry alongside physical parameters. Fig. 3 visualizes these results qualitatively: objects reconstructed with our method align better with expected deformation patterns and maintain realistic contact with the ground, whereas baselines often produce incorrectly scaled or distorted reconstructions.

4.3 Ablation Study↩︎

Loss Functions.

Tab. 3 isolates the contribution of each image-space loss and the distribution regularizer from 3.5 on the plasticine subset of our synthetic dataset. Adding the distribution regularizer to the baseline substantially reduces CD, confirming that without it, disconnected particles simulate independently and distort future prediction. The absence of positional guidance from optical flow or silhouette supervision leaves the optimization without sufficient constraints to recover accurate trajectories or material parameters. Optical flow supervision alone has little effect on future-prediction accuracy but significantly improves material parameter estimation (MAE \(\log E\) and MAE \(\log \sigma_y\)). This is consistent with its role: flow provides per-particle motion targets that constrain the physical dynamics, but trajectory accuracy in CD also requires correct shape alignment. Silhouette supervision alone produces the opposite pattern (note that \(\mathcal{L}_{\text{sil}}\) requires the differentiable position map 3.4): it drives a large geometry improvement but contributes less to material estimation compared to flow. This reflects the silhouette loss’s role as a global shape alignment signal rather than a motion supervisor. Combining both losses yields the best option overall, indicating that flow and silhouette supervision are complementary: flow anchors the physical trajectory while silhouette aligns object shape, and accurate geometry in turn benefits material estimation.

Geometry Refinement.

We first illustrate how physics-aware geometric refinement improves the reconstruction quality of the initial geometry in the first frame. To isolate geometric accuracy from scale ambiguity, both the initial geometry and our refined geometry are first aligned to the true global scene scale, and then compared against the ground-truth point cloud using Chamfer Distance (CD). Tab. 4 illustrates the effect on a representative scene, where refinement reduces both initial CD and downstream future CD and MAE \(\log E\). This improvement is consistent across our synthetic dataset: aggregated results show that geometric refinement reduces average initial CD from 11.41 to 7.23, confirming this is a systematic effect rather than a single-scene artifact. Fig. [fig:error95map-n] visualizes the per-region geometric error before (top row) and after (bottom row) refinement, showing reduced error across multiple views.

Differentiable Position Map. To illustrate how the differentiable position map provides a gradient pathway for direct positional guidance, we visualize the resulting gradients in 5. The first two panels show the target and current rendered images, while the third renders the positional gradient of each Gaussian, \(-\partial \mathcal{L}_{\text{sil}}/\partial \mathbf{x}_i\), as an image. The color of the gradient map indicates that the loss correctly guides the Gaussians toward the target direction (to the left, red). We refer the reader to 8 for further details.

5 Conclusion↩︎

We present MonoPhysics, a framework for monocular inverse physics of general deformable objects that jointly recovers geometry, appearance, and material parameters from a single-camera video. Our key insight is that scale, geometry, and material are deeply entangled in the monocular setting and must be solved jointly. To this end, we propose three methods for that: a global camera-space scale, physics-aware particle management, and a differentiable position map. Together, these contributions yield more accurate physical parameter estimation and future prediction than prior monocular baselines.

Our method also has limitations that point to promising future directions. First, our refinement reduces but does not eliminate geometric errors. Regions where neither visual nor physical signals are informative remain difficult, and improving overall geometric and physical parameter estimation remains an open direction. Second, our pipeline relies on known camera parameters and known scene geometry (e.g., the ground plane), limiting its applicability to fully in-the-wild videos. Removing this assumption is a natural next step toward truly assumption-free monocular inverse physics.

This work is supported by a National Institute of Health (NIH) project #R21EB035832 "Next-gen 3D Modeling of Endoscopy Videos".

6 Implementation Details↩︎

Optimization. We optimize the physical dynamics parameters over a total of 250 iterations (200 for Neo-Hookean). The number of frames per iteration starts at 4 and increases linearly to the maximum (16 frames for both Vid2Sim and our dataset), reaching this maximum 50 iterations before the end of optimization. The remaining 50 iterations use the full sequence for refinement. At each iteration, we randomly sample frames and additionally optimize appearance-related parameters (excluding position and physical parameters) for 100 inner steps. Since baselines use between 200 and 450 physics simulation iterations, our method does not substantially increase computational cost. After dynamics optimization, we further refine appearance for an additional 5,000 iterations.

Loss Hyperparameters. As shown in 9 , all loss terms are combined with unit weights, without any relative weighting. For the particle distribution regularizer \(\mathcal{L}_{\text{distr}}\), we use \(K=3\) nearest neighbors per particle, and set the lower and upper bounds to \(\tau_{\min}=0.3\) and \(\tau_{\max}=0.8\), respectively, which leaves sufficient room for adaptive geometry while preventing overly aggressive regularization.

Learning Rates. We use the Adam optimizer with fixed learning rates throughout the optimization, and apply the same hyperparameters across all datasets and scenes. The global scale \(s\) and the initial velocity \(v_0\) both use a learning rate of \(0.02\). For material parameters, we use \(0.1\) for Young’s modulus \(E\), \(0.01\) for Poisson’s ratio \(\nu\), and \(0.05\) for yield stress \(\sigma_y\). For Gaussian features, we use \(0.0002\) for both position and (visual) Gaussian scale, and \(0.01\) for both color and opacity.

Computational Resources and Cost. All experiments are conducted on a single NVIDIA RTX A6000 GPU. Optimization typically takes approximately 2 hours per scene, which dominates the overall cost since our method is optimized per scene.

7 Physics-Aware Geometry Refinement↩︎

Per-particle physical volume. The detailed algorithm is described in 6. In our implementation, we run the iteration for 5 steps. Given that 100 to 200 simulation steps are typically required between two consecutive frames in our datasets, this incurs less than \(1/10\) of the computational cost of a single forward simulation pass for one frame.

Particle management. Every 10 iterations, we identify the bottom \(1\%\) of Gaussians by either visual importance \(p^{\text{vis}}_i\) or physical importance \(p^{\text{phys}}_i\). And replace them by deterministically selecting the same number of Gaussians with the highest overall importance \(p_i\), splitting each in half, and relocating the resulting copies to the positions of the removed ones. For relocation, we use the algorithm proposed in 3DGS-MCMC [29].

8 Differentiable Position Map↩︎

We expand on the position-map reparameterization introduced in 3.4, showing why standard 3DGS rendering blocks pixel-coordinate gradients and why the proposed reparameterization restores them.

Standard 3DGS rendering carries no positional gradient from pixel-coordinate losses. In 1 , the rendered color at pixel \(\mathbf{p}\) is a function of the per-Gaussian quantities \(\{\mathbf{c}_i, o_i, \mathbf{x}_i^{2D}, \boldsymbol{\Sigma}_i^{2D}\}\), while \(\mathbf{p}\) itself is a fixed integer pixel index. For any loss \(\mathcal{L}(\mathbf{p})\) defined on pixel coordinates (rather than rendered colors), we therefore have \[\label{eq:supple95zerograd} \frac{\partial \mathcal{L}(\mathbf{p})}{\partial \mathbf{x}_i^{2D}} = 0.\tag{10}\] A pixel-coordinate loss has no gradient path to any Gaussian’s image-space position through standard rasterization.

The differential position map yields a unit positional gradient. For a pixel \(\mathbf{p}\) covered by Gaussian \(i\), the differentiable position from 4 is \[\tilde{\mathbf{p}}_i = \mathbf{x}_i^{2D} + (\boldsymbol{\Sigma}_i^{2D})^{1/2} z, \qquad z = \mathrm{sg}\!\left((\boldsymbol{\Sigma}_i^{2D})^{-1/2}(\mathbf{p} - \mathbf{x}_i^{2D})\right).\] At evaluation, \((\boldsymbol{\Sigma}_i^{2D})^{1/2} z = \mathbf{p} - \mathbf{x}_i^{2D}\), so \(\tilde{\mathbf{p}}_i = \mathbf{p}\) numerically (the rendered position map equals the pixel grid). Because \(z\) is treated as a constant under stop-gradient, however, \[\label{eq:supple95unitgrad} \frac{\partial \tilde{\mathbf{p}}_i}{\partial \mathbf{x}_i^{2D}} = \mathbf{I}.\tag{11}\] A pixel-coordinate loss applied to \(\tilde{\mathbf{p}}_i\) therefore produces a unit-magnitude positional gradient on the covering Gaussian’s 2D mean, which then propagates through the rasterizer and the differentiable MPM simulation back to the Gaussians’ 3D positions and physical state.

9 Dataset Details↩︎

Scene composition. Our synthetic dataset is built from 10 source meshes drawn from Google Scanned Objects (GSO) [34]: 5 objects simulated with an elastic (Neo-Hookean) constitutive model and 5 simulated with a plasticine constitutive model. For each scene, the material parameters (Young’s modulus \(E\), Poisson’s ratio \(\nu\), and yield stress \(\sigma_y\) for plasticine) and the initial velocity are independently sampled at random. All scenes share a fixed density of \(1000\) kg/m\(^3\), gravity of 9.81 m/s\(^2\), and a frictionless ground plane at \(z=0\). Each sequence is simulated with MPM at 20 fps.

Scenes. 4 lists the 10 source GSO objects and the constitutive model used for each.

Constitutive models. Both material models share the same Lamé parameter conversion from \(E\) and \(\nu\): \[\mu = \frac{E}{2(1+\nu)}, \qquad \lambda = \frac{E\,\nu}{(1+\nu)(1-2\nu)}.\]

Neo-Hookean (elastic). We use the compressible Neo-Hookean Kirchhoff stress: \[\boldsymbol{\tau} = \mu\,\mathbf{F}\mathbf{F}^{\!\top} + \bigl(\lambda \ln J - \mu\bigr)\mathbf{I}, \qquad J = \det \mathbf{F},\] where \(\mathbf{F}\) is the deformation gradient.

Plasticine. The plasticine model combines a Fixed Corotated elastic stress with a von Mises return mapping in principal log-strain space. With the polar decomposition \(\mathbf{F} = \mathbf{R}\mathbf{S}\), the elastic Kirchhoff stress is \[\boldsymbol{\tau} = 2\mu\,(\mathbf{F} - \mathbf{R})\mathbf{F}^{\!\top} + \lambda\,J\,(J - 1)\,\mathbf{I}.\] Plasticity is enforced by a return mapping on the trial elastic deformation gradient. Given the SVD \(\mathbf{F}^{\text{trial}} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\!\top}\), we form the principal log strains \(\boldsymbol{\varepsilon} = \ln \boldsymbol{\Sigma}\) and their deviatoric part \(\hat{\boldsymbol{\varepsilon}} = \boldsymbol{\varepsilon} - \tfrac{1}{3}\,\mathrm{tr}(\boldsymbol{\varepsilon})\,\mathbf{I}\). The von Mises yield indicator is \[y = \|\hat{\boldsymbol{\varepsilon}}\| - \frac{\sigma_y}{2\mu}.\] If \(y > 0\), we project \(\boldsymbol{\varepsilon} \leftarrow \boldsymbol{\varepsilon} - y\,\hat{\boldsymbol{\varepsilon}} / \|\hat{\boldsymbol{\varepsilon}}\|\) and update \(\mathbf{F} \leftarrow \mathbf{U}\exp(\boldsymbol{\varepsilon})\mathbf{V}^{\!\top}\); otherwise \(\mathbf{F}\) is left unchanged.

10 Details on Losses↩︎

Optical Flow Loss \(\mathcal{L}_{flow}\) The probability-based optical flow loss in the main paper uses a per-pixel uncertainty \(\sigma\) produced by the flow estimator [32]. The full loss with explicit uncertainty is: \[\label{eq:flow95full} \begin{align} \mathcal{L}_{\text{flow}} = -\frac{1}{|\Omega_t|}\sum_{j \in \Omega_t}\!\Big[ &\log p\!\left(\mathbf{f}_{0 \to t}(\mathbf{p}_j) \mid \hat{\mathbf{f}}_{0 \to t}(\mathbf{p}_j),\, \sigma_{0 \to t}(\mathbf{p}_j)\right) \\ + &\log p\!\left(\mathbf{f}_{t\text{-}1 \to t}(\mathbf{p}_j) \mid \hat{\mathbf{f}}_{t\text{-}1 \to t}(\mathbf{p}_j),\, \sigma_{t\text{-}1 \to t}(\mathbf{p}_j)\right)\Big], \end{align}\tag{12}\] where \(\sigma_{i \to j}(\cdot)\) is the per-pixel uncertainty from frame \(i\) to frame \(j\) at pixel \(\mathbf{p}\). The likelihood downweights high-uncertainty pixels, which improves robustness when physical dynamics change visibility between frames.

References↩︎