Literature
Display
← All papers

Learning Physics-Guided Residual Dynamics for Deformable Object Simulation

Shivansh Patel · Kaifeng Zhang · Sanjay Pokkali · Svetlana Lazebnik · Yunzhu Li

Preprint posted · CoRL 2026 (accepted) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the 18-page arXiv v1 paper, including all five appendices. Its evidence concerns six tracked real objects, short training rollouts extended at evaluation, and small controlled manipulation studies. All original figures, tables, equations and 98 references remain; the figure artwork is restored from the PDF.

Original Figure 1, native PDF artwork with all panels and labels.
Fig. 1: Physics-Guided Residual Dynamics Overview. Given RGBD observations, we extract surface points to instantiate the simulation. Red dots indicate the point where gripper held the object. The framework rolls out an optimized spring-mass physics backbone from the current object state Xt,VtX_{t},V_{t} and robot actions ata_{t} to produce simulation prediction Xt+1sim,Vt+1simX_{t+1}^{\text{sim}},V_{t+1}^{\text{sim}}. Subsequently, our residual dynamics network predicts per particle residual velocities, which are added to the simulator velocities and time integrated to obtain the final positions X^t+1,V^t+1\hat{X}_{t+1},\hat{V}_{t+1}.

At a glance

Physics-Guided Residual Dynamics (PGRD) first fits a spring–mass simulator, then learns corrections to its predicted velocities. A point-cloud encoder, position-conditioned decoder and gated temporal transformer improve tracked deformation and rendered-frame accuracy across rope, paper, flag, two soft toys and a duster. The model also drives predictive control and interactive Gaussian rendering; cable rerouting succeeds in 8/10 trials versus 2/10 for the fitted physics backbone.

I Introduction

Analytical simulators supply useful structure but imperfect materials, discretization and contact models leave systematic errors. Pure learned dynamics can fit those errors but require data and may drift. PGRD separates parameter fitting from residual learning so the backbone need not be differentiable. Velocity corrections are integrated into positions, and training recursively feeds back predicted states to expose the model to rollout errors. The evidence supports improved accuracy and empirical stability in the tested settings; this is not a conservation or stability theorem.

II Related Works

The paper relates spring–mass, finite-element, position-based and material-point simulators to learned graph/point-cloud dynamics. Differentiable simulators enable parameter identification but face difficult contact gradients; PGRD instead uses black-box fitting followed by learned corrections. Its closest comparison includes a custom differentiable spring–mass variant inspired by PhysTwin, and PGND as a learned point-based baseline. Hybrid modeling complements these approaches rather than eliminating their geometry and observation requirements.

III Method

The model has two training stages: optimize a small set of physical parameters against observed trajectories, then learn residual velocities on top of that fitted backbone. The following sections specify the simulator, corrected state update and temporally informed network.

III-A Physics Backbone

Particles are linked by springs and advanced with semi-implicit Euler integration. CMA-ES fits stiffness, damping, connection distance, maximum springs per node and ground friction. Grasped particles follow the gripper rigidly; non-prehensile interaction uses a spherical gripper collision model that projects penetrating particles to its surface. Volumetric toys receive internal particles from RaySt3R surface completion, a watertight marching-cubes mesh and interior sampling. This geometry prior helps prevent collapse but introduces reconstruction assumptions.

III-B Residual Dynamics Framework

The network adds a residual to the simulator velocity and the corresponding time-step-scaled correction to its position prediction. Corrections are zero on particles rigidly held by the gripper. Training minimizes mean squared particle-position error across autoregressive rollouts: each prediction becomes the next input. The model trains for five steps and is evaluated over 37, testing extension of rollout horizon. Directly correcting positions sometimes causes numerical divergence in the authors’ comparison; the velocity formulation improves this empirical behavior.

Equations 1
V^t+1=Vt+1sim+Δ​Vt+1residual.\hat{V}_{t+1}=V^{\text{sim}}_{t+1}+\Delta V^{\text{residual}}_{t+1}.
X^t+1=Xt+1sim+Δ​Vt+1residual⋅d​t.\hat{X}_{t+1}=X^{\text{sim}}_{t+1}+\Delta V^{\text{residual}}_{t+1}\cdot dt.
ℒres=1T​∑t=1T1N​‖X^t−Xt∗‖22,\mathcal{L}_{\text{res}}=\frac{1}{T}\sum_{t=1}^{T}\frac{1}{N}\left\|\hat{X}_{t}-X_{t}^{*}\right\|_{2}^{2}, (1)

III-C Network Architecture

Point Transformer V3 encodes particle positions and velocities from the current/history states and the physics prediction. A Fourier-position-conditioned decoder predicts an initial three-dimensional velocity residual. A sliding-window transformer uses recent predictions, sinusoidal temporal encodings and repeated latest features when the buffer is not full. Its final projections produce a bounded correction and sigmoid gate. The retained formula blends the initial and temporal residuals and scales the entire blend by 0.1; it does not simply add an unrestricted temporal update. The appendix specifies layer sizes but does not give a numeric window length.

Original Figure 2, native PDF artwork with all panels and labels.
Fig. 2: Network Architecture. The points encoder extracts spatiotemporal features, which the decoder projects to estimate initial residual velocity. Finally, the temporal aggregator refines these estimates using a recurrent history to output the final residuals Δ​Vt+1residual\Delta V^{\text{residual}}_{t+1}.
Equations
𝜹t=tanh⁡(Projδ​(𝐡t)),gt=σ⁡(Projg​(𝐡t)),\boldsymbol{\delta}_{t}=\tanh(\text{Proj}_{\delta}(\mathbf{h}_{t})),\quad g_{t}=\sigma(\text{Proj}_{g}(\mathbf{h}_{t})),
Δ​Vt+1residual=0.1⋅[(1−gt)⊙Δ​Vt+1initial+gt⊙𝜹t],\Delta V^{\text{residual}}_{t+1}=0.1\cdot[(1-g_{t})\odot\Delta V^{\text{initial}}_{t+1}+g_{t}\odot\boldsymbol{\delta}_{t}],

IV Experiments

Experiments compare predicted particle trajectories and object-only rendered video on six real objects. The paper also ablates the learned components and evaluates the simulator inside manipulation planning; the observation and split protocol matters when interpreting these results.

IV-A Data Collection

An xArm7 manipulates rope, paper, flag, sloth toy, duster and teddy toy, observed by four calibrated depth cameras. Most interactions are prehensile; the teddy is pushed, and the flag’s stick moves rigidly with the gripper. The reported collection protocol creates 100 training and 20 validation episodes of four seconds with a two-second stride: adjacent windows overlap. The unique recorded durations are 202 and 42 seconds, respectively. These windows should not be interpreted as 120 independent four-second trials; the paper does not fully specify their partitioning.

Original Figure 3, native PDF artwork with all panels and labels.
Fig. 3: Experimental Objects (see text).
 
Method Metric     Rope Paper Flag Sloth Toy Duster Teddy Toy
 
Spring-Mass [34] MDE ↓\downarrow     4.4±2.04.4_{\pm 2.0} 2.3±2.02.3_{\pm 2.0} 5.7±2.55.7_{\pm 2.5} 3.2±0.83.2_{\pm 0.8} 3.8±2.03.8_{\pm 2.0} 5.8±3.45.8_{\pm 3.4}
MPM [73]     7.3±2.57.3_{\pm 2.5} 15.5±4.215.5_{\pm 4.2} 23.1±7.023.1_{\pm 7.0} 7.4±1.87.4_{\pm 1.8} 6.0±2.66.0_{\pm 2.6} –
GBND [94]     5.5±1.75.5_{\pm 1.7} 3.0±1.43.0_{\pm 1.4} 30.9±6.230.9_{\pm 6.2} 7.7±2.67.7_{\pm 2.6} 5.1±2.25.1_{\pm 2.2} 1.5±0.41.5_{\pm 0.4}
PGND [93]     3.3±1.83.3_{\pm 1.8} 2.1±0.52.1_{\pm 0.5} 3.2±1.43.2_{\pm 1.4} 4.0±1.34.0_{\pm 1.3} 3.8±0.13.8_{\pm 0.1} 1.6±1.31.6_{\pm 1.3}
Diff. Spring-Mass [34]     3.5±1.73.5_{\pm 1.7} 1.9±1.61.9_{\pm 1.6} 4.6±2.24.6_{\pm 2.2} 2.9±0.72.9_{\pm 0.7} 3.5±1.63.5_{\pm 1.6} 4.6±2.74.6_{\pm 2.7}
Ours     2.6±1.31\mathbf{2.6_{\pm 1.31}} 1.7±0.7\mathbf{1.7_{\pm 0.7}} 2.8±1.2\mathbf{2.8_{\pm 1.2}} 2.7±0.4\mathbf{2.7_{\pm 0.4}} 2.9±1.8\mathbf{2.9_{\pm 1.8}} 1.3±0.3\mathbf{1.3_{\pm 0.3}}
 
Spring-Mass [34] CD ↓\downarrow     4.5±2.44.5_{\pm 2.4} 1.7±1.21.7_{\pm 1.2} 9.1±4.79.1_{\pm 4.7} 2.9±0.72.9_{\pm 0.7} 4.7±2.54.7_{\pm 2.5} 5.1±3.05.1_{\pm 3.0}
MPM [73]     6.1±2.46.1_{\pm 2.4} 14.8±6.514.8_{\pm 6.5} 23.6±10.423.6_{\pm 10.4} 7.1±1.77.1_{\pm 1.7} 5.2±1.75.2_{\pm 1.7} –
GBND [94]     6.6±2.46.6_{\pm 2.4} 5.0±1.75.0_{\pm 1.7} 49.1±13.049.1_{\pm 13.0} 6.5±1.86.5_{\pm 1.8} 6.3±2.86.3_{\pm 2.8} 2.7±0.32.7_{\pm 0.3}
PGND [93]     2.7±1.42.7_{\pm 1.4} 2.2±0.92.2_{\pm 0.9} 4.3±2.44.3_{\pm 2.4} 3.2±0.83.2_{\pm 0.8} 4.0±0.14.0_{\pm 0.1} 1.6±1.01.6_{\pm 1.0}
Diff. Spring-Mass [34]     3.4±1.83.4_{\pm 1.8} 1.5±1.01.5_{\pm 1.0} 6.8±3.66.8_{\pm 3.6} 2.8±0.72.8_{\pm 0.7} 4.3±2.14.3_{\pm 2.1} 3.8±2.33.8_{\pm 2.3}
Ours     2.3±1.3\mathbf{2.3_{\pm 1.3}} 1.3±0.6\mathbf{1.3_{\pm 0.6}} 4.3±2.2\mathbf{4.3_{\pm 2.2}} 2.6±0.3\mathbf{2.6_{\pm 0.3}} 3.8±2.5\mathbf{3.8_{\pm 2.5}} 1.3±0.2\mathbf{1.3_{\pm 0.2}}
 
Spring-Mass [34] EMD ↓\downarrow     2.4±1.32.4_{\pm 1.3} 1.8±1.21.8_{\pm 1.2} 4.9±2.64.9_{\pm 2.6} 1.5±0.41.5_{\pm 0.4} 1.9±1.21.9_{\pm 1.2} 2.9±1.82.9_{\pm 1.8}
MPM [73]     3.9±1.63.9_{\pm 1.6} 11.4±5.511.4_{\pm 5.5} 19.4±7.819.4_{\pm 7.8} 3.9±0.93.9_{\pm 0.9} 2.8±1.22.8_{\pm 1.2} –
GBND [94]     3.0±1.33.0_{\pm 1.3} 2.1±0.92.1_{\pm 0.9} 24.5±6.324.5_{\pm 6.3} 3.1±1.13.1_{\pm 1.1} 2.8±1.42.8_{\pm 1.4} 0.8±0.20.8_{\pm 0.2}
PGND [93]     1.4±0.61.4_{\pm 0.6} 1.5±0.51.5_{\pm 0.5} 2.3±1.32.3_{\pm 1.3} 1.6±0.41.6_{\pm 0.4} 1.6±0.41.6_{\pm 0.4} 0.8±0.60.8_{\pm 0.6}
Diff. Spring-Mass [34]     1.9±1.01.9_{\pm 1.0} 1.6±1.01.6_{\pm 1.0} 3.9±2.13.9_{\pm 2.1} 1.6±0.41.6_{\pm 0.4} 1.6±1.01.6_{\pm 1.0} 2.3±1.32.3_{\pm 1.3}
Ours     1.2±0.6\mathbf{1.2_{\pm 0.6}} 1.4±0.7\mathbf{1.4_{\pm 0.7}} 2.2±1.2\mathbf{2.2_{\pm 1.2}} 1.4±0.2\mathbf{1.4_{\pm 0.2}} 1.4±1.1\mathbf{1.4_{\pm 1.1}} 0.6±0.1\mathbf{0.6_{\pm 0.1}}
 
TABLE I: Tracking accuracy across diverse objects. Physics-Guided Residual Dynamics achieves the lowest tracking error across all objects and metrics, outperforming both physics-based methods and learning-based approaches.

IV-B Evaluation Tasks and Metrics

Particle evaluation uses correspondence-based mean distance error (MDE), bidirectional nearest-neighbor Chamfer Distance and Earth Mover’s Distance, reported in centimeters. Reference trajectories are reconstructed from image/depth tracking rather than directly instrumented particle positions. For video, Gaussians fitted to the initial observation follow predicted particle positions and covariances, while appearance remains fixed. Object-only overlap, boundary F-score and perceptual distance measure rendering; background pixels are ignored. Table II scales overlap and F-score by 10 and LPIPS by 100.

IV-C Baselines

Five baselines are evaluated: fitted global spring–mass, Material Point Method (MPM), graph-based neural dynamics, PGND and differentiable spring–mass. The last is a PhysTwin-inspired implementation using a registered canonical spring graph, global fitting and spatial gradient refinement, rather than an unchanged PhysTwin system. The tested MPM implementation cannot handle the non-prehensile teddy interaction, so those entries are absent. Failure of this implementation on sparse thin objects does not establish a limitation of every MPM formulation.

IV-D Results

PGRD’s MDE means are 2.6, 1.7, 2.8, 2.7, 2.9 and 1.3 cm for rope, paper, flag, sloth, duster and teddy, versus 4.4, 2.3, 5.7, 3.2, 3.8 and 5.8 for global spring–mass. It gives the lowest reported means across the metrics, with a tie on flag Chamfer Distance. Qualitative comparisons show improvements for heterogeneous duster motion and teddy compression. The video table also favors PGRD’s mean overlap, boundary and perceptual scores. Reported uncertainty and some small margins preclude treating every comparison as established statistical dominance.

Original Figure 4, native PDF artwork with all panels and labels.
Fig. 4: Qualitative comparison of PGRD on dynamics prediction. Columns shaded in green show the input state with gripper positions (red spheres) and the ground truth object configuration after taking the action. The remaining columns show predictions from each method. PGRD accurately captures deformations across all objects. For the duster, only PGRD maintains stem rigidity while allowing deformation for the feathers. The MPM baseline is not evaluated on the Teddy Toy because it does not model the non-prehensile manipulation used in this experiment. For the Teddy Toy, PGND uses sampled gripper surface points for simulation, which are visualized as red spheres, making this example denser than the others.
Original Figure 5, native PDF artwork with all panels and labels.
Fig. 5: Example pair used to compute the visual metrics. We evaluate only the object of interest, ignoring the background. The mask error visualization shows true-positive object pixels in black, false negatives in red, and false positives in cyan. The RGB error heat map shows the per-pixel appearance difference within the evaluated object region, with stronger colors indicating larger RGB error. Part of the upper arm is missing because the sloth was grasped by the gripper, which is now removed.
 
Method Metric     Rope Paper Flag Sloth Toy Duster Teddy Toy
 
Spring-Mass [34] 𝒥\mathcal{J}-Score / IoU (×10\times 10) ↑\uparrow     3.17±1.513.17_{\pm 1.51} 6.82±1.476.82_{\pm 1.47} 4.60±1.954.60_{\pm 1.95} 5.30±0.875.30_{\pm 0.87} 6.66±0.826.66_{\pm 0.82} 5.43±1.995.43_{\pm 1.99}
MPM [73]     2.44±1.452.44_{\pm 1.45} 4.51±0.944.51_{\pm 0.94} 3.78±1.123.78_{\pm 1.12} 5.22±0.885.22_{\pm 0.88} 5.92±0.875.92_{\pm 0.87} –
GBND [94]     0.53±0.550.53_{\pm 0.55} 5.57±1.405.57_{\pm 1.40} 5.08±1.425.08_{\pm 1.42} 3.70±1.493.70_{\pm 1.49} 5.13±1.045.13_{\pm 1.04} 7.52±0.667.52_{\pm 0.66}
PGND [93]     3.18±2.133.18_{\pm 2.13} 7.08±0.907.08_{\pm 0.90} 5.47±1.665.47_{\pm 1.66} 5.86±0.935.86_{\pm 0.93} 6.69±0.466.69_{\pm 0.46} 7.13±1.027.13_{\pm 1.02}
Diff. Spring-Mass [34]     3.45±1.433.45_{\pm 1.43} 7.10±1.377.10_{\pm 1.37} 4.98±1.824.98_{\pm 1.82} 5.58±0.825.58_{\pm 0.82} 6.70±0.786.70_{\pm 0.78} 5.82±1.855.82_{\pm 1.85}
Ours     3.95±1.69\mathbf{3.95_{\pm 1.69}} 7.53±0.82\mathbf{7.53_{\pm 0.82}} 5.54±1.89\mathbf{5.54_{\pm 1.89}} 6.21±0.78\mathbf{6.21_{\pm 0.78}} 6.77±0.83\mathbf{6.77_{\pm 0.83}} 7.82±1.14\mathbf{7.82_{\pm 1.14}}
 
Spring-Mass [34] ℱ\mathcal{F}-Score (×10\times 10) ↑\uparrow     6.59±1.606.59_{\pm 1.60} 4.71±2.614.71_{\pm 2.61} 2.08±1.812.08_{\pm 1.81} 5.76±1.125.76_{\pm 1.12} 5.32±0.975.32_{\pm 0.97} 4.73±2.724.73_{\pm 2.72}
MPM [73]     4.92±2.454.92_{\pm 2.45} 4.73±0.824.73_{\pm 0.82} 2.49±0.972.49_{\pm 0.97} 5.13±0.865.13_{\pm 0.86} 5.00±1.085.00_{\pm 1.08} –
GBND [94]     1.95±1.911.95_{\pm 1.91} 4.95±1.354.95_{\pm 1.35} 2.54±2.042.54_{\pm 2.04} 3.59±1.153.59_{\pm 1.15} 4.11±1.094.11_{\pm 1.09} 7.71±0.967.71_{\pm 0.96}
PGND [93]     5.87±2.965.87_{\pm 2.96} 5.39±1.655.39_{\pm 1.65} 2.65±1.982.65_{\pm 1.98} 5.53±1.115.53_{\pm 1.11} 5.29±0.975.29_{\pm 0.97} 7.27±1.547.27_{\pm 1.54}
Diff. Spring-Mass [34]     6.95±1.506.95_{\pm 1.50} 4.98±2.454.98_{\pm 2.45} 2.22±1.702.22_{\pm 1.70} 6.03±1.056.03_{\pm 1.05} 5.45±0.915.45_{\pm 0.91} 5.05±2.505.05_{\pm 2.50}
Ours     7.38±1.85\mathbf{7.38_{\pm 1.85}} 5.91±1.76\mathbf{5.91_{\pm 1.76}} 2.68±2.14\mathbf{2.68_{\pm 2.14}} 6.30±1.07\mathbf{6.30_{\pm 1.07}} 5.51±1.02\mathbf{5.51_{\pm 1.02}} 8.03±1.75\mathbf{8.03_{\pm 1.75}}
 
Spring-Mass [34] LPIPS (×100\times 100) ↓\downarrow     2.6±0.92.6_{\pm 0.9} 5.6±2.25.6_{\pm 2.2} 9.9±3.09.9_{\pm 3.0} 5.9±2.05.9_{\pm 2.0} 7.0±1.97.0_{\pm 1.9} 3.6±1.73.6_{\pm 1.7}
MPM [73]     3.8±1.43.8_{\pm 1.4} 6.5±1.46.5_{\pm 1.4} 9.1±2.59.1_{\pm 2.5} 7.7±1.77.7_{\pm 1.7} 6.8±1.76.8_{\pm 1.7} –
GBND [94]     5.2±1.75.2_{\pm 1.7} 5.7±1.35.7_{\pm 1.3} 8.8±1.38.8_{\pm 1.3} 10.5±2.010.5_{\pm 2.0} 6.4±1.36.4_{\pm 1.3} 2.2±0.52.2_{\pm 0.5}
PGND [93]     2.6±1.32.6_{\pm 1.3} 5.2±1.15.2_{\pm 1.1} 8.9±1.98.9_{\pm 1.9} 6.5±1.86.5_{\pm 1.8} 6.6±0.96.6_{\pm 0.9} 2.0±0.82.0_{\pm 0.8}
Diff. Spring-Mass [34]     2.35±0.852.35_{\pm 0.85} 5.05±2.005.05_{\pm 2.00} 8.95±2.708.95_{\pm 2.70} 5.45±1.855.45_{\pm 1.85} 6.35±1.756.35_{\pm 1.75} 3.25±1.553.25_{\pm 1.55}
Ours     2.3±1.1\mathbf{2.3_{\pm 1.1}} 4.9±1.4\mathbf{4.9_{\pm 1.4}} 8.5±2.9\mathbf{8.5_{\pm 2.9}} 5.4±2.1\mathbf{5.4_{\pm 2.1}} 5.9±1.9\mathbf{5.9_{\pm 1.9}} 1.7±0.9\mathbf{1.7_{\pm 0.9}}
 
TABLE II: Action-conditioned video prediction with dynamics prediction and 3DGS. PGRD produces the most accurate visual predictions, demonstrating superior rendering quality.
Original Figure 6, native PDF artwork with all panels and labels.
Fig. 6: Planning results on rope manipulation tasks. Top row shows start and end configurations for four tasks. The bottom row shows the Chamfer Distance to the target during MPPI planning. PGRD (green) converges faster and achieves lower final error than the optimized spring-mass baseline (red) across all tasks.

V Applications

The fitted dynamics are used as a forward model for receding-horizon manipulation and as a motion model for interactive rendering. These applications reuse the learned object representation and calibrated observations.

Original Figure 7, native PDF artwork with all panels and labels.
Fig. 7: Cable Rerouting Execution. Comparison of PGRD (top) and the Spring-Mass backbone (bottom) for rope rerouting through a slot. PGRD navigates the rope through the opening while the Spring-Mass backbone exhibits repeated collisions with the slot edges.

V-A Planning using PGRD

MPPI samples and refines action sequences using predicted Chamfer Distance to a target point cloud. Only the first action is executed before observing and replanning. Four rope objectives—lift, lower, rotate and curve—each receive ten trials with the same initial/goal states and equal planning budgets for PGRD and spring–mass. The curves demonstrate convergence in these matched settings rather than broad task success rates.

Cable rerouting uses ten varied start/end configurations and no rope–slot interaction examples in residual training. Success requires the entire rope to pass through the opening and the gripper to move below the final slot height: PGRD achieves 8/10 versus 2/10. This is a small, specific generalization test.

For language-conditioned goals, Nano Banana Pro edits an initial image; Depth Anything V2 estimates its depth, aligned affinely to the original depth using unchanged table points. Known camera intrinsics produce a target point cloud for MPPI. AnyGrasp supplies candidates: choose the rope’s right end, or use DINOv2 correspondences to identify the sloth region with largest requested motion. These single-camera demonstrations still depend on initial depth, calibration and generated-goal quality; no separate aggregate language-task success rate is reported.

Original Figure 8, native PDF artwork with all panels and labels.
Fig. 8: Cable rerouting through narrow slot. Left: distribution of start and end configurations. Right: Chamfer Distance during MPPI planning steps. PGRD achieves lower distance than the spring-mass baseline, successfully threading the rope through the slot in 8 out of 10 trials compared to 2 out of 10 for the baseline.
Original Figure 9, native PDF artwork with all panels and labels.
Fig. 9: Execution of PGRD with generated goals. The first column shows the initial image, and the second column shows the language-conditioned generated goal image. Subsequent columns show robot execution from a higher-quality demo camera. The target point cloud from the goal image is overlaid on the last image. PGRD successfully executes a range of tasks, including passing a rope through a specified slot and moving a sloth toy to target configurations.
Original Figure 10, native PDF artwork with all panels and labels.
Fig. 10: Interactive rope manipulation with PGRD rendered using 3DGS. The visualization overlays three timesteps, with later configurations in lighter opacity. As PGRD predicts the dynamics, the 3D Gaussians attached to the particles are updated to render images that maintain visual consistency.

V-B Interactive Photo-Realistic Simulation.

Keyboard manipulation commands drive the physics model and learned residuals, then particle motion updates Gaussian rendering. Fixed color/opacity retain the initial object appearance. The demonstration shows interactive action-conditioned deformation, without establishing generalization to new materials, lighting or arbitrary viewpoints, or reporting an inference-rate benchmark.

VI Conclusions

The central result is that a fitted physics backbone plus learned, temporally refined velocity corrections improves the tested real-object simulations and downstream planning. Limits include dependence on reconstructed geometry and tracked observations, object-specific training, short/small controlled evaluations and imperfect generated goals. Empirical rollout stability is useful evidence, but the formulation supplies no general conservation or stability guarantee.

A Ablation Study

Table III tests architectural alternatives on rope and sloth, reporting MDE. Full PGRD obtains 2.6/2.7 cm. These component ablations support the chosen combination in two objects rather than isolating universal design laws.

Ablation variant Rope Sloth Toy
PGRD 2.6 2.7
Replace PTv3 with PointNet++ 3.0 3.2
Remove decoder 2.8 3.0
Remove temporal aggregator 2.8 3.1
Replace gating with addition 2.7 2.9
Remove internal particles N/A 3.8
TABLE III: Ablations of the main components of our model. We report MDE. Replacing velocity with position residuals leads to divergence. Rope has no internal particles, so it is N/A.

A-A Encoder choice

Replacing PTv3 with PointNet++ raises rope/sloth MDE from 2.6/2.7 to 3.0/3.2 cm, supporting the stronger encoder for these residual-prediction tasks.

A-B Decoder

Directly projecting encoder features without the position-conditioned decoder raises MDE to 2.8/3.0 cm. Explicit spatial decoding remains helpful even after point-cloud encoding.

A-C Temporal aggregator

Removing temporal aggregation raises MDE to 2.8/3.1 cm. Recent motion history helps predict deformations that are ambiguous from a single configuration.

A-D Gated temporal refinement

Replacing the gate with direct addition gives 2.7/2.9 cm. The experiment supports controlled blending of temporal information; it does not prove that gating guarantees stable dynamics.

A-E Internal particles for volumetric objects

Removing interior particles raises sloth MDE to 3.8 cm; rope has no corresponding volumetric ablation. The qualitative rationale is that a surface-only spring network can collapse. The teddy also uses interior particles, but this table reports only sloth.

B Network Architecture Details

The encoder uses 18 input channels and 64-dimensional features. The conditional decoder has two hidden layers of width 64 and three velocity outputs. The temporal transformer has 64-dimensional embeddings, four heads, two layers and a 128-dimensional feedforward block; the final residual scale is 0.1. Table IV retains these settings exactly.

Parameter Value
Encoder (PTv3)
Feature dimension 64
Input channels 18
Decoder
Hidden layers 2
Hidden dimension 64
Output channels 3
Temporal Aggregator
Embedding dimension (dd) 64
Attention heads (HH) 4
Transformer layers (LL) 2
Feedforward dimension (dffd_{\text{ff}}) 128
Gating scale 0.1
TABLE IV: Network architecture hyperparameters.
Original Figure 11, native PDF artwork with all panels and labels.
Fig. 11: Planning results on sloth toy manipulation tasks. Top row displays the initial and target configurations for four manipulation objectives. Bottom row plots the Chamfer Distance evolution throughout MPPI planning. PGRD (green) consistently achieves superior convergence compared to the tuned spring-mass baseline (red).

C Planning on Sloth Toy

Sloth planning repeats lift/lower/rotate/curve objectives, with ten trials each using identical start/target states and matched computational budgets. PGRD reduces target Chamfer Distance more consistently, while the spring–mass model may plateau on complex limb deformation. Figure 11 extends the convergence evidence to a volumetric toy, with the same limited-state trial design.

D Data Collection and Processing

Four RealSense D455 cameras record synchronized color/depth at 30 Hz. Grounded SAM 2 segments each object, CoTracker tracks image points, and depth plus camera calibration lift displacements into a shared three-dimensional frame. Overlapping views’ velocities are averaged. Persistent particles are propagated by their velocities, then updated from the mean velocity of their five nearest observed neighbors. The reported approximately 2 Hz for 1,000 particles describes this offline processing pipeline, not the dynamics model’s inference speed.

E MPPI Planning Parameters

Simple tasks use eight sampled trajectories, a ten-step horizon and three MPPI optimization iterations; rerouting uses 15 trajectories, a 15-step horizon and six iterations. Both use covariance 0.1. These budgets define the matched comparisons rather than a universal controller configuration.

MPPI Hyperparameter Simple tasks Cable rerouting
Number of sampled trajectories 8 15
Planning horizon length 10 15
Optimization iterations per step 3 6
Action noise covariance 0.1 0.1
TABLE V: MPPI hyperparameters used for planning. Separate settings are used for the simpler tasks and for cable rerouting.
Concept demonstration

Open image ↗