Literature
Display
← All papers

Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation

Bohan Wu · Roberto Martín-Martín · Li Fei-Fei

· ICRA 2026 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

RAPiD learns deformable-object manipulation in simulation, first using privileged particle histories and then replacing them with learned embeddings inferred from depth images, joint angles and actions. The frozen policy adapts its conditioning during execution without updating network weights. On a 22-DoF TIAGo mobile manipulator, it succeeds in 17/20 insertion trials and 16/20 covering trials on real objects, for 82.5% overall.

I Introduction

Different materials call for different motions: placing a stiff hat over a bowl differs from flinging or sweeping soft cloth. RAPiD extends Rapid Motor Adaptation by using recent simulated particle positions to teach a representation of changing shape, in addition to object mass and position. A second stage learns to infer the useful representation from onboard sensing.

The result is a learned policy with a changing latent context, rather than an explicit online fit of physical stiffness or a particle simulator running on the robot. This edition follows all eight pages of v1, dated 18 March 2026. The PDF’s entire introduction and Figures 1–3 were absent from the HTML; they are restored here, along with corrected figure/section numbering, all six complete figures, the full results table and 52 references.

Complete original PDF Figure 1
Fig. 1: RAPiD: Rapid Adaptation of Particle Dynamics (Left) is a method for learning to perform real-world deformable object mobile manipulation tasks by inferring and adapting to the unknown dynamics of deformable objects in real-time. Top Right: In simulation, RAPiD first learns a visuomotor policy to achieve a deformable object mobile manipulation task, such as bowl-covering, by inferring the deformable object’s dynamics from the robot’s visual observations and actions. Bottom Right: RAPiD then deploys the learned visuomotor policy directly to the real world to achieve the deformable object mobile manipulation task, under unseen object dynamics, instances, categories, and lighting conditions, using only onboard sensor signals.

II Related Work

Model-based manipulation

Model-based methods use particles, points, graphs or differentiable simulation to estimate deformable state and plan actions. Learning particle dynamics and differentiable particles illustrate this direction. RAPiD avoids requiring an explicit particle reconstruction at deployment, which can be useful under occlusion. It still performs learned latent inference; the claim should not be read as eliminating all state estimation or proving that every model-based method requires complete visibility.

Rapid motor adaptation

RMA trains a privileged dynamics representation and then predicts it from recent observations/actions. Extensions include in-hand object rotation. RAPiD adds a shape-history representation so the policy can respond to deformation as well as mass/position. This supplies a useful training signal without requiring the deployment module to identify named material parameters.

Learning from real-world data

Generalist policies such as π₀ and OpenVLA learn from real robot data and can require embodiment-specific adaptation. RAPiD instead trains its task policy in simulation without real robot demonstrations. The paper does not benchmark those generalist models on TIAGo; its argument that adequate demonstrations would be infeasible is a motivation, not an experimental comparison or a lasting limit on that model family.

Learning from simulation

Prior sim-to-real approaches use randomization, real-data calibration and dense correspondences. RAPiD’s distinction is a history-conditioned adaptation module used during execution, rather than collecting several real trajectories to retune simulation before deployment. The claimed advantage is demonstrated on the selected tasks and baselines, not every possible history-aware or calibration-based method.

III RAPiD: R apid A daptation of P art i cle D ynamics

Training has two stages. First, jointly learn a visuomotor policy and privileged shape/dynamics encoders in simulation. Second, train observation-based modules to imitate their latent outputs and fine-tune the policy on those predicted embeddings. Deployment discards the privileged encoders and freezes all learned weights.

The source formulates control as a finite-horizon Markov decision process with simulator state, onboard observation, robot action, transition, reward, discount and initial-state distribution. Although both phases involve RL for the policy, the adaptation modules in Phase II are trained by supervised L₁ embedding losses. Figure 2 preserves the complete computation and gradient-flow diagram.

Complete original PDF Figure 2
Fig. 2: The RAPiD Method. Below we explain training (top and middle) in simulation and deploying in the real world (bottom) a deformable object mobile manipulation solution with RAPiD. Training with RAPiD (top and middle) is a two-step procedure. In the first step (top), the agent is asked to solve a simulated version of the task by learning a visuomotor policy and two encoders: a Shape Encoder, μs, and a Dynamics Encoder, μd, using privileged state information of the object from the simulator, such as mass and particle positions. In the second step (middle), RAPiD frees itself from privileged simulator information by learning a new Shape Adaptation module, φs, and Dynamics Adaptation module φd, using L₁-losses to replace the Shape Encoder and the Dynamics Encoder, respectively. At test time (bottom), both the Shape and Dynamics adaptation modules and the visuomotor policy are directly deployed to the real world, with the adaptation modules updating the Shape Embedding ẑts and Dynamics Embeddings ẑtd to the visuomotor policy once every 5 timesteps.

Phase I: privileged encoders and policy

The shape encoder consumes recent ground-truth particle positions/position changes and corresponding robot actions. Its embedding feeds a dynamics encoder together with privileged object mass and position. The policy uses the resulting dynamics embedding and the current visual observation to produce full-DoF actions. The experiments also specify joint-angle observations.

The policy and both encoders are trained end-to-end by RL. This makes the embedding task-relevant but does not force its coordinates to equal physical material parameters. Keeping two encoders allows separate supervision of shape and dynamics during the next phase.

Phase II: adaptation modules

The shape adaptation module replaces particle inputs with recent depth images, joint angles and actions, regressing its output toward the privileged shape embedding with L₁ loss. The dynamics adaptation module similarly predicts the privileged dynamics embedding from observation/action history and the inferred shape representation.

Dynamics-module gradients are stopped before entering the shape module, and RL gradients are stopped before entering either adaptation module. The policy is then fine-tuned on the predicted latents. These restrictions help preserve separate supervisory targets; they do not prove that shape and dynamics factors are statistically disentangled.

Test-time deployment

The robot maintains a buffer of ten recent observation–action pairs. Shape and dynamics embeddings are updated every five timesteps to encourage short-term behavioral consistency, while the policy uses its current observation and latest embedding. Network weights stay fixed: “online adaptation” is recurrent inference of context, not online gradient learning.

Figure 6 shows selected latent coordinates across objects and an object whose effective bending behavior changes with rotation. The illustrated coordinate changes with the interaction, supporting a qualitative connection to softness. There is no calibrated stiffness target, correlation coefficient, controlled material-identification error or proof that this coordinate represents softness independently of shape, pose or action history. The camera runs at 3 Hz, but complete action/update latency is not reported.

Complete original PDF Figure 6
Fig. 6: Visualization of RAPiD’s dynamics embedding in 1D_Inserting (Top-Left) and 2D_Covering (Top-Right) tasks. The number below each object photo is the value of a single dimension of the dynamics embedding. Bottom: the rollout of the policy given a special object whose rigidity changes with rotation. The line charts below the rollout visualize the value of a single dimension of the dynamics embedding during robot execution. We see that certain dimensions of the dynamics embedding are strongly correlated with the softness of the objects, demonstrating the adaptation module’s ability to infer the softness of the objects.

IV Experiments

Robot and simulation

TIAGo has 22 controlled degrees of freedom: three for the mobile base, fourteen across two arms, one torso, two head and two grippers. A head-mounted RGB-D camera supplies 224×224 depth images at 3 Hz; the policy also receives joint angles. The authors report autonomous execution without human intervention or QR markers.

Training uses OmniGibson/BEHAVIOR-1K with a matching robot model, success reward of 1 and a small distance-based shaping reward. Real tables, objects and lighting are held out from simulation. Each task is evaluated 20 times with different objects and initial locations. Training-set size, network dimensions, optimizer/RL details, compute, action frequency and repeated-seed variability are not specified sufficiently for a complete reproduction.

Complete original PDF Figure 3
Fig. 3: Simulated and Real-World Version of the Two Tasks: 1D Inserting (Left Half) and 2D Covering (Right Half). Each task is trained in simulation (left image of each half) and then deployed directly to the real world (right image of each half) under unseen objects, environments, and lighting conditions. By interacting with deformable objects across a broad range of instances, geometries, and categories in simulation, RAPiD successfully learns a visuomotor policy that adapts to and excels at manipulating real-world deformable objects of unseen dynamics, instances, and categories on a real bimanual mobile manipulator.

Tasks and success definitions

Insertion requires picking an elongated object and placing either end into a container within 300 seconds. Covering requires picking a planar object and covering at least 90% of a container opening within the same limit. Real test objects span 20 categories per task; containers span 20 categories in the insertion description. These are 20 trials per task, not 20 trials for every category combination.

The complete Figure 4 inventory includes soft ropes/cloth as well as comparatively rigid items such as a marker, hammer, lid and cardboard. Thus the benchmark spans stiffness rather than consisting exclusively of highly deformable objects. The source says instance variation supplies diversity without explicit pose/stiffness/friction parameter randomization; it does not quantify coverage of those physical ranges. Some figure panels label the insertion task “1D Threading,” while the text and success rule call it insertion; the original labels are retained.

Complete original PDF Figure 4
Fig. 4: All unseen real-world deformable objects used for the 1D_Inserting (left) and 2D_Covering (right) mobile manipulation tasks, respectively. 1D categories (left): belt, hammer, marker, banana, HDMI adapter, GPU connector, jumping rope, ethernet cable, zip tie, rake, mouse, screwdriver, stick, rubber, velvet rope, VGA adapter, anchor rope, garden rope, ribbon, fan. 2D categories (right): polyester bag, pants, towel, plastic bag, face mask, paper bag, glove, shorts, cap, lid, sock, soft cuff, cardboard, envelope, pouch, wallet, eye-mask, t-shirt, sponge, wool hat. Learning from deformable objects across a wide range of instances in simulation enables RAPiD to zero-shot transfer to real-world objects under unseen dynamics, instances, and categories.

Baselines and ablations

DMfD uses simulated expert demonstrations; DDOD’s rope descriptors and fabric correspondences underpin the other comparison. The DDOD controller is scripted instead of using its original human-demonstration procedure. Both baselines use RAPiD for grasping/navigation before their manipulation stage, so the reported comparison is against adapted hybrid systems, not untouched original baselines.

No-Adapt removes both adaptation modules. No-Shape retains dynamics adaptation but removes shape adaptation. E2E keeps the broader architecture but skips privileged pretraining and L₁ imitation, training by end-to-end RL. These ablations test useful components/training choices together; they do not isolate a universal requirement for exactly this architecture.

Comparison with baselines

Table I reports insertion/covering success of 85/80% for RAPiD, 15/5% for DMfD and 10/25% for DDOD. Totals are 33/40 (82.5%), 4/40 (10%) and 7/40 (17.5%), respectively. Against the stronger baseline on each task, the absolute gains are 70 points for insertion and 55 for covering; the 65-point gain is the aggregate comparison with DDOD. The introduction’s “65%+” claim should not be interpreted as at least 65 percentage points on each task.

Qualitative failures involve segmentation or correspondence errors under gripper occlusion, deformation and oblique views, plus insufficient dynamic adaptation even when perception works. Figure 5 contrasts hanging a rope with inserting an adapter, and sweeping a towel with placing a hat. These examples are consistent with adaptive behavior but do not isolate perception, planning and dynamics inference as separate causes.

The table supplies exact counts but no uncertainty intervals, formal significance tests, per-category replication or training-seed variability. Large observed differences support this experimental setup; they do not establish generalization across all deformable manipulation or all unseen embodiments.

Complete original PDF Figure 5
Fig. 5: RAPiD’s Dynamics Adaptive Behavior in 1D_Inserting (Top) and 2D_Covering (Bottom) tasks. Top Left: RAPiD hangs the rope over the top of the container before lowering it. Top Right: RAPiD flips the gripper upside down before performing a direct adapter insertion. Bottom Left: RAPiD moves the robot arm sideways and sweeps a towel horizontally from right to center to cover the bowl. Bottom Right: RAPiD moves the arm vertically and places a hat directly on top of the bowl. This shows RAPiD’s ability to infer different object dynamics and adapt actions accordingly to achieve the deformable object mobile manipulation tasks.
TABLE I: Successful Trials Out of 20 and Success Rates for RAPiD, Baselines and Ablations
Complete original PDF Table I
Original selectable table
RAPiD DMfD DDOD RAPiD-No-Adapt aRAPiD-No-Shape RAPiD-E2E
1D_Inserting 17 (85%) 3 (15%) 2 (10%) 7 (35%) 7 (35%) 5 (25%)
2D_Covering 16 (80%) 1 (5%) 5 (25%) 5 (25%) 9 (45%) 4 (20%)
Total 33 / 40 (82.5%) 4 / 40 (10%) 7 / 40 (17.5%) 12 / 40 (30%) 16 / 40 (40%) 9 / 40 (22.5%)

Removing adaptation

No-Adapt scores 7/20 insertion and 5/20 covering, totaling 30%. The full method’s 82.5% is 52.5 percentage points higher. The authors describe rigid-like handling of soft objects in the ablation. This supports the value of history-based adaptation within the tested system, while leaving architecture/capacity and training-budget controls unspecified.

Removing shape inference

No-Shape scores 7/20 insertion and 9/20 covering, totaling 40%, a 42.5-point deficit from RAPiD. It can grasp and approach the target but often fails at the final dynamic manipulation. The result supports adding shape-history supervision; it does not show that the inferred representation reconstructs the complete physical shape or material model.

End-to-end training

E2E scores 5/20 insertion and 4/20 covering, totaling 22.5%, a 60-point deficit. The authors attribute this to difficult RL optimization over high-dimensional histories and a failure to learn useful dynamics latents. The table establishes the performance gap, but no detailed convergence curves or gradient diagnostics verify that causal explanation. Privileged pretraining plus supervised adaptation is effective here; end-to-end RL is not ruled out with other architectures, budgets or optimization methods.

V Conclusion

RAPiD demonstrates simulation-trained latent adaptation on two real mobile-manipulation tasks without real robot demonstrations. Its strengths are the complete embodied experiment, explicit privileged-to-observed training path and substantial ablation gaps. Remaining limits include only 40 evaluation trials, altered baselines, unquantified training/physical coverage, sparse implementation detail and qualitative rather than calibrated evidence for material inference. Broader task, embodiment and real-data comparisons remain open.

Concept demonstration

Open image ↗