Literature
Display
← All papers

MoRE: Mixture-of-Experts for Multi-Physics Robotic World Models

Ziyu Zhu · Shaowei Liu · Albert J. Zhai · Henry Che · Wei-Chiu Ma · Shenlong Wang

Conference session · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

MoRE predicts manipulated-object motion from histories of object and tool point clouds. A shared representation feeds three motion experts: rigid non-prehensile, deformable non-prehensile, and deformable prehensile. A supervised router chooses an expert from observed interaction history, combining a rigid transformation constraint with free-form deformation models.

Training mixes simulated and real interactions. The model improves reported geometric prediction metrics and supports model predictive control (MPC) for box pushing and toy manipulation. The reported 20 frames per second refers to model inference, not a demonstrated 20 Hz perception-and-planning control loop.

figure 1
Figure 1: MoRE predicts action-conditioned 3D dynamics by routing tool-object interactions to specialized motion experts. This unified MoE model handles both rigid and deformable objects, as well as both prehensile and non-prehensile contact. Trained on mixed simulation and real data, MoRE supports real-time prediction at 20 FPS and generalizes across tasks, materials, and objects.

1. Introduction

Manipulation prediction must accommodate both material behavior and contact type. A single unconstrained predictor can distort rigid objects, while a rigid model cannot represent squeezing or stretching. MoRE addresses this conflict by sharing scene and motion features while separating output rules according to physical regime.

The contribution is a unified predictor over three selected regimes and multiple tools, with learned routing at inference. It does not cover every material or manipulation category: rigid grasping and simultaneous interactions among multiple manipulated objects are outside the modeled scope.

3.1. Problem formulation

Equation 1 defines future object point clouds from current state, current tool geometry, and a finite state/action history. Actions are represented by the tool’s full point cloud, obtained from robot commands through forward kinematics. This abstracts away particular joint coordinates while still requiring the tool geometry and kinematic mapping.

The implementation uses eight history steps, a three-step prediction horizon during training, 150–512 object points according to object size, and 100 end-effector points. Longer predictions feed generated states back into the history buffer.

Equation
equation 1

3.2. Mixture of motion experts

Static geometry and motion features are fused into a shared representation. A gating network predicts a one-hot selection over three experts; Equation 2 retains the selection constraint. The main method re-evaluates this router at each rollout step, allowing the selected regime to change.

Hard routing selects one expert’s prediction rather than averaging incompatible rigid and deformable outputs. Expert labels supervise routing during training; test-time selection uses observed motion history rather than a supplied interaction label.

figure 2
Figure 2: Overview of MoRE. Given the current and historical object states and actions, represented as point clouds, we first encode the scene and actor inputs into point tokens using two encoders. An explicit gating network then selects among specialized motion experts, each tailored to a distinct physical regime, to predict the next object state in 3D. P = prehensile, NP = non-prehensile.
Equation
equation 2

3.3. Physics-aware experts

The experts cover rigid pushing or pulling, deformable pushing or pulling, and deformable grasping. Rigid prehensile interactions are omitted under the authors’ assumption that negligible-slip rigid grasping has relatively little motion uncertainty. This assumption limits the scope of the three-expert decomposition.

Rigid expert

The rigid head predicts a global rotation and translation, using a continuous six-dimensional rotation representation. Applying the same rigid transform to every object point preserves internal distances and prevents arbitrary geometric distortion. Zero rigidity error is therefore partly built into the output representation, rather than independent evidence that all contact dynamics are correct.

Deformable experts

Two separate heads predict pointwise 3D displacements with the same architecture but different parameters. One specializes in sustained grasping interactions; the other models deformation under pushing or sliding. They can represent local shape change but do not impose a complete constitutive material law.

3.4. Training

Self-rollout training uses predicted states as subsequent inputs, with gradients detached through those rolled-in predictions [63]. Equation 3 combines point reconstruction error across the rollout with supervised expert-selection cross-entropy. Experts and the router are warmed up separately, then jointly fine-tuned on mixed data.

The main text describes simulation pretraining for each expert, while the appendix’s simulation generation describes non-prehensile interactions. It does not fully explain the corresponding prehensile simulation data or give a complete optimizer, training schedule, and split specification.

Equation
equation 3

Autoregressive inference

Starting from an observed object state and history, each prediction is appended to a rolling buffer and becomes the next input. The process repeats to the chosen horizon. This avoids test-time dynamics fitting, but predictions can still accumulate error when rolled out beyond observed states.

3.5. Planning

MPC samples candidate tool trajectories, predicts their object trajectories, and selects the plan with the lowest goal cost. The robot executes an initial action, observes the outcome, and replans. Chamfer distance to the target object point cloud measures task progress; an additional obstacle cost may be used.

The appendix adds a necessary initialization step: random probing interactions generate motion history for routing. Thus a static image or point cloud alone is insufficient for the reported system’s initial regime identification.

4.1. Implementation

Training uses about 20,000 simulation episodes and 7,000 real episodes, sampled with a fixed 1:1 simulation/real mixture. The main text cites DROID and AdaptiGraph for real data, while Appendix A identifies DROID and the Particle-Grid Neural Dynamics (PGND) dataset. The edition retains this source discrepancy rather than treating the provenance descriptions as identical.

Simulation covers roughly 300 rigid objects, 300 deformable objects, and 30 tools. Physical properties and action noise are randomized. The static geometry encoder is frozen; motion encoding, fusion, routing, and output heads learn the dynamics.

4.2. Tasks and metrics

Evaluation covers the three expert regimes. Geometric metrics compare predicted and reference point clouds; the tables label the correspondence metric as root mean squared error (RMSE), although parts of the prose and captions call it mean squared error (MSE). Chamfer distance (CD) evaluates unordered geometric agreement.

Rigid error is residual alignment error after fitting a rigid transform. Deformable evaluation uses a Laplacian error. These are geometric or structural proxies, not direct measurements of force accuracy or physical safety.

4.3. Dynamics results

Table 2 shows better geometric and structural metrics than AdaptiGraph and ParticleFormer across the listed regimes. Relative to privileged domain-specific experts, results are mixed: deformable non-prehensile RMSE is 0.024 versus 0.018, while CD improves from 0.032 to 0.027. Learned routing approaches specialized performance without winning every metric.

On DROID, Table 4 compares only moving target-object points, not whole-scene flow. MoRE’s RMSE is 0.023/0.014/0.021 across the three regimes, versus PointWorld’s 0.031/0.028/0.029. It also improves the listed CD and structural errors. Figure 3 preserves the cluttered-scene examples, including small and partially observed target objects.

table 2
Table 2: Dynamic model comparison across interaction types and physics. We report geometric accuracy (MSE, CD) and material structural metrics (rigidity or Laplacian). Privileged experts (gray) are domain-specific and assume known interaction types at test time, while our Mixture-of-Experts model handles diverse interactions in a unified framework with consistently strong performance.
table 4
Table 4: Comparison with Point-World on the DROID dataset. Our method performs better across all three cases, showing strong generalization to real-world object dynamics.
figure 3
Figure 3: Qualitative results on DROID evaluation. We compare model predictions with ground truth on challenging DROID scenes with clutter, diverse objects, and partial observations. Even for small, low-resolution manipulated objects, our model produces accurate dynamics, demonstrating robustness to occlusion and noise.

4.4. Expert and routing ablations

Capacity-matched ablations compare a single expert, soft implicit routing, an implicit mixture inside the Transformer, and training without self-rollout. The complete model has the best or tied-best values in Table 3.

Removing self-rollout increases deformable prehensile RMSE from 0.026 to 0.047. The comparisons support explicit regime-specific heads, supervised hard routing, and training on predicted states as useful components of this implementation. They do not establish that all soft mixtures or shared models must fail.

table 3
Table 3: Ablation of MoRE under different interaction types and physical regimes. We evaluate the impact of expert specialization, routing strategy, and self-rollout training using geometric and structural metrics. Performance degrades when removing expert specialization, explicit routing, or self-rollout training, highlighting the importance of each component for robust dynamics prediction.

4.5. Real-world control

Real-robot experiments use rigid box pushing and deformable toy-sloth pose matching, with goal point clouds and CD-based success thresholds. Baselines are a translation heuristic and a single Transformer without motion experts or routing.

Figure 4 retains the complete error curves and success-versus-threshold plots; Figure 5 retains the prediction sequences. MoRE generally lowers goal error, especially for the rigid box. On the toy task, the success curves cross at some thresholds, so the plot does not show strict superiority at every threshold. The paper does not report repeated-seed confidence intervals or a full control-loop timing breakdown.

figure 4
Figure 4: Model Predictive Control (MPC) performance. (a) Box pushing and (b) toy manipulation results. Each section details the prediction error across planning steps (left) and success rate vs. error threshold (right). MoRE achieves lower error and higher success than the baselines.
figure 5
Figure 5: Real world dynamics predictions. We visualize predictions of point motion on real world data. The red points are the end-effector (action) points. The Vanilla model struggles to treat the box as rigid and the toy sloth as soft. In contrast, our model correctly handles the different cases.

5. Discussion

MoRE combines a common geometric representation with specialized motion outputs. The main evidence is improved prediction on the three tested regimes and useful downstream control on two real-world task families. Its rigidity constraint and motion-history routing are central to that behavior.

Limitations

The model predicts one manipulated object at a time and does not model simultaneous multi-object dynamics. Mass, friction, elasticity, and hidden internal state are not explicitly represented; visually similar objects such as empty and filled bottles may therefore behave differently without being distinguishable from the available history.

The reported experiments leave uncertainty about broader material coverage, held-out object/tool splits, and full-system runtime. Generalization claims should be read within the described datasets and examples.

Acknowledgments

The paper acknowledges six National Science Foundation awards, computing resources from the National Center for Supercomputing Applications through Delta and Delta AI, and NVIDIA’s academic grant program.

Appendix

The full appendix’s dataset, architecture, planning, routing, and qualitative material is retained below. The project page is more-robo.github.io.

A.1. Dataset curation

Simulation and real trajectories share a point-cloud representation and are sampled equally during training. This balances controlled simulated variation with real sensing noise and contact behavior.

A.1.1. Simulation data

Isaac Lab generates interactions with approximately 300 rigid assets across 130 categories, 300 deformable assets across 100 categories, and 30 tools. Contact points and normals determine pushing and pulling directions.

Full mesh states provide temporally corresponding surface samples, reduced with farthest point sampling. Object point counts range from 150 to 512. Randomized mass, friction, deformable stiffness, and per-step Gaussian action noise diversify the training trajectories.

A.1.2. Real-world data

PGND supplies reconstructed object-motion sequences. DROID is processed through PointWorld’s scene-flow pipeline, then converted to object-centric flow: Qwen3 identifies the end-effector, manipulated object, and interaction type; SAM2 segments that object. If several masks remain, the nearest to the end-effector is selected.

This curation supplies target-object tracks and interaction labels automatically. Errors in recognition, segmentation, flow, or mask selection can propagate into the learned model.

A.2. Model architecture

Frozen Uni3D features and learned Transformer motion features each have 1,024 dimensions. Cross-attention combines the current object geometry with temporal object/tool tokens. A two-layer multilayer perceptron (MLP) produces routing logits.

The selected MLP head predicts either a global rigid transform or point displacements. This design separates geometric feature sharing from the final motion constraints.

A.3. MPC pipeline

After random interaction probes initialize history, non-prehensile planning samples 48 pushes from 16 contact points and three directions. For prehensile control, grasping is established first, then bounded tool displacements are sampled. Start/end waypoints are interpolated into dense candidate action sequences.

The planner evaluates both candidate trajectory and stopping time, allowing early termination when a predicted intermediate state better matches the goal. Real-world perception fuses four RealSense D455 cameras, applies SAM3 text-prompt masks, removes outliers with Open3D, recenters coordinates, and samples points as in training. These perception and probing steps are part of deployment even though dynamics inference needs no online optimization.

A.4. Gating evaluation

Across 50 random interactions per category, Table 5 reports selection accuracies of 95.4% for pushing, 94.9% for toy manipulation, and 99.8% for soft grasping, with 96.7% overall. The text does not fully specify how frame-level or trial-level decisions are aggregated into these percentages.

table 5
Table 5: Gating Network Accuracy. Expert selection accuracy (%) evaluated over 50 random interactions per task category. The gating network maintains consistently high accuracy across diverse regimes, indicating reliable expert routing for different physical dynamics.

B.1. Benchmark visualization

Figure 6 preserves representative simulation and real-world setups for rigid and soft objects. These examples illustrate the tested regimes and tools.

figure 6
Figure 6: Benchmark tasks visualization: Simulation and real-world benchmarks covering rigid and soft objects under non-prehensile and prehensile interactions, designed to evaluate generalization across diverse physical regimes.

B.2. Simulation visualization

Figure 7 retains both complete panels: dynamic prediction sequences and MPC rollouts across rigid and deformable simulation tasks, including all original subcaptions.

figure 7
Figure 7: Simulation results on diverse manipulation tasks.

B.3. Real-world visualization

Figure 8 retains the full real-world prediction and control sequences, including box pushing, both plate-pushing directions, and sloth movement. These are qualitative examples rather than additional independent success-rate measurements.

figure 8
Figure 8: Real-world results on diverse manipulation tasks.
Concept demonstration

Open image ↗