Literature
Display
← All papers

SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model

Qiayuan Liao · Yiyang Shao · Junfeng Long · Kevin Zakka · Roberto Horowitz · Pieter Abbeel · Koushil Sreenath

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

At a glance

SuperMimic jointly trains a humanoid motion-tracking policy and one differentiable dynamics model shared across a motion library. Simulator transitions fit the world model; short imagined rollouts provide analytic policy gradients through that learned model. The simulator itself need not be differentiable.

Three hours on one H100 surpasses the released SONIC checkpoint’s normalized return and reduces failed motions from 179 to 25 on a shared 129,514-motion benchmark. A two-H100 run reaches zero failures after 9.8 hours. These are deterministic evaluations of training-motion coverage. The headline GPU-hour ratio compares different hardware and training datasets; a separate LAFAN1 experiment supplies a more closely matched PPO comparison.

1. Introduction

Large motion libraries share one robot’s body and contact dynamics. SuperMimic seeks to learn that shared structure once and reuse it for policy improvement across many reference motions. Figure 1 illustrates the learned-model rollout and reported training budgets.

The central tradeoff is fast, differentiable prediction in place of fresh simulator rollouts for every policy update. Its gradient is exact for the learned surrogate objective, not necessarily for the original simulator or physical robot.

Complete original figure 1
Figure 1: A learned differentiable world model trains a humanoid whole-body controller and reduces GPU-hours by more than 1000× versus cluster-scale model-free reinforcement learning (§4.1). Left: the policy improves through imagined rollouts of the world model. Right: 128 L40 GPUs train for 168 h in SONIC [1], against one H100 for 3 h in this work.

Contributions

The paper combines an online learned world model, short-horizon actor–critic updates, a motion-library scaling study, physical Unitree G1 demonstrations, and a preliminary velocity-command interface. The evidence concerns the studied flat-ground motion-tracking setting.

3. Method

Training alternates simulator data collection, world-model fitting, and actor–critic optimization on imagined trajectories. Appendix D retains the complete algorithm.

3.1 SuperMimic training loop

The current deterministic policy collects simulator transitions into a fixed-size replay buffer. Optional action noise broadens model-training coverage, and the buffer stores the executed noisy action. Short replay windows fit the world model; replay states then initialize imagined policy rollouts. Model weights are frozen only during each policy update.

The policy receives a proprioceptive observation and goal. Soft survival weights use c0=1 and ct+1=ct(1−dt+1), with objective J(θ)=E[∑t≥0γtctrt+1]. Simulator terminations are binary; model terminations are probabilities, keeping imagined survival differentiable. Observations factor as ot=ōt+ηt, with known sensor noise added outside the model.

Complete original figure 2
Figure 2: SuperMimic training loop. (1) The current policy πθ collects transitions in simulation into a fixed-size replay buffer B, where new transitions replace the oldest. (2) The differentiable world model is fit to k-step windows sampled from B (§3.2). (3) The policy is improved by backpropagation-through-time through the frozen world model over H imagined steps; a critic Vψ, trained in the same phase, bootstraps the return beyond H (§3.3). Dashed arrows are gradients; Snowflake icon = frozen, flame icon = trained. Symbols: goal gt (e.g., reference motion), action at, action noise εt, observation ot, state s and predicted state ˆs, and sim context ξ (domain-randomization parameters). LWM is the world-model loss, ˆ ̄ot the predicted noise-free observation, ηt the sensor noise, ˆrt+1 the predicted reward, and ˆJ(θ) the policy objective.

3.2 Learning a differentiable world model

The deterministic model conditions on simulation context: friction, joint calibration offsets, and torso center-of-mass offsets. Its dynamics and reward, observation, and termination heads are retained below; individual heads may be learned or analytic.

Equation 1 fits eight-step autoregressive windows using state, reward, and noise-free observation errors plus termination cross-entropy. Analytically computed heads contribute no learned-head loss. Context information is available to the model during training; the deployed actor uses noisy proprioception and a goal.

Equations
Complete original equation heads
Complete original equation 1

3.3 Learning the policy through the world model

Equation 2 sums discounted rewards across 32 imagined steps and bootstraps with the critic at the horizon. Rollouts start from replay states with their recorded context, and sensor noise is reintroduced before policy inference. Backpropagation through the fixed world model supplies the actor gradient.

The model is trained on eight-step windows but used over longer imagined horizons. This gives useful gradients in the reported experiments; stable long rollouts are an empirical claim, not a bound on accumulated error.

Equation
Complete original equation 2

Critic objective

Equation 3 preserves the critic loss and backward return recursion. The terminal target is the critic’s horizon estimate; targets stay fixed during the critic fit. Updates use imagined transitions and a GAE-style λ return [26].

The critic extends the policy objective beyond the imagination window. Similar nominal tracking scores without it can conceal worse recovery from disturbances.

Equation
Complete original equation 3

4. Experimental results

Training uses MuJoCo Warp and H100 80 GB SXM5 GPUs. Normalized return is undiscounted episode reward divided by its maximum; early termination lowers it. The authors report stable training across several seeds but provide no seed-variation bands.

Checkpoint evaluation, training-episode return, imagined-step efficiency, and simulator throughput are different measurements. The figures retain their separate definitions and budgets.

4.1 Tracking and compute comparison

Figure 3 compares all checkpoints on the same 129,514 motions. After three hours on one H100, SuperMimic exceeds the released SONIC checkpoint’s 0.90 normalized return and has 25 failures versus 179. After 9.8 hours on two H100s, it has none under this protocol.

The three-hour threshold is chosen conservatively: later checkpoints maintain at least half the gap between SONIC and SuperMimic’s best result on both return and failure count. Appendix F defines the training-set benchmark and failure rule.

Complete original figure 3
Figure 3: SuperMimic surpasses SONIC’s tracking return with far fewer GPU-hours. (a) SuperMimic on one and two H100 GPUs surpasses the return of the released SONIC checkpoint (dashed), all three scored on the same benchmark of 129,514 motions (264 h). Two budget rows compare SuperMimic (dot) with SONIC (diamond): SuperMimic reduces wall-clock time by more than 50× and GPU-hours by more than 1000×. (b, c) On a shared dataset-length sweep, GPU-hours to target return (b) grow slowly while the world model’s 8-step prediction error (c) stays within 0.007–0.011 as the dataset grows from 1 to 264 h, consistent with one world model amortizing shared body and contact dynamics across motions. The prediction error counts the state only, is normalized by a predefined nominal error, and is measured online on policy rollouts.

Compute accounting

SONIC’s reported 128 L40 GPUs for 168 hours total 21,504 GPU-hours, compared with three H100 GPU-hours. The approximate ratios are 7,200× in GPU-hours and 56× in elapsed time, summarized as over 1,000× and 50×. SuperMimic’s time includes data collection, model fitting, and policy updates; post-training checkpoint evaluation is excluded.

The training data also differs: 264.56 hours of public BONES-SEED for SuperMimic versus SONIC’s 611 hours. GPU-hours do not normalize hardware capability or utilization. The paper discusses peak throughput differences, but this remains a system-level comparison rather than an isolated algorithm-only speedup.

Controlled PPO-Sim comparison

On LAFAN1’s forty clips, PPO-Sim shares the motions, observations, reward, termination, randomization, network sizes, and evaluation protocol. It uses BeyondMimic PPO hyperparameters and twice as many simulator environments per GPU.

SuperMimic reaches evaluation return 0.82 within 1.2 hours on one GPU. At the eighteen-hour budget, PPO-Sim reaches 0.60, 0.70, and 0.72 with one, four, and eight GPUs. Figure 4 also retains a separate horizon ablation based on training return, whose reset distribution changes during learning and differs from checkpoint evaluation.

Complete original figure 4
Figure 4: In controlled comparisons, SuperMimic reaches a higher return sooner than PPOSim, and the critic enables shorter imagination horizons. All runs use LAFAN1 (40 clips, 2.45 h). (a) SuperMimic backpropagates its imagined return through its world model on one GPU; PPO-Sim runs PPO in the simulator on 1, 4, or 8 GPUs (§4.1). Each point on a curve is a saved checkpoint scored on all 40 clips. (b, c) Final training return (b) and wall-clock hours until the training return reaches 0.82 (c) for each imagination horizon H, with and without the critic. The dashed line in (a) and (b) marks this target return of 0.82. Training return is the return of training episodes. Their initial states follow return-based reset sampling (Appendix D.1) and shift during training, so training return differs from the checkpoint scores in (a). In (c), a cross marks a horizon at which return 0.82 is not reached: both variants at H=8, no critic at H=16. The shaded band marks the default H=32.

4.2 Properties of the learned world model

Three analyses examine model accuracy across data sizes, simulation versus imagination throughput, and analytic updates versus PPO on imagined data.

Shared dynamics across the motion library

Across one, seven, forty, and 264 hours of motion, eight-step normalized state prediction error stays between 0.007 and 0.011, while compute to the target return grows slowly. The error is measured online on policy rollouts, not on an independently fixed held-out state distribution. The result supports reuse of one body’s dynamics within this sweep.

Imagination rollout speed

On one H100, model rollouts exceed MuJoCo Warp throughput by more than 10× over practical batches of 2,000–16,000 environments and by 62× at 64,000. A step is one 50 Hz control interval; the simulator computes four 5 ms physics substeps while the learned model runs at training TF32 precision. This is rollout throughput, not the whole training loop’s speedup.

Complete original figure 5
Figure 5: Imagined rollouts are fast and differentiable. (a) On an H100, over the batch sizes used in practice (shaded, 2k–16k parallel environments), the world model runs more than 10× faster than MuJoCo Warp; the gap widens to 62× at 64k, where MuJoCo Warp has flattened. One step is one 50 Hz control step, which MuJoCo Warp computes as four 5 ms substeps; the world model runs at the TF32 precision used in SuperMimic’s training. (b) On walk/run/sprint motions from LAFAN1 [29], SuperMimic, trained with its actor–critic update, reaches the best training return of PPO trained on world-model rollouts (PPO-WM) with about 64× fewer imagined steps. PPO-WM runs at four batch sizes (parallel rollouts × steps per rollout); 16k×32 is the best. Each curve is a centered moving average over ±0.05 decade of imagined steps.

Analytic gradients

On LAFAN1 walking, running, and sprinting motions, analytic actor–critic training reaches the best PPO-on-world-model training return with about 64× fewer imagined steps. Figure 5 keeps all four PPO batch configurations. This mechanism ablation isolates the update route more closely than the external SONIC comparison.

4.3 Critic, horizon, and recovery

The critic makes a sixteen-step horizon comparable to a critic-free thirty-two-step horizon. The default thirty-two-step version reaches training return 0.82 fastest; increasing the horizon from sixty-four to 128 nearly triples time and reduces final return.

Under simulated T-pose velocity pushes of 0.5–4 m/s, the critic policy settles in 0.5–2.4 seconds. The critic-free policy continues stumbling even at 0.5 m/s and falls at 4 m/s. These are simulator perturbation results, separate from the physical demonstrations.

4.4 Real-robot transfer

Figure 6 retains complete frame sequences for a skipping jump, side roll, 90° flip, and in-place 360° flip on a physical Unitree G1. One general tracker is used without per-motion tuning. The paper gives qualitative demonstrations rather than repeated-trial success rates or a physical comparison with SONIC.

Complete original figure 6
Figure 6: One general motion tracker, trained through the learned world model, transfers to a real Unitree G1. (a) Skipping jump, (b) side roll, (c) 90◦flip, and (d) in-place 360◦flip.

4.5 Downstream task interfaces

A velocity encoder trained on its own task reward is compared with one trained through the motion-tracking objective using commands derived from demonstrations. Both share a learned world model and motion decoder; this preliminary experiment is detailed in Appendix G.

Figure 7 retains the command grids, error-size encoding, fall markers, and reference-data density. Task-specific reward training generally improves step-response accuracy, especially outside well-covered reference velocities.

Complete original figure 7
Figure 7: The policy with the RL-trained velocity encoder tracks step commands more accurately than the distillation baseline. Each panel is a plane of the command space with the third component at zero, and each ring is one command. The outer disc’s area is proportional to the larger of the two policies’ relative steady-state errors in the step response to that command, and the hole’s area to the smaller one; the color marks the better policy. Details in Appendix G.

5. Conclusion

The experiments show substantial training-cost reductions for flat-ground humanoid motion tracking and useful physical transfer. They support the proposed roles of shared learned dynamics, fast rollouts, and analytic gradients. The broad claim that compute ceases to be a bottleneck is limited by the tested robot, tasks, data range, and hardware.

6. Limitations and future work

The world model learns only from simulator transitions, and training excludes terrain and object interaction. Whether real data improves the model or supports system identification remains open.

The downstream velocity reward updates only its encoder, with decoder weights frozen for that gradient. The benefit of concurrent interface training is untested; joint optimization of the shared decoder on downstream rewards is future work.

Acknowledgments

The authors acknowledge collaborators, support from Komatsu, the BAIR Center for Humanoid Intelligence, Apple, and the RAI Institute, compute from Lambda, and Bones Studio motion data. Abbeel and Sreenath also hold Amazon Scholar appointments; the paper identifies the work as UC Berkeley research unaffiliated with Amazon.

A. Observation, action, reward, and context

Appendix A specifies the deployed policy inputs, goals and actions, tracking reward, termination, and model-training context.

Observation

Table A.1 retains all 93 per-frame observation dimensions and their sensor-noise ranges: joint positions and velocities, base angular velocity, projected gravity, and previous action. The actor receives ten observation frames.

Complete original table a1
Table A.1: Proprioceptive observation ot. The noise column gives the per-dimension sensor noise ηt ∼ρη.

Goal

The motion encoder receives ten future reference frames, extending BeyondMimic’s single-frame target. Other goal interfaces replace the reference with their command representation.

Action

The normalized action maps to joint setpoint qsp=αat+q0. Low proportional-derivative gains make this a torque-related control proxy rather than strict position tracking.

Reward

Table A.2 preserves every tracking and regularization formula and weight. SuperMimic removes BeyondMimic’s self-collision penalty and lowers the joint-limit penalty from −10 to −2. The authors argue the self-contact exploit addressed by that penalty has no benefit for their deterministic imagined policy.

Complete original table a2
Table A.2: Motion-tracking interface reward, following BeyondMimic [3] with the self-collision penalty removed and the joint-limit weight lowered from −10 to −2 (§3.3). T is the set of tracked bodies.

Termination

Tracking terminates if root height error exceeds 0.25 m or the absolute difference in reference/robot gravity-direction z-components exceeds 0.8. For an upright reference, the latter corresponds to about 78° tilt. End-effector deviation does not terminate an episode.

Simulation context and randomization

Table A.3 retains friction, joint-offset, and torso center-of-mass sampling ranges. Unlike BeyondMimic, training omits root-velocity pushes because those abrupt state changes are not represented by this deterministic model. Physical-parameter randomization is reported sufficient for the demonstrated hardware transfer.

Complete original table a3
Table A.3: Domain-randomization parameters that make up the sim context ξ, following BeyondMimic [3] with the root-velocity perturbations removed. (U[·]: uniform distribution.)

B. Joint training across goal interfaces

Equation A.1 defines per-interface encoders and a shared decoder. Each interface has its own goals, reward, termination, and critic, while all share the body-dynamics model. The lifted objective is maxθEi∼p(I)[Ĵi(θ)].

Downstream gradients pass through the decoder to the selected encoder, but stop at the decoder’s weights. Motion-tracking updates still train the decoder. The velocity encoder additionally uses an observation-history feature and latent alignment on reference-command rollouts.

Equation
Complete original equation a1

C. Network architectures

Table A.4 retains every input, output, and hidden width. The dynamics MLP has three 2,048-unit layers; the shared policy latent has 32 dimensions bounded by tanh. Motion and velocity encoders have different input structures and capacities.

Complete original table a4
Table A.4: World-model, policy, and critic networks. MLP: multilayer perceptron, listed by hidden widths. The latent z is 32-dimensional and bounded by tanh.

D. Training loop and hyperparameters

Appendix D retains the full nine-step training algorithm and hyperparameters for the one- and two-GPU runs.

D.1 Algorithm

Algorithm A.1 preserves data collection, reset sampling, autoregressive model fitting, interface selection, imagined rollout, actor update, and critic fitting. The model updates each iteration and is held fixed only for policy backpropagation.

Complete original algorithm a1

Return-based reset sampling

Resets use one-second motion bins with running discounted returns. The mixture is 50% uniform, 40% softmaxb(−25Gb/Gmax), and 10% proportional to past softmax weights. This emphasizes difficult motion regions while retaining practice after improvement.

D.2 Hyperparameters

Table A.5 retains all optimizer, batch, clipping, noise, buffer, schedule, and parallel-training settings. Defaults include eight-step model windows, thirty-two-step imagination, discount 0.99, GAE λ=0.95, 4,096 environments per GPU, and 1,000 buffered steps per environment. The two-GPU learning rate is √2 times the one-GPU rate.

Complete original table a5
Table A.5: SuperMimic hyperparameters.

E. Motion data filtering

The public G1-retargeted BONES-SEED release has 142,220 clips. Filtering removes terrain/object interactions and infeasible retargets, leaving 129,514 clips. Table A.7 retains counts after each pass; the benchmark therefore excludes some behaviors unavailable to a free-standing flat-ground robot.

Complete original table a7
Table A.7: Number of BONES-SEED clips before filtering and after each filter.

Name filter

Action-name keywords remove 12,553 clips, including stairs, ladders, boxes, chairs, sitting, climbing, and specified raised-support actions. The list adapts SONIC’s public processing code and adds missed terms.

Feasibility filter

Forward kinematics removes another 153 clips. Criteria include a lowest body-link height above 0.20 m for at least 0.6 s; below −0.12 m at any time or below −0.08 m for over 80% of a clip; and both feet more than 0.02 m above the floor for at least 0.5 s while upright, at standing height, and nearly stationary.

The floating-feet test uses geometry-to-plane distance and excludes ballistic jumps through its stationary gate. These thresholds are tuned for the G1 retarget at 50 Hz.

F. Evaluation

Appendix F defines the common simulation, failure rule, full motion-group counts, and dataset-size sweep.

F.1 Benchmark metrics

Each of the 129,514 training motions is run once from its first frame with friction 1.0, deterministic physics, no domain randomization, and no observation noise. Policies retain their own actuator gains, action scales, and observation construction.

Failure uses SuperMimic’s root-height/tilt termination. SONIC’s original criterion instead checks root and end-effector height without an orientation term. Thus zero failures describes this shared deterministic training-set protocol, not unseen-motion generalization or a reliability guarantee.

F.2 Failed motions

Table A.8 retains every motion group. SONIC’s released checkpoint fails all 44 flips, all 22 side rolls, and all eight dance-routine clips, totaling 179 failures. Its separate v1.1 checkpoint has 239 failures. SuperMimic has 25 after three hours on one GPU and none after 9.8 hours on two.

Figure A.1 retains failure counts across saved checkpoints, evaluated every 4,000 iterations. The hardest five listed groups contain fourteen of the twenty-five early SuperMimic failures.

Complete original table a8
Table A.8: Failed motions per motion group on the filtered BONES-SEED training-set benchmark. SONIC: released checkpoint. Bold: the policy fails every motion in the group.
Complete original figure a1
Figure A.1: Failed motions during training on the filtered BONES-SEED training-set benchmark. Checkpoints are scored every 4000 iterations, for the runs of Figure 3(a). SONIC: released checkpoint.

F.3 Experiment protocols

The dataset-length sweep uses one, seven, forty, and 264 hours from the filtered BONES-SEED set. The shared benchmark in Figure 3 uses the complete filtered training-motion set.

G. Velocity-command task

Appendix G gives the complete separate experiment for velocity-command encoders.

G.1 Training setup

Each encoder is trained from scratch in its own 100,000-iteration run, together with the motion interface, using walking/running clips from LAFAN1 and BONES-SEED. Half of simulator environments and each imagined batch use the velocity interface. This mixed-batch implementation differs from Algorithm A.1’s single sampled interface per update.

Commands

Commands specify heading-frame forward/lateral velocity and world-frame yaw rate. Reference commands use a centered twenty-five-frame velocity mean. The RL run resamples command source every reset and every five seconds: half reference-derived, half uniform over vx∈[−1.5,2] m/s, vy∈[−1.32,1.32] m/s, and ωz∈[−3.35,3.35] rad/s. Uniform commands become standing with probability 0.1.

Encoder

On reference-command rollouts, a mean-squared latent-alignment loss of weight 0.1 pulls the velocity latent toward a stop-gradient motion latent.

Rollouts, reward, and termination

Each interface starts imagination from its own logged states. The RL encoder uses the velocity reward and fall-only termination in Table A.6; distillation uses the tracking reward and termination. All terms in both reward tables are multiplied by the 0.02-second control period.

Complete original table a6
Table A.6: Velocity-tracking reward and termination used to train the RL encoder. vxy is the planar root velocity in the heading frame, ωz the world yaw rate, and (vcmd xy , ωcmd z ) the command. φtorso is the sideways lean of the torso z-axis (forward lean is not penalized) and ωroll, ωpitch the torso roll and pitch rates, all in the heading frame. hroot is the root height and uz the z-component of the unit gravity direction in the root frame, so 1 + uz is 0 upright and 2 upside down, and the tilt condition triggers at about 73◦. Every term, here and in Table A.2, is multiplied by the control period ∆t = 0.02 s.

G.2 Step test

From standing, each command is held for five seconds in four trials. Relative steady-state error is the norm of mean velocity error over the last two seconds divided by command norm. The reported per-command value is the median among non-falling trials.

Three fifteen-by-fifteen command planes, excluding standing, supply 672 responses. Figure 7 caps errors at 0.9 and places a policy at the cap when all four trials fall.

Results

The RL encoder has lower error on 650 responses and distillation on fifteen; distillation falls in all four trials on seven fast-turn commands. Median relative errors are 0.16 and 0.49. These runs have one training seed each, and error statistics condition on surviving trials.

Concept demonstration

Open image ↗