Literature
Display
← All papers

Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL

Martin Schuck · Maks Sorokin · Simone Manni · Duy Ta · Angela P. Schoellig · Marco Hutter · Simon Le Cleac'h · Jan Brüdigam

Preprint posted · CoRL 2026 (accepted) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

The paper combines sampling-based model predictive control, offline-to-online reinforcement learning, and hierarchical whole-body control for state-based locomotion and manipulation.

Refer to caption
Figure 1: The three main stages of our proposed pipeline. We first collect a dataset from an expert SMPC, which is easy to tune in near real-time. We then train complex loco-manipulation policies with sparse rewards solely by mixing in the offline expert data, skipping tedious reward tuning. Finally, the sparse-reward policies are deployed on hardware. Website: https://pages.rai-inst.com/smpc2rl/

At a glance

A sampling-based model predictive controller generates simulated expert transitions that bootstrap sparse-task-reward reinforcement learning. A high-level learned policy then controls a frozen whole-body stabilizer and transfers to an arm-equipped Spot and a G1 humanoid. Learned policies complete the tested tasks faster than the planner. The method reduces repeated RL reward-shaping cycles, but does not eliminate manual engineering: the planner uses interactively tuned dense costs, the controller and action constraints are designed, and hardware evaluation relies on motion capture.

1 Introduction

Complex locomotion–manipulation tasks make uninformed sparse-reward exploration difficult. The paper moves the initial behavior-design work into a simulator-based planner whose costs can be adjusted while inspecting trajectories, then trains a fast policy from its data and subsequent simulated interaction. The division of labor is useful because teacher behavior need only seed successful exploration, while later reward optimization can improve it. Optimality remains local to the available data and training process.

2 Related Work

The method combines offline-data-assisted online RL [1] with the parallelized FastTD3 [21] implementation. Unlike human teleoperation or motion retargeting, the planner supplies demonstrations for different morphologies through simulation. The contribution is their integration for whole-body tasks with sparse task rewards, rather than a new general solution to planning or reinforcement learning.

3 Learning Loco-Manipulation With SMPC Demonstrations

The pipeline shares a hierarchical action interface between data generation and deployment. A simulated planner first supplies successful transitions; a high-level RL policy learns from a mixture of these transitions and its own experience, then gradually relies on online data alone.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 2: Real-world deployment of our tasks. Our framework learns complex loco-manipulation skills across different embodiments (Spot and G1), objects (boxes and tires), and skills (pushing, rolling, or lifting). From left to right, top to bottom: Spot reach, box pushing, tire uprighting, tire rolling, and G1 box pushing.

3.1 Overview

The high-level action changes commanded planar velocity, arm joint targets, and—on G1—torso height and pitch. A pretrained, frozen ReLIC whole-body controller [27] tracks these commands while coordinating the legs for balance. Spot holds torso height and pitch fixed. This decomposition constrains what the task policy can change and delegates dynamic stabilization to a separately trained controller.

3.2 Training Off-Policy RL with Sparse Rewards

The task reward is zero at the goal, minus one per unfinished step, and a terminal crash penalty of −2/(1−γ), with γ=0.99. Thus “sparse” means no dense progress-to-goal shaping; constant time cost and crash penalties still guide the objective. Initially, expert transitions make up half the replay mixture. Once empirical success reaches the chosen threshold, the expert component is removed so learning can improve beyond the teacher distribution.

Action deltas impose speed and acceleration bounds. Critic outputs are bounded using the known return range to reduce overestimation and instability. These constraints and action regularization remain part of the learning system; the claim of tuning-free training should be limited to avoiding repeated dense RL task-reward design.

Equation 1
r={0at goal−21−γrobot crashed−1else,r=\begin{cases}0\quad&\text{at goal}\\ -\frac{2}{1-\gamma}\quad&\text{robot crashed}\\ -1&\text{else}\end{cases}, (1)

3.3 Data Collection with Sampling-Based Model Predictive Control

The expert uses the same vectorized simulator with a dense, manually tuned cost. Random trajectory sampling and warm starts are sufficient for the tested tasks. Interactive execution is about half real time on an RTX 5090, allowing rapid qualitative cost adjustment. Tiled sampling scales data collection to roughly one million transitions per hour in the reported setting. The main text’s four-hour summary does not cover every embodiment: the appendix reports more than ten hours for G1 data collection.

Refer to caption
Figure 3: We scale our SMPC data collection by solving multiple SMPC in parallel on the GPU using vectorized environments divided into tiles. Each tile broadcasts its initial task to all its environments upon reset, runs sampled actions, selects the best trajectories, and adds them to our expert buffer. Note the different x-axes scales for a task’s trajectory steps and total simulation steps.

4 Experiments

Experiments examine real deployment, completion time, expert dataset size and quality, multimodal demonstrations, and alternatives that directly imitate or reuse the planner’s dense objective. Additional appendix studies test replay mixture, critic bounds, and the expert-data removal threshold.

Original figure
Original figure
Original figure
Original figure
Original figure
Figure 4: Training performance of our framework on different tasks. We show the success rate of our policy over time in increasingly complex tasks. Our framework reliably reaches success rates close to 100% in simulation without any reward shaping. Without adding expert SMPC data, learning complex loco-manipulation policies from sparse rewards fails to learn anything. Statistics are averaged over five seeds, with the shaded area representing the standard deviation.

Real-world deployment on diverse platforms

Five simulated tasks comprise Spot reaching, box pushing, tire uprighting, tire rolling, and G1 box pushing. Five-seed training curves approach high success with expert data, while sparse training without it fails on the complex tasks. Hardware Spot results are 10/10 reaching, 9/10 pushing, 10/10 uprighting, and 8/10 rolling. G1 deployment is demonstrated, but Table 1 does not provide the same ten-trial quantitative result for G1. These are state-based transfers with the specified controller and sensing setup.

Table 1: Real-world success rates for Spot tasks across ten tasks.
Reach Box Pushing Tire Uprighting Tire Rolling
10 / 10 9 / 10 10 / 10 8 / 10

Q1: Improving over seen expert data

Trained policies finish the simulated tasks faster than the SMPC demonstrations, with some mean-time improvements exceeding 50% and reported duration-standard-deviation reductions of 11–45%. Planner statistics use the four-million-transition dataset, while learned-policy statistics use 50,000 simulated episodes without exploration noise. This supports improvement over the sampled teacher behavior, not globally optimal task execution.

Original figure
Figure 5: We compare the performance of policies trained on sparse rewards (pink) and the SMPC used to generate the offline dataset on all tasks (blue). Our trained policies consistently finish the tasks faster than the SMPC experts. SMPC statistics are computed over the complete dataset of 4M samples, policy statistics are averaged over 50k simulated episodes without exploration noise.

Q2: Amount of data

Simpler tasks tolerate substantially smaller expert datasets; coordinated tire manipulation benefits from millions of transitions and broader coverage. The authors attribute this to critic estimates becoming more reliable near useful states. Dataset coverage is a plausible explanation, but the experiment does not isolate it from all other effects of dataset size.

Original figure
Original figure
Original figure
Original figure
Figure 6: Training performance with different SMPC dataset sizes. For simple tasks (i.e., navigation), reducing the number of samples has little effect. As tasks get more complex, the minimum required dataset size increases. On our most difficult task, agents require four million samples collected within 4 GPU hours to converge.
Original figure
Original figure
Original figure
Original figure
Figure 7: Performance under varying data quality. We reduce the number of sampling environments per tile during data generation and train policies on the resulting datasets. Most tasks are robust to lower-quality data, whereas tasks that require high coordination are sensitive.

Q3: Quality of the data

Reducing the number of sampled planner candidates lowers demonstration quality. Most tested tasks remain learnable, while tire rolling is more sensitive. The result suggests that acceptable teacher quality depends on coordination and precision demands, rather than requiring uniformly optimal demonstrations.

Original figure
Figure 8: Using multi-modal SMPC data heavily affects training and agents fail to learn viable policies.

Q4: Multimodality in the data

A tire dataset containing disparate strategies—leg kicks, body pushes, or stepping into the tire—fails to initialize the tested learner, even with slightly higher demonstration success. Planner contact penalties encourage a more consistent arm-based strategy and restore learning. This shows a limitation of the chosen dataset and learner combination; it is not evidence that Markov policies or offline RL inherently require unimodal data.

Q5: Training directly on the SMPC cost and data

Online FastTD3 with the planner’s dense costs, offline IQL, and Diffusion Policy all solve reaching but achieve at most 10% success on the harder Spot tasks under the tested setups. The proposed method reaches roughly 98–100% simulated success. These comparisons show that direct reuse of planner costs or imitation of its trajectories is insufficient here. The authors attribute imitation failures partly to missing recovery samples; that explanation is not independently tested with a full corrective-data study.

Table 2: Comparison against baselines. FastTD3 and IQL use the dense cost function of the SMPC expert. IQL and DP train on the SMPC data. We evaluate five runs, each on 100 simulated episodes, with an increased maximum duration to allow slower policies (IQL and DP) to finish.
Task FastTD3 IQL DP Ours
Reach 1.00 1.00 1.00 0.98 ±\pm 0.03
Box pushing 0.00 0.03 0.10 0.99 ±\pm 0.005
Tire uprighting 0.00 0.01 0.01 1.00 ±\pm 0
Tire rolling 0.02 0.04 0.01 0.98 ±\pm 0.02

5 Conclusion

Planner demonstrations can make sparse-task-reward learning practical for coordinated whole-body skills and allow a compact policy to improve on its teacher. The main advantage is moving behavior specification into faster planner tuning and then using online RL for refinement. It still depends on a simulator, a successful teacher distribution, engineered control interfaces, and suitable learning hyperparameters.

6 Limitations

Behavioral improvement remains local to the demonstration distribution and is not expected to discover wholly different globally optimal strategies. Network size and learning rates still affect results. The frozen low-level controller cannot adapt to task-specific disturbances despite its generic disturbance training. Deployment uses state estimates; unstructured or outdoor operation would require visual learning or policy distillation into a perception-based system.

Acknowledgments

The authors thank Annika Wollschläger and Dario Bellicoso for help exporting policies and deploying them on G1.

Appendix A Extended Ablations

The extended ablations average five training seeds and show standard deviations. They study how expert-data scheduling and bounded value estimates influence convergence.

Original figure
Original figure
Original figure
Original figure
Figure 9: We ablate the influence of the fraction of expert data in the buffer on training. Most tasks are robust to the choice of the expert data ratio. The exception is tire rolling, which greatly benefits from more expert data initially and collapses below 50% expert data. Note that all runs additionally use the phase-out threshold of 0.1, so the expert data ratio only directly influences the initial training phase.

Percentage of Expert Data

Reaching, pushing, and uprighting are relatively insensitive to the initial expert fraction under the phase-out schedule. Tire rolling fails with too little expert data and benefits from higher initial fractions. Because expert data are removed after 10% success, any later benefit must arise through the learned initialization rather than continued access to those transitions.

Original figure
Figure 10: Bounded critics stabilize the training (here Spot box pushing). In parameter ranges at the edges of what is still stable, bounded critics are still well-behaved (pink), whereas unbounded critics start to exhibit sharp drops in performance (blue).

Bounded Critic Networks

At a deliberately increased critic learning rate of 8×10⁻⁴, both bounded and unbounded critics can converge on box pushing, but unbounded runs exhibit sharp performance drops. Bounding widens the stable hyperparameter range in this test; it does not remove the need for tuning or guarantee stability.

Original figure
Original figure
Original figure
Original figure
Figure 11: Ablations on the phase-out threshold. Keeping the offline SMPC data in the buffer for too long harms training performance.

Phase-Out Threshold

Keeping planner data too long slows improvement once the learner begins succeeding. Thresholds at or above 50% prevent some tire-task runs from converging within the step budget; indefinite mixing is also harmful. The default removes expert data at 10% success. The proposed explanation is distribution mismatch between the increasingly specialized policy and the original teacher trajectories.

Appendix B RL Training Details

Training uses TD3 in mjlab/MuJoCo Warp with randomized object mass, size, and friction. The actor receives noisy observations while the critic receives exact simulated state. Rewards are scaled by 0.01, and action regularization and G1-specific learning-rate adjustments are retained. High-level learning takes 20–90 minutes on an RTX 5090, excluding expert-data generation and low-level-controller preparation. The original hyperparameter table specifies replay capacity, batches, update ratios, and other settings.

Table 3: Hyperparameters for the sparse TD3 training.
Parameter Value
Buffer size BB 16,777,216=22416{,}777{,}216=2^{24}
Gradient steps per update N∇N_{\nabla} 88
Phase-out threshold η\eta 0.10.1
Batch size NbN_{b} 81928192
Discount factor γ\gamma 0.990.99
Polyak factor τ\tau 0.005
Actor learning rate τ\tau 3×10−43\text{\times}{10}^{-4}
Critic learning rate τ\tau 1×10−41\text{\times}{10}^{-4}
Policy delay 2
Smoothing noise σs\sigma_{\mathrm{s}} 0.05
Action regularization 5×10−45\text{\times}{10}^{-4}
Reward scale 0.01

B.1 Bounding the Critic

Before reward scaling, the largest return is zero and the lowest is the immediate crash penalty −2/(1−γ). A final tanh, affine scaling, and initialization bias constrain critic predictions to that interval and start them near zero. This uses the deliberately simple reward’s known bounds rather than estimating limits from data.

Equation
minT>=0(∑i=0T−1−γi)−γT⋅2(1−γ)\displaystyle\min_{T>=0}\left(\sum_{i=0}^{T-1}-\gamma^{i}\right)-\gamma^{T}\cdot\frac{2}{(1-\gamma)}
=(∑i=0−1−γi)−γ0⋅2(1−γ)\displaystyle=\left(\sum_{i=0}^{-1}-\gamma^{i}\right)-\gamma^{0}\cdot\frac{2}{(1-\gamma)}
=−2(1−γ)\displaystyle=-\frac{2}{(1-\gamma)}

Appendix C SMPC Data Collection

The planner broadcasts each initial state across a tile of simulation candidates, samples smooth spline action sequences, evaluates them, and selects the best. Warm starts shift the preceding best sequence; only the first ten actions are executed before replanning. The specified 50 Hz action rate yields 5 Hz replanning, with a 50-step horizon, ten spline control points, and 256 candidates per tile. State caching avoids redundant simulation. Algorithm 1 and Table 4 preserve the full collection procedure and parameters.

Algorithm 1 SMPC Expert Data Collection
1: 𝒟←∅\mathcal{D}\leftarrow\emptyset, A¯←0\bar{A}\leftarrow 0 ⊳\triangleright Initialize expert dataset 𝒟\mathcal{D} and warm-start actions A¯\bar{A}
2: sckpt←GetStates​(envs,M)s_{\text{ckpt}}\leftarrow\textsc{GetStates}(\text{envs},M) ⊳\triangleright MM initial tile states
3: while |𝒟|<target|\mathcal{D}|<\text{target} do
4:   sfull←Repeat​(sckpt,K)s_{\text{full}}\leftarrow\textsc{Repeat}(s_{\text{ckpt}},K) ⊳\triangleright Repeat for KK samples
5:   SetStates​(envs,sfull)\textsc{SetStates}(\text{envs},s_{\text{full}}) ⊳\triangleright Broadcast MM states to M×KM\times K parallel envs
6:   A←SampleSplines​(A¯,σmin,σmax,Nc)A\leftarrow\textsc{SampleSplines}(\bar{A},\sigma_{\mathrm{min}},\sigma_{\mathrm{max}},N_{c}) ⊳\triangleright Sample action sequences around warm start
7:   O,Rsmpc,Rsparse,sNe←SimulateHorizon​(envs,A)O,R_{\mathrm{smpc}},R_{\mathrm{sparse}},s_{N_{e}}\leftarrow\textsc{SimulateHorizon}(\text{envs},A) ⊳\triangleright Batched obs. and rewards
8:   Rτ←∑i=1Tγi⋅Rsmpc​[i]R_{\tau}\leftarrow\sum_{i=1}^{T}\gamma^{i}\cdot R_{\mathrm{smpc}}[i] ⊳\triangleright Compute discounted trajectory reward
9:   j←ArgmaxPerTile​(Rτ)j\leftarrow\textsc{ArgmaxPerTile}(R_{\tau}) ⊳\triangleright Find elite trajectory
10:   A∗←A⁡[j]A^{*}\leftarrow A[j] ⊳\triangleright Elite action sequences
11:   sckpt←sNe​[j]s_{\text{ckpt}}\leftarrow s_{N_{e}}[j] ⊳\triangleright Elite states after NeN_{e} steps
12:   A¯←ShiftTimeForward​(A∗,Ne)\bar{A}\leftarrow\textsc{ShiftTimeForward}(A^{*},N_{e})
13:   ResetStaleWarmStarts​(A¯)\textsc{ResetStaleWarmStarts}(\bar{A})
14:   τ←ExtractCommittedSteps​(O,A,Rsparse,j,Ne)\tau\leftarrow\textsc{ExtractCommittedSteps}(O,A,R_{\mathrm{sparse}},j,N_{e})
15:   𝒟←𝒟∪τ\mathcal{D}\leftarrow\mathcal{D}\cup\tau
16:   if any episodes finished then
17:    DiscardFailedEpisodes​(𝒟)\textsc{DiscardFailedEpisodes}(\mathcal{D})
18:   end if
19: end while
20: return 𝒟\mathcal{D}
Table 4: Hyperparameters for the SMPC expert.
Parameter Value
Horizon TT 5050
Spline control points NcN_{\mathrm{c}} 1010
Frequency ff 5050
Execution steps NeN_{\mathrm{e}} 1010
Discount factor γ\gamma 0.990.99
Samples per tile KK 256256
Minimum noise σmin\sigma_{\mathrm{min}} 0.50.5
Maximum noise σmax\sigma_{\mathrm{max}} 1.01.0

SMPC Tuning in Practice

Costs begin with goal distance and fall penalties, then add gripper proximity, contact restrictions, and smoothing as needed. Sliders support qualitative tuning while observing a few trajectories; each task reportedly takes less than an hour to tune. Data-generation times are 1h39m for reaching, 2h41m for box pushing, 3h28m for uprighting, 3h15m for rolling, and 10h06m for G1. This is rapid manual planner tuning, not the absence of task-specific design.

Appendix D Hardware Setup

Spot inference runs offboard with commands sent over WiFi; G1 inference runs on a mounted computer. OptiTrack tracks robots and objects alongside onboard sensing. The box is a 0.5 m cube weighing 1.2 kg; the tire has 0.33 m radius, 0.34 m width, and 14.3 kg mass. These controlled state-estimation and object conditions are material to the reported hardware transfer.

Concept demonstration

Open image ↗