Literature
Display
← All papers

Combining Constrained Sampling and Reinforcement Learning for Manipulation

Marc Toussaint · Cornelius V. Braun · Armand Jordana · Sayantan Auddy · Eckart Cobo-Briesewitz · Denis Shcherba · Tilman Burghoff · Justin Carpentier

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the 25-page CoRL version, arXiv v3 dated 29 October 2026, including all six appendices and every sampled-state panel. The earlier preprint date remains in the publication metadata. Resources are linked from the author project page and code repository.

At a glance

CSRL changes where manipulation training begins and which goals it attempts. A constrained solver samples collision-free, statically supportable contact configurations; a goal-conditioned RL policy then learns dynamic transitions between them. Nearest-sample projection of interpolated states and a short-to-long curriculum make distant goals easier to learn.

Across six simulated domains, constrained resets greatly improve exploration of whole-body and dexterous contacts. Benefits from interpolation, curriculum and HER depend on the domain. Hardware transfer is a qualitative ball-end-effector pushing demonstration with external state tracking, not a sensor-based general manipulation benchmark.

1 Introduction

Learning contact-rich behavior is difficult because random motion rarely reaches useful contacts. Planning complete trajectories through contact switches is also hard. The paper separates these problems: use physical constraints to generate diverse useful states, then let RL learn how to connect them.

“Universal” means goal-conditioned control within each fixed robot/object environment. The work does not demonstrate one policy across arbitrary new objects or robots. Static feasibility supplies a useful prior, but does not prove that every sampled start–goal pair is dynamically reachable.

Complete original Figure 1.
Figure 1: (a) Complementary to a physics simulator that samples dynamics, we leverage a physics-informed state sampler to globally sample starts and goals. State constraints account for the structure of contact, including coverage of diverse contact modes. (b) Example constrained samples.

2 Related Work

The method combines contact models from planning with reset and goal selection for reinforcement learning.

Model-based Manipulation Planning.

Explicit contact-mode planning and contact-implicit optimization exploit rigid-body structure but face nonsmooth, complementary contact dynamics. CSRL uses those constraints for isolated state generation rather than solving a complete trajectory, drawing on force-based manipulation planning.

Goal-conditioned RL & Universal Policies.

Universal value functions, goal relabeling, hierarchies and curricula usually assume useful goals are readily available. Here the difficult part is constructing feasible contact-rich starts and goals; the RL algorithm itself can be exchanged.

RL for Robot Manipulation.

Prior manipulation RL improves exploration, representations or trajectory-based supervision. CSRL instead changes the state distribution, including whole-arm and environmental contacts without reference motions. Experiments include both sparse rewards and a local progress-shaping variant.

Goal and Reset Distributions in RL.

Reverse curricula, Go-Explore and self-play expand reachable experience, while scene-drop and grasp samplers directly construct useful states. CSRL samples contact configurations globally under a model, avoiding commitment to a single grasp or pushing primitive.

3 Methods

The training recipe has three parts: define feasible contact states, generate a finite offline dataset, and use that dataset to schedule start–goal pairs for RL.

3.1 Problem Formulation

A constrained goal-conditioned MDP defines equality and inequality constraints over states, then optimizes expected discounted return over starts and goals. The mathematical objective specifies a uniform distribution on the feasible set.

The implemented sampler projects uniform box samples and rejects failed contact-mode solves. That procedure is not shown to sample the manifold uniformly; solver attraction regions and mode feasibility can bias coverage. Feasible reset states also need not contain every transient state visited during dynamic control.

Equations 1–3
𝒮c={s∈𝒮:gc(s)≤0,hc(s)=0},\displaystyle\mathcal{S}_{c}=\{s\in\mathcal{S}:g_{c}(s)\leq 0,h_{c}(s)=0\}, (1)
maxπ𝔼s,g∼p0​{Vπ​(s,g)},\displaystyle\max_{\pi}\quad\mathbb{E}_{s,g\sim p_{0}}\!\left\{V^{\pi}(s,g)\right\}~, (2)
with Vπ(s,g)=𝔼{∑tγtR(st​+​1,st,g)|π,st=0=s}\displaystyle\text{with }V^{\pi}(s,g)=\mathbb{E}\!\left\{\textstyle\sum_{t}\gamma^{t}R(s_{t{{\hskip 0.0pt\text{\rm+}\hskip-0.5pt}1}},s_{t},g)\,|\,\pi,s_{t=0}=s\right\} (3)

p0​(s)=𝒰⁡(𝒮c)p_{0}(s)=\mathcal{U}(\mathcal{S}_{c})

3.2 State Sampler

The constraint model assumes rigid shapes and goals that can be held in quasi-static equilibrium. The learned transitions may still be dynamic.

State Constraints.

Each potential shape pair has a binary contact indicator. All shapes must avoid penetration; active contacts must share a point of attack, apply a compressive normal force and remain inside a Coulomb friction cone. Unactuated objects must satisfy static force/torque balance.

The solver requires geometry, friction and Jacobians of distances/normals. These conditions describe model feasibility rather than guaranteed robustness to uncertain real contacts or actuator limits.

Equations 4–7
di​j=0,di​jp=0,dj​ip=0,\displaystyle d_{ij}=0,~d^{p}_{ij}=0,~d^{p}_{ji}=0, (4)
ni​j⊤​fi​j≤0,\displaystyle n_{ij}^{\!\top\!}f_{ij}\leq 0, (5)
‖(𝐈−ni​j​ni​j⊤)​fi​j‖2≤μ2​‖ni​j⊤​fi​j‖2,\displaystyle|\!|({\rm\bf I}-n_{ij}n_{ij}^{\!\top\!})f_{ij}|\!|^{2}\leq\mu^{2}|\!|n_{ij}^{\!\top\!}f_{ij}|\!|^{2}~, (6)
Fi=Mi​gigrav,F_{i}=M_{i}g_{i}^{\text{grav}}, (7)

Non-linear Constrained Sampling.

For one manipulated free object, first choose one, two or three support contacts, then a support subset. Starting from a uniform configuration inside box bounds, an augmented-Lagrangian solver minimizes distance to that seed while satisfying contact and equilibrium constraints. Failed/local-optimum solves are discarded until the dataset is full.

This is simpler than the authors’ earlier MCMC/NLP sampler. The practical restriction to one object and at most three selected supports limits the claimed contact diversity. It samples endpoints, not paths or a certificate that transitions between them exist.

Equation 8
mins,f:,p:12||s−s¯||22s.t.gc(s,f:,p:)≤0,hc(s,f:,p:)=0,\displaystyle\min_{s,f_{:},p_{:}}\textstyle\frac{1}{2}|\!|s-\bar{s}|\!|^{2}_{2}~~\text{s.t.}~~g_{c}(s,f_{:},p_{:})\leq 0,~h_{c}(s,f_{:},p_{:})=0~, (8)
Complete original Figure 2.
Figure 2: The domains for our experiments. For each task the object shall be moved to any feasible given target pose.

3.3 Additional Reset Strategies

Projected interpolation chooses a random start candidate and goal, interpolates their embeddings, and finds the nearest feasible dataset sample with a kd-tree. This preserves feasibility approximately through dataset membership; it is not a continuous constraint projection or a trajectory planner.

An optional curriculum increases the interpolation range from near-goal to full-range and proportionally lengthens episode truncation. Start–goal batches refresh every 5,000 training steps. The embedding combines weighted robot coordinates with object goal markers; sphere/cube weights are 0.5/0.2. HER can additionally relabel achieved goals.

Equations 9
s=argmins∈𝒟‖ϕ⁡(s)−[t​ϕ​(s′)+(1−t)​ϕ​(g)]‖witht∼𝒰⁡[0,1].\displaystyle s=\argmin_{s\in\mathcal{D}}|\!|\phi(s)-[t\phi(s^{\prime})+(1-t)\phi(g)]|\!|\quad\text{with}\quad t\sim\mathcal{U}[0,1]~. (9)

t∼𝒰⁡[0,α]t\sim\mathcal{U}[0,\alpha]

α=(t+Tepoch)/Tcurr\alpha=(t+T_{\text{epoch}})/T_{\text{curr}}

4 Experiments

Default learning uses TD7 with a custom HER extension. Ablations change the state sampler, interpolation, curriculum and RL engine. Final tables report means and standard deviations over five runs; learning curves use standard errors.

4.1 Evaluation Domains

Double-Sphere uses a controllable sphere plus floor/walls; Panda-Sphere allows the sphere to rest against arm links. Sphere-Cube adds object orientation, and Panda-Cube uses gripper/finger contacts. Allegro-Cube requires in-hand position and orientation control; AllegroDown-Cube turns the hand over, making drops difficult to recover from.

Observations have 21, 42, 45, 66 and 96 dimensions, with 3-, 7- or 16-dimensional control. These are separate privileged-state environments with fixed objects. The paper does not specify a shared held-out start–goal test set or evaluation episode count in enough detail to establish identical coverage across different goal samplers.

Double-Sphere Panda-Sphere
goals starts sparse sparse
R R 0.521

±\pm0.394

0.596

±\pm0.325

0.051

±\pm0.073

0.058

±\pm0.076

Ri 0.769

±\pm0.032

0.772

±\pm0.036

0.073

±\pm0.075

0.035

±\pm0.069

Ric 0.766

±\pm0.017

0.755

±\pm0.044

0.112

±\pm0.058

0.059

±\pm0.076

C R 0.867

±\pm0.018

0.685

±\pm0.373

0.005

±\pm0.004

0.007

±\pm0.002

Ri 0.877

±\pm0.010

0.697

±\pm0.386

0.154

±\pm0.175

0.012

±\pm0.013

Ric 0.892

±\pm0.016

0.850

±\pm0.030

0.287

±\pm0.230

0.021

±\pm0.023

C C 0.963

±\pm0.011

0.953

±\pm0.004

0.527

±\pm0.255

0.472

±\pm0.450

Ci 0.972 ±\pm0.007 0.783

±\pm0.430

0.630

±\pm0.415

0.701

±\pm0.466

Cic 0.972 ±\pm0.007 0.969 ±\pm0.010 0.929 ±\pm0.029 0.932 ±\pm0.007
Table 1: Comparison of alternative goal and reset sampling methods. We report the mean success rate of the final policy with std. deviation over 5 training runs and highlight the best mean per column. Constrained sampling (C) or Random drops (R) is used to sample goals and starts, optionally combined with projected interpolation (i) and a curriculum (c) for sampling starts.
Double-Sphere Panda-Sphere
goals starts TD7HER{}_{\text{HER}} TD7 TD3sb{}^{\text{sb}} TD7HER{}_{\text{HER}} TD7 TD3sb{}^{\text{sb}}
C C 0.963

±\pm0.011

0.923

±\pm0.052

0.131

±\pm0.050

0.527

±\pm0.255

0.324

±\pm0.296

0.008

±\pm0.005

Ci 0.972

±\pm0.007

0.982 ±\pm0.005 0.806

±\pm0.024

0.630

±\pm0.415

0.837

±\pm0.139

0.043

±\pm0.014

Cic 0.972

±\pm0.007

0.983 ±\pm0.005 0.800

±\pm0.006

0.929 ±\pm0.029 0.814

±\pm0.104

0.066

±\pm0.009

Table 2: Evaluation of alternative RL methods inside CSRL: The stablebaselines3 TD3 implementation (with same hyperparameters as TD7), a HER extension of TD7, and the sb3 HER extension of TD3. (Bold: Best per problem.)
(a) 
goals starts Sphere-Cube Panda-Cube Allegro-Cube AllegroDown-Cube
C C 0.878 ±\pm0.032 0.949 ±\pm0.009 0.954 ±\pm0.011 0.903 ±\pm0.025
Ci 0.912 ±\pm0.048 0.966 ±\pm0.005 0.957 ±\pm0.006 0.901 ±\pm0.023
Cic 0.928 ±\pm0.056 0.964 ±\pm0.006 0.966 ±\pm0.010 0.891 ±\pm0.016
 (b) 
goals starts Double-Sphere Panda-Sphere
R Rrev 0.846 ±\pm0.021 0.197 ±\pm0.042
C Crev 0.988 ±\pm0.023 0.274 ±\pm0.018
Table 3: (a) Performance on the cube domains. (b) Comparison to reverse curriculum.

4.2 Comparison of Alternative State Samplers and Reset Strategies

The random baseline drops uniformly initialized configurations in simulation until stable and rejects out-of-bounds states. With constrained goals, replacing those resets with constrained resets raises Panda-Sphere success from 0.005 to 0.527; adding interpolation/curriculum reaches 0.929. The corresponding sparse setting reaches 0.932. Double-Sphere reaches about 0.972.

The add-ons are not uniformly better. HER-free TD7 wins Double-Sphere in Table 2, while HER helps the full Panda curriculum. TD3 uses TD7 hyperparameters, so its weak results are not a fully tuned algorithm comparison. Table 2’s caption mentions TD3 HER although its columns are TD7-HER, TD7 and TD3-sb.

For cubes, full curriculum scores 0.928/0.964/0.966/0.891 across the four domains. Panda-Cube interpolation alone is slightly higher, and AllegroDown does best with plain constrained resets at 0.903. Constrained reverse curriculum scores 0.988 on Double-Sphere, above interpolation/curriculum’s 0.972, contradicting the claim that interpolation wins both sphere domains. Panda-Sphere strongly favors interpolation/curriculum over reverse curriculum’s 0.274.

Figure 3 retains all eight learning plots. Its caption says 80 environment samples per training step, whereas Appendix B specifies 2,000 samples per 20 updates, implying 100; the conversion is unresolved.

Complete original Figure 3.
Figure 3: Success rate during RL training. Mean and std. error (shading) over 5 independent runs for each method. The x-axis are TD7 training steps, which each corresponds to 80 environment samples in our setting.

4.3 Discussion of Found Behaviors

Qualitative behaviors include balancing the sphere, scooping it onto arm links, rotating a cube onto an edge/corner, gripper squeezing and fingertip balancing. Some agile near-goal cube behavior misses the precision threshold. The inverted-hand domain trains useful behavior despite drop risk; it is not evidence of zero-shot transfer of the upright policy.

4.4 Real-World Transfer

A Sphere-Cube policy controls a ball attached to a Panda arm through inverse kinematics. OptiTrack supplies cube state. Figure 4 pairs real and synchronized simulated views by action index; each blocking IK movement sets wall-clock duration.

This demonstrates basic pushing transfer with privileged tracking. The paper reports no hardware success rate, trial count or fixed real-time action frequency, and does not transfer its dexterous-hand policies to hardware.

Complete original Figure 4.
Figure 4: Real (top) vs. synced view (bottom), paired by action index tt (number of policy actions taken). The synced view mirrors the OptiTrack cube pose and the robot’s joint state; per-action wall-clock time is set by a blocking IK move.

5 Conclusion

Constraint-based reset design is the central contribution: useful contact states can improve RL exploration without solving every transition beforehand. Interpolation and a curriculum are inexpensive additions once the state dataset exists, but their gains vary across domains.

The explicit limitations are privileged state access, no sensor-based student policy, potential sim-to-real mismatch in contact/admittance observations, and expensive CPU constraint solving. Dedicated GPU solvers are future work. Single-object sampling, restricted support modes and unproven uniformity/reachability further delimit “universal” coverage.

Acknowledgments

Support includes BMFTR/Robotics Institute Germany, Amazon Fulfillment Technologies and Robotics, France 2030 (PEPR O2R, PR[AI]RIE-PSAI ANR-23-IACL-0008 and RODEO ANR-24-CE23-5886), and EU ARTIFACT 101165695 and AGIMUS 101070165. The authors state that their views do not necessarily represent the funders.

Appendix A Gym Environment Details

MuJoCo advances at 1 ms; actions arrive every 50 ms. PD motors track a quadratic reference overwritten by each action, with λ equal to two action intervals. Equation 10 makes the reference reach its initial position plus half the action after λ, while preserving reference position at the update.

Sphere goals use the object center; cube goals use its center and four corner markers. Success is the Euclidean marker-vector error below the domain threshold, not simply a position error in meters. Sparse reward gives one on success; local shaping rewards progress within ten thresholds. The claimed telescoping shaping bounds concern undiscounted episode sums.

Observations include joint positions/velocities, object velocity, control error, scene points and goal error. The control error exposes the reference state and indirectly contact forces. The prose says five Allegro fingertip points, but its 27 scene coordinates minus 15 object coordinates imply four 3D fingertip points; all original table dimensions remain.

At reset, arm/sphere reference positions match the configuration. Allegro resets instead back-project sampled contact forces to joint torques and offset the reference by torque divided by proportional gain, preloading a supporting grasp. This only approximately balances the cube in simulation. Table 4 preserves all training lengths, truncation times, thresholds, action scales, replay capacities and network widths.

Equations 10–11
r⁡(t)\displaystyle r(t) =a−2​λ​v02​λ2​(t−t0)2+v0​(t−t0)+r0.\displaystyle=\frac{a-2\lambda v_{0}}{2\lambda^{2}}(t-t_{0})^{2}+v_{0}(t-t_{0})+r_{0}~. (10)
y\displaystyle y =(q,q˙,q˙o,r−q,P,G∗−G)\displaystyle=(q,\dot{q},\dot{q}_{o},r-q,P,G^{*}-G) (11)
qq PP GG obs. dim TcurrT_{\text{curr}} TendT_{\text{end}} ttrunct_{\text{trunc}} ϵG\epsilon_{G} ascalea_{\text{scale}} BsizeB_{\text{size}} hench_{\text{enc}} hQ,πh_{\text{Q,$\pi$}}
Double-Sphere 3 3 3 21 50k 75k 1. 0.01 2. 1M 256 256
Panda-Sphere 7 12 3 42 50k 100k 2. 0.01 5. 2M 512 256
Sphere-Cube 3 15 15 45 100k 150k 3. 0.03 2. 4M 512 256
Panda-Cube 7 24 15 66 100k 200k 3. 0.05 5. 4M 1024 512
Allegro*-Cube 16 27 15 96 100k 200k 3. 0.02 2. 5M 1024 512
Table 4: Dimensionality, training and environment parameters for each environments: robot joint state qq (which also gives the action dimension and control error dimension), additional scene point observations PP, goal scene points GG, and total observation dimension. RL training steps TendT_{\text{end}}, curriculum length TcurrT_{\text{curr}}, gym truncation time ttrunct_{\text{trunc}}, goal threshold ϵG\epsilon_{G}, action scale ascalea_{\text{scale}}, replay buffer size BsizeB_{\text{size}}, and TD7 zs and enc layer size hench_{\text{enc}}.
Equation

r0=r⁡(t0)r_{0}=r(t_{0})

v0=q˙​(t0)v_{0}=\dot{q}(t_{0})

r⁡(t0+λ)=r0+12​ar(t_{0}+\lambda)=r_{0}+\frac{1}{2}a

λ=2​τaction\lambda=2\tau_{\text{action}}

dt=‖Gt−G∗‖d_{t}=|\!|G_{t}-G^{*}|\!|

dt​+​1≤ϵGd_{t{{\hskip 0.0pt\text{\rm+}\hskip-0.5pt}1}}\leq\epsilon_{G}

d¯t=[1.−dt/(10ϵG)]+∈[0,1]\bar{d}_{t}=[1.-d_{t}/(10\epsilon_{G})]_{+}\in[0,1]

r=d¯t​+​1−d¯tr=\bar{d}_{t{{\hskip 0.0pt\text{\rm+}\hskip-0.5pt}1}}-\bar{d}_{t}

r0=s0+τ/kpr_{0}=s_{0}+\tau/k_{p}

ϕ⁡(s)=(wq​q,G)∈ℝn+3​nG\phi(s)=(w_{q}q,G)\in{\mathbb{R}}^{n+3n_{G}}

Appendix B TD7 & HER Extension Details

Following FastTD3-style parallel collection, each iteration adds 2,000 transitions and performs 20 TD7 updates with batch size 10,000. Learning rate is 10⁻³ except for Allegro. Shaped-return targets are clipped nonnegative as a stabilizing heuristic.

The custom HER buffer tracks previous transitions of each parallel episode. After at least three steps, the current next state becomes a relabeled goal for one earlier transition; goal-error observations, rewards and termination flags are all recomputed. This is the paper’s implementation-specific response to the cited wrapper behavior, not a statement about every current HER implementation.

Appendix C Statistics of Constraint Sampling

The offline datasets contain 20,000 states per sphere/Panda-cube domain and 50,000 per Allegro domain. Feasible-solve rates span 5.1%–48.8%; per-sample times span 38.7–576.8 ms, and total times span 0.22–4.80 hours.

Panda-Cube has the lowest feasible rate and most constraint queries; AllegroDown takes longest overall because it generates 50,000 states. Geometry-query cost and failed solves are material preprocessing costs, distinct from the RL training curves. Table 5 preserves all measurements.

Double- Sphere Panda- Sphere Sphere- Cube Panda- Cube Allegro- Cube AllegroDown- Cube
number |𝒟||\mathcal{D}| of samples 20k 20k 20k 20k 50k 50k
feasibility rate 0.356 0.116 0.214 0.051 0.488 0.121
queries/sample 578.3 2934.9 876.0 6681.5 394.3 2645.7
time/sample [msec] 38.7 221.3 44.6 576.8 77.6 345.2
total time [h] 0.22 1.23 0.24 3.20 1.08 4.80
Table 5: Compute statistics for constrained sampling: How many NLP solver runs on (8) find feasible samples; how often are constraint functions g,hg,h queried per feasible sample, and wall clock time for a single and for all samples.

Appendix D Real World Transfer Setup

The physical Panda carries a custom ball, positioned 16 cm from the flange. Twelve OptiTrack Flex13 cameras track the cube. Figure 5 retains the setup photograph and exact dimensional drawing. The prose specifies a 3 cm ball radius, but the drawing labels its diameter 80 mm, implying a 4 cm radius; that mismatch is unresolved.

Complete original Figure 5.
Figure 5: (a) Picture of the real world setup, showing the Franka Emika Panda manipulator, the custom end-effector and the cube. (b) Dimensions of the custom end-effector in millimeter.

Appendix E Comparison to Behavior Cloning from Diverse Trajectory Data

The appendix preserves a negative result from arXiv v1: generating diverse open-loop trajectories was costly, and adding a BC loss contributed less than using their states as resets. The current version reports the method and qualitative interpretation but refers to v1 for the numerical results; later RL hyperparameter improvements prevent direct score comparison.

E.1 Zero-Order Optimization of Open-Loop Trajectories

For sampled starts/goals, CMA-ES searches a low-dimensional B-spline action sequence under deterministic simulated dynamics. The terminal objective compares weighted object position, robot configuration, their velocities and contact-proximity features. Nearby-support features supply a useful search signal before object motion occurs.

There is no explicit control-effort cost; smoothness comes from the spline parameterization. Only trajectories meeting the terminal threshold enter the dataset. This is a black-box optimizer over a known simulator, not a demonstration that model-based trajectory generation is impossible.

Equations 12–13
minθ⁡‖ϕ⁡(g)−ϕ⁡(x⁡(T))‖2​s.t.​x​(0)=s,x˙=f⁡(x,u),\displaystyle\min_{\theta}|\!|\phi(g)-\phi(x(T))|\!|^{2}~~\text{s.t.}~~x(0)=s,\dot{x}=f(x,u)~, (12)
𝒟u={(si,gi,ui​(t),xi​(t))}i=1U,\displaystyle\mathcal{D}_{u}=\left\{\left(s_{i},g_{i},u_{i}(t),x_{i}(t)\right)\right\}_{i=1}^{U}~, (13)

ci=1−[d1​i/σ][0,1]c_{i}=1-[d_{1i}/\sigma]_{[0,1]}

ϕ⁡(x)=(2​p,1​q,0.1​p˙,0.1​q˙,0.1​c)∈ℝ6+2​n+m−1\phi(x)=(2p,1q,0.1\dot{p},0.1\dot{q},0.1c)\in{\mathbb{R}}^{6+2n+m-1}

E.2 Behavior Cloning from Trajectory Data

The proposed TD7+BC extension follows TD3+BC. Aligned quadratic spline segments are converted into the same 20 Hz action parameterization, producing state–goal–action examples that reproduce the deterministic simulated path.

Equation 14 prints a maximization with a plus sign before λ times squared imitation error. With positive λ that would reward disagreement; the sign convention is not clarified, so the original formula is retained. The experiment also changes reset states to trajectory states, making it essential to separate the BC-loss effect from the reset-distribution effect.

Equations 14–15
maxπ⁡𝔼s,g,a∼𝒟​{Q⁡(s,g,π⁡(s,g))+λ​(π⁡(s,g)−a)2},\displaystyle\max_{\pi}\mathbb{E}_{s,g,a\sim\mathcal{D}}\!\left\{Q(s,g,\pi(s,g))+\lambda(\pi(s,g)-a)^{2}\right\}~, (14)
𝒟BC={(sj,gj,aj)}j=0U⋅T,\displaystyle\mathcal{D}_{\text{BC}}=\{(s_{j},g_{j},a_{j})\}_{j=0}^{U\cdot T}~, (15)

E.3 Results: What is the role of BC in effective universal RL?

The authors report that trajectory-derived resets substantially improve learning while the imitation term has little additional impact. Reactive learned rollouts can correct disturbances that the open-loop spline plans do not handle. This is a negative result for the tested data/training regime, not a general rejection of behavior cloning; current-version numerical comparisons are absent.

Appendix F Random Constrained Samples from our Domains

Figures 6–11 preserve all six 28-panel sample grids, totaling 168 configurations. The introduction’s “four domains” is stale wording. Figures 10 and 11 retain the original erroneous “panda cube” captions; their images depict Allegro-Cube and AllegroDown-Cube, respectively. Samples illustrate coverage, not a measured uniform distribution or guaranteed executable transitions.

Complete original Figure 6.
Figure 6: Constrained samples from the double sphere domain.
Complete original Figure 7.
Figure 7: Constrained samples from the panda sphere domain.
Complete original Figure 8.
Figure 8: Constrained samples from the sphere cube.
Complete original Figure 9.
Figure 9: Constrained samples from the panda cube domain.
Complete original Figure 10.
Figure 10: Constrained samples from the panda cube domain.
Complete original Figure 11.
Figure 11: Constrained samples from the panda cube domain.
Concept demonstration

Open image ↗