Literature
Display
← All papers

SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training

Jiaheng Hu · Peter Stone · Roberto Martín-Martín

· CoRL 2025 ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

At a glance

SLAC learns a reusable latent action decoder in a coarse simulator, then learns a task policy directly on the robot. Disentangled skill discovery encourages each latent factor to control a selected state factor; safety penalties discourage damaging exploration. The downstream learner combines masked value-function decomposition with frequent off-policy updates.

On a bimanual Tiago, it reaches 70–90% success over ten evaluation trials per task after 50 minutes of real-world interaction. The expanded version also tests coordinated virtual drones. Safety violations are reduced but not eliminated, and real-world interaction time excludes the simulation pretraining budget.

1 Introduction

Accurate simulation is difficult for contact, deformable objects and effects such as wiping marker traces. Direct real-world exploration avoids those modeling errors but is expensive in high-dimensional control spaces. SLAC transfers an action repertoire that reflects approximate affordances, then learns how to use that repertoire from real rewards.

The decoder supplies temporally extended, factored actions rather than a complete transferred task policy. A high-level policy can compensate for some behavior mismatch by choosing different latent actions. The approach still depends on adequate skill coverage, scene geometry, selected state factors and engineered rewards.

Complete original Figure 1.
Figure 1: SLAC uses a task-agnostic action space trained in low-fidelity simulation (left) to learn downstream tasks in the real world. This latent action space is safe, temporally extended, and disentangled, enabling a bimanual mobile manipulator to solve challenging contact-rich whole-body tasks (right) with less than an hour of autonomous real-world interactions.

Novelty Statement:

This edition follows the expanded arXiv v5 manuscript dated 17 July 2026. Its CoRL 2025 predecessor was titled Simulation-Pretrained Latent Action Space for Whole-Body Real-World RL, with the same three authors.

The expansion adds multi-robot experiments, Q-decomposition during skill pretraining, and automatic inference of action–reward dependencies. These additions belong to the expanded manuscript, not necessarily to the original conference version.

2 SLAC Problem Formulation

The real task is a partially observed decision process with a composite reward. A low-fidelity simulator shares the native control interface and approximately preserves relevant affordances, while omitting detailed real dynamics, appearance and task-specific rewards.

The decoder maps a latent action and low-dimensional observations shared across simulation and reality into native controls. The task policy maps real onboard observations to latent actions. Their composition forms the deployed hierarchical policy; training the high-level policy in reality is what adapts action selection to real task outcomes.

Equations 1–2
π∗=arg⁡maxπ​𝔼π​[∑t=0∞γt​R𝑡𝑎𝑠𝑘​(st,at)]\pi^{*}=\arg\max_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{\mathit{task}}(s_{t},a_{t})\right] (1)
π⁡(a|o)=∫zπ𝑑𝑒𝑐​(a|o𝑑𝑒𝑐,z)​π𝑡𝑎𝑠𝑘​(z|o)​𝑑z\pi(a|o)=\int_{z}\pi_{\mathit{dec}}(a|o_{\mathit{dec}},z)\,\pi_{\mathit{task}}(z|o)\,dz (2)
Complete original Figure 2.
Figure 2: The two-step SLAC procedure to enable real-world policy learning. (Left) In the first step, SLAC learns a Latent Action Decoder that maps each latent action, z∈𝒵z\in\mathcal{Z}, to a sequence of low-level robot actions, (a0,…,aT),at∈𝒜(a_{0},\ldots,a_{T}),a_{t}\in\mathcal{A}. This decoder is learned in low-fidelity simulation via unsupervised skill discovery with novel objectives that encourage the robot to independently control different state features (e.g., camera directions, contacts with table, base locations) while being safe. (Right) In the second step, once the decoder is trained, the robot learns downstream tasks with RL in the real world using the SLAC latent action space. The task policy directly takes in the onboard sensor observations of the robot (i.e., images, proprioception) and outputs latent actions zz that are decoded into safe robot actions. SLAC applies Factorized Latent-Action SAC to optimize the policy for downstream tasks with multi-term reward (e.g., look at the objects, keep a bag close, sweep the trash) directly in the real world with very few samples, converging in less than an hour, by taking advantage of high-frequency off-policy updates and masked Q decomposition.

3 SLAC Step 1: Learning Latent Action Space in Simulation

Unsupervised skill discovery pretrains the decoder without a downstream task reward. The simulator must still expose useful geometry, contacts and unsafe regions. A fixed latent action drives several low-level steps, reducing the high-level decision frequency. Algorithm 1 retains the complete pretraining loop.

3.1 Disentangled Unsupervised Skill Discovery for Structured Latent Action Space

Disentangled Unsupervised Skill Discovery (DUSDi) associates different latent factors with different state entities. A structured repertoire can be recombined more readily than an opaque one-hot skill label whose effects change many state components at once. The method builds on the authors’ earlier hierarchical-RL work.

3.1.1 DUSDi Objective

For N selected state factors, the latent action contains N categorical variables with k choices each, yielding kᴺ combinations. The objective maximizes mutual information between each latent factor and its corresponding state factor while penalizing information about all other factors.

A small penalty weight, recommended around 0.1, allows unavoidable physical coupling. The objective encourages selective effects; it cannot make inherently coupled dynamics independent or guarantee perfect disentanglement. State-factor selection is an input to the method.

Equation 3
𝒥⁡(θ)=∑i=1NI⁡(𝒮i,𝒵i)−λ​I​(𝒮¬i,𝒵i),\mathcal{J}(\theta)=\sum_{i=1}^{N}I({\bS}^{i};{\bZ}^{i})-\lambda I({\bS}^{\neg i};{\bZ}^{i}), (3)
Algorithm 1 SLAC Unsupervised Latent Action Space Learning
1 Create 𝑠𝑖𝑚\mathit{sim} environment, skill prior distribution p⁡(z)p(z), replay buffer 𝒟𝑠𝑘\mathcal{D}_{\mathit{sk}};
2 Initialize latent action decoder π𝑑𝑒𝑐\pi_{\mathit{dec}}, N discriminators qϕiq^{i}_{\phi}, qψiq^{i}_{\psi}, and N value functions Q𝑑𝑒𝑐iQ^{i}_{\mathit{dec}};
3 for k←1k\leftarrow 1 to s​k​i​l​l​_​l​e​a​r​n​i​n​g​_​e​p​o​c​h​sskill\_learning\_epochs do
4     Sample latent action z∼p⁡(z)z\sim p(z);
5     for j←1j\leftarrow 1 to s​t​e​p​s​_​p​e​r​_​s​k​i​l​lsteps\_per\_skill do
6         (o𝑑𝑒𝑐,a,o′𝑑𝑒𝑐)←sim.step​(π𝑑𝑒𝑐​(a∣o𝑑𝑒𝑐,z))(o_{\mathit{dec}},a,{o^{\prime}}_{\mathit{dec}})\leftarrow\texttt{sim.step}(\pi_{\mathit{dec}}(a\mid o_{\mathit{dec}},z));
7         Store transition (o𝑑𝑒𝑐,a,z,o′𝑑𝑒𝑐)(o_{\mathit{dec}},a,z,{o^{\prime}}_{\mathit{dec}}) into replay buffer 𝒟𝑠𝑘\mathcal{D}_{\mathit{sk}};
8         for i←1i\leftarrow 1 to n​_​u​p​d​a​t​e​sn\_updates do
9             Sample mini-batch {(o𝑑𝑒𝑐,a,z,o′𝑑𝑒𝑐}\{(o_{\mathit{dec}},a,z,{o^{\prime}}_{\mathit{dec}}\} from 𝒟𝑠𝑘\mathcal{D}_{\mathit{sk}};
10             Update qϕq_{\phi} and qψq_{\psi} via gradient ascent;
11             Calculate intrinsic reward rr based on Eq. 7;
12             Update Q𝑑𝑒𝑐Q_{\mathit{dec}} with rr using SAC critic update and Q-Decomposition;
13             Update π𝑑𝑒𝑐\pi_{\mathit{dec}} with Q𝑑𝑒𝑐Q_{\mathit{dec}} using SAC policy update;
14 return π𝑑𝑒𝑐\pi_{\mathit{dec}}
  Note: Omitting standard SAC steps (e.g., target network creation and updates) for simplicity.

3.1.2 DUSDi Tractable Optimization

Two discriminators predict each latent factor from its intended state factor and from the remaining state factors. Alternating discriminator updates with RL converts their log probabilities into an intrinsic reward. Perfect posterior estimators recover the intended mutual-information objective.

Both individual mutual-information estimates use variational lower bounds. Subtracting one bound from another does not make the combined approximate reward a guaranteed lower bound on the complete objective. Fixed target distributions can optionally focus exploration toward chosen state values; the Tiago experiments use this option.

Equations 4–6
I⁡(𝒮i,𝒵i)=H⁡(𝒵i)−H⁡(𝒵i|𝒮i)≥C+𝔼z,s​log⁡qϕi​(zi|si),\displaystyle I({\bS}^{i};{\bZ}^{i})=H({\bZ}^{i})-H({\bZ}^{i}|{\bS}^{i})\geq C+\mathbb{E}_{z,s}\log{q^{i}_{\phi}}(z^{i}|s^{i}), (4)
I⁡(𝒮¬i,𝒵i)≥C+𝔼z,s​log⁡qψi​(zi|s¬i),\displaystyle I({\bS}^{\neg i};{\bZ}^{i})\geq C+\mathbb{E}_{z,s}\log{q^{i}_{\psi}}(z^{i}|s^{\neg i}), (5)
r𝑑𝑢𝑠𝑑𝑖​(s,a)≜∑i=1Nlog⁡qϕi​(zi|si)−λ​log⁡qψi​(zi|s¬i)r_{\mathit{dusdi}}(s,a)\triangleq\sum_{i=1}^{N}\log{q^{i}_{\phi}(z^{i}|s^{i})}-\lambda\log{q^{i}_{\psi}(z^{i}|s^{\neg i})} (6)

3.2 Safety-Aware Latent Action Learning

The skill reward adds penalties for large actions, abrupt action changes, collisions and excessive force. The same coefficients are used across the reported environments. These are soft training preferences, not hard safety constraints or a certificate of safe execution. Algorithm 2 preserves the subsequent real-world learning procedure.

Equation 7
r𝑙𝑎𝑡𝑒𝑛𝑡=r𝑑𝑢𝑠𝑑𝑖+r𝑠𝑎𝑓𝑒r_{\mathit{latent}}=r_{\mathit{dusdi}}+r_{\mathit{safe}} (7)
Algorithm 2 FLA-SAC for Real-World Downstream Task Learning
1 Initialize task policy π𝑡𝑎𝑠𝑘​(z∣o)\pi_{\mathit{task}}(z\mid o), factored Q-functions {Qi}i=1m\{Q^{i}\}_{i=1}^{m};
2 Load pre-trained latent action decoder π𝑑𝑒𝑐\pi_{\mathit{dec}};
3 Initialize replay buffer 𝒟\mathcal{D} with NumStartSteps of random transitions ; // Standard SAC warmstart
4 Dependency matrix ℬ\mathcal{B} ←\leftarrow infer(𝒟\mathcal{D}) ; // Sec. 4.2
5 for k←1k\leftarrow 1 to t​a​s​k​_​l​e​a​r​n​i​n​g​_​s​t​e​p​stask\_learning\_steps do
6     z←π𝑡𝑎𝑠𝑘​(z∣o)z\leftarrow\pi_{\mathit{task}}(z\mid o) ;
7     rsum←[0]mr_{\text{sum}}\leftarrow[0]^{m} ;
8     for t←1\text{t}\leftarrow 1 to s​t​e​p​s​_​p​e​r​_​s​k​i​l​lsteps\_per\_skill do
9         (odec,r=[ri]i=1m,o′)←robot.real_world_step​(π𝑑𝑒𝑐​(a∣odec,z))(o_{\text{dec}},r=[r^{i}]_{i=1}^{m},o^{\prime})\leftarrow\texttt{robot.real\_world\_step}(\pi_{\mathit{dec}}(a\mid o_{\text{dec}},z));
10         rsum=rsum+rr_{\text{sum}}=r_{\text{sum}}+r ;
11     Store (o,z,rsum,o′)(o,z,r_{\text{sum}},o^{\prime}) into replay buffer 𝒟\mathcal{D};
12     for j←1j\leftarrow 1 to u​t​d​_​r​a​t​i​outd\_ratio do
13         Sample mini-batch {(o,z,r,o′)}\{(o,z,r,o^{\prime})\} from 𝒟\mathcal{D} with small batch size;
14         Update Qi​(o,ℬi⊙z)Q^{i}(o,\mathcal{B}_{i}\odot z) with rir^{i} in parallel for all i=1,…,mi=1,\dots,m;
15         Update π𝑡𝑎𝑠𝑘​(z^∣o)\pi_{\mathit{task}}(\hat{z}\mid o) with Q=∑iQiQ=\sum_{i}Q^{i} using SAC loss, with z^\hat{z} sampled via Gumbel-Softmax (Eq. 12) for differentiability;
16 return π𝑡𝑎𝑠𝑘\pi_{\mathit{task}}
  Note: Omitting standard SAC steps (e.g., target network creation and updates) for simplicity.

3.3 Accelerating Latent Action Space Learning through Q-Decomposition

For a fixed policy and an additive reward, linearity of expectation gives an exact decomposition of the total Q-function into one component per reward term. SLAC trains those critics separately and sums them for policy improvement; an SAC entropy bonus can be treated as another additive term.

Separating targets may ease credit assignment and reduce interference in optimization. The identity alone does not guarantee lower estimator variance or faster convergence; the empirical pretraining ablation supplies evidence for these settings.

Equation 8
Qtotal,π​(s,a)\displaystyle Q_{\text{total},\pi}(s,a) =𝔼π​[∑t=0∞γt​rtotal,t]\displaystyle=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{\text{total},t}]
=𝔼π​[∑i=1m∑t=0∞γt​rti]\displaystyle=\mathbb{E}_{\pi}[\sum_{i=1}^{m}\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}]
=∑i=1m𝔼π​[∑t=0∞γt​rti]\displaystyle=\sum_{i=1}^{m}\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}]
=∑i=1mQπi​(s,a)\displaystyle=\sum_{i=1}^{m}Q^{i}_{\pi}(s,a) (8)

4 SLAC Step 2: Sample-Efficient Learning of Downstream Tasks in the Real World

Factorized Latent-Action Soft Actor-Critic (FLA-SAC) trains the task policy over the frozen decoder. It combines reward-specific critics, action masks inferred from exploratory data, a high update-to-data ratio, smaller minibatches and differentiable relaxed categorical actions.

Complete original Figure 3.
Figure 3: SLAC infers the dependencies between reward terms and latent action dimensions through estimating and thresholding the conditional-mutual information (CMI) on randomly collected robot interaction data. To ensure computational efficiency, we introduce element-wise max as a mechanism to model different action conditioning.

4.1 Factorized Policy Learning via Masked Q-Decomposition

Each task reward component receives its own critic. A binary dependency matrix masks latent dimensions judged irrelevant to that component, so both value learning and actor gradients avoid unnecessary action inputs. The intended benefit is less spurious correlation and clearer credit assignment.

The source defines edges using conditional dependence of immediate reward on latent actions given state and the other actions. Applying that structure to long-horizon Q-values assumes the selected dependencies remain adequate for future return; immediate reward independence alone does not establish that in arbitrary dynamics.

Equation 9
Ri⟂⟂𝒵j|s0,𝒵/𝒵jR^{i}\;\perp\!\!\!\perp\;{\bZ}^{j}\;\Big|\;s_{0},\;\bZ/{\bZ}^{j} (9)

4.2 Autonomous Detection of Action-Reward Dependencies

Random interactions collected for the replay-buffer warm start also train dependency models. For each reward channel, separate state/action encoders produce 128-dimensional features, combined by an elementwise maximum. Masking an action feature to −∞ lets one model predict the reward with or without that input.

The model maximizes full and partially masked reward likelihoods on training data. Held-out log-likelihood ratios estimate conditional mutual information; pairs above a threshold become dependency edges. This uses m models for m reward terms rather than one model per action–reward pair, following causal dynamics modeling. Finite exploratory coverage, model error and threshold choice can still yield incorrect masks.

Equations 10–11
CMI​(zi,rj)\displaystyle\text{CMI}(z_{i},r_{j}) =𝔼zi,s,rj​[l​o​g​p⁡(zi,rj|{s,𝐳\zi})p⁡(zi|{s,𝐳\zi})​p​(rj|{s,𝐳\zi})]\displaystyle=\mathbb{E}_{z_{i},s,r_{j}}\left[log\frac{p(z_{i},r_{j}|\{s,\mathbf{z}\backslash z_{i}\})}{p(z_{i}|\{s,\mathbf{z}\backslash z_{i}\})p(r_{j}|\{s,\mathbf{z}\backslash z_{i}\})}\right] (10)
=𝔼zi,s,rj​[l​o​g​p⁡(rj|{s,𝐳})p⁡(rj|{s,𝐳\zi})]\displaystyle=\mathbb{E}_{z_{i},s,r_{j}}\left[log\frac{p(r_{j}|\{s,\mathbf{z}\})}{p(r_{j}|\{s,\mathbf{z}\backslash z_{i}\})}\right]
L=∑j[logp^(rj|s,𝐳)\displaystyle L=\sum_{j}[\log\hat{p}(r_{j}|s,\mathbf{z}) +logp^(rj|{s,𝐳\zi})]\displaystyle+\log\hat{p}(r_{j}|\{s,\mathbf{z}\backslash z_{i}\})] (11)
Complete original Figure 4.
Figure 4: Training curves for SLAC and baseline methods. SLAC can learn challenging contact-rich whole-body mobile manipulation tasks within an hour of real-world interactions, significantly outperforming prior methods.
Table 1: We compare the number of successful trials (↑\uparrow) over 10 rollouts after training and the total safety violation counts (↓\downarrow) during training between SLAC and baseline methods across four tasks. In all four tasks, SLAC achieves the highest success rate while also inducing the least number of safety violations.
Method / Task Board Board-Obstacle Table-Tray Table-Bag
Success # Unsafe # Success # Unsafe # Success # Unsafe # Success # Unsafe #
SLAC (ours) 9 / 10 1 8 / 10 4 9 / 10 0 7 / 10 0
SERL (Luo et al., 2024) 0 / 10 8 0 / 10 22 0 / 10 6 0 / 10 9
Sim2Real (Tobin et al., 2017) 2 / 10 - 2 / 10 - 4 / 10 - 0 / 10 -
RLPD (Ball et al., 2023) 4 / 10 34 2 / 10 37 3 / 10 26 0 / 10 33

4.3 Further Boosting Sample Efficiency via High UTD Ratio and Regularization

Ten actor–critic updates per new high-level transition reuse limited real experience. The reported setup reduces minibatch size from 256 to 64, which empirically worked better than the tested layer normalization and critic ensembles. The proposed explanation is useful gradient noise; it is not a general guarantee that smaller batches prevent overfitting.

4.4 Handling Large Discrete Action Space

Enumerating every joint categorical action becomes expensive as the factored space grows. A non-straight-through Gumbel-softmax relaxation permits gradients through sampled latent actions, with temperature fixed at 1.0. The source reports feasibility for action spaces of order one million combinations. The retained formula describes the relaxed sample, not an exact discrete gradient.

Equation 12
z^​(s)=\displaystyle\hat{z}(s)= 𝑠𝑜𝑓𝑡𝑚𝑎𝑥⁡(log⁡πθ​(z∣s)+gzτ),\displaystyle\mathit{softmax}\left(\frac{\log\pi_{\theta}(z\mid s)+g_{z}}{\tau}\right), (12)
gz∼𝐺𝑢𝑚𝑏𝑒𝑙⁡(0,1)\displaystyle g_{z}\sim\mathit{Gumbel}(0,1)

5 Experimental Results

Evaluation covers four physical mobile-manipulation tasks, two simulated multi-robot tasks, and component ablations. Real-world trials, simulation curves and skill-prediction accuracy answer different questions and use different evaluation protocols.

5.1 SLAC for Real-World Mobile Manipulation

Two coarse iGibson scenes—table and whiteboard—are reported to take under 20 minutes each to construct from existing assets. The real robot uses RGB-D observations and proprioception, with a randomly initialized PointNet and MLP. A scene-specific decoder is reused across its two downstream tasks.

SLAC, zero-shot Sim2Real and RLPD receive 10 million simulation steps. SLAC, SERL and RLPD receive 30,000 native real-world steps, corresponding to 50 minutes of interaction. Sim2Real and RLPD additionally need task rewards and objects in simulation, so the methods require different engineering inputs.

Across Board, Board-Obstacle, Table-Tray and Table-Bag, SLAC succeeds in 9, 8, 9 and 7 of ten final trials. Training incurs 1, 4, 0 and 0 safety violations, versus SERL’s 8, 22, 6 and 9 and RLPD’s 34, 37, 26 and 33. SERL succeeds in none of the final trials; the two simulation-based baselines achieve some successes. These small evaluation samples and reported counts do not establish a universal safety guarantee.

Complete original Figure 5.
Figure 5: Motion sequence of the “Trash to Bag” task (left to right). The robot approaches the table and sweeps the trash into the bag that it is holding.
Complete original Figure 6.
Figure 6: Motion sequence of the “Board with Obstacle” task (left to right). The robot wipes the mark on the whiteboard while positioning itself to avoid colliding with the obstacle.

5.2 SLAC for Multi-Robot Systems

A centralized controller learns to coordinate ten virtual drones with ten matching stations. SLAC learns both simultaneous binary-command matching and ordered sequential matching, while the SERL baseline remains poor in the shown curves. Figure 7 averages three random seeds and shades the standard error.

This broadens the evidence beyond Tiago’s control structure, but the drones are simulated. It does not demonstrate transfer to a physical drone team or unrestricted embodiment independence.

Complete original Figure 7.
Figure 7: SLAC on the Multi-Robot Domain. SLAC is a general framework that can be applied to different domains. Results are averaged over 3 random seeds, with the shade showing the standard error. SLAC significantly outperforms the SERL baseline.

5.3 Ablations

The ablations separate downstream task-learning choices from the pretraining value-function architecture. Their curves identify useful components within the tested system rather than prove each component necessary for every application.

5.3.1 Ablations of Downstream Learning

Removing disentanglement also removes the usable reward–action factorization, so that ablation changes more than one isolated mechanism. Replacing FLA-SAC with on-policy PPO or making high-level decisions at the native 10 Hz rate also reduces sample efficiency on the shown tasks. The results support structured skills, off-policy reuse and temporal abstraction as a combined design.

Complete original Figure 8.
Figure 8: Ablation analyses for SLAC. The first three plots evaluate the contribution of key techniques in SLAC to downstream task learning, showing that removing any component substantially reduces sample efficiency. The last two plots evaluate the effect of Q-decomposition in latent action learning by measuring prediction accuracy (↑\uparrow) of latent action dimension ziz^{i} from state factor sis^{i}, averaged across 3 independent runs with different random seeds. Higher prediction accuracy indicates that Q-decomposition encourages the policy to control state factors in more distinguishable ways, enabling more effective downstream learning.

5.3.2 Ablations of Action Space Learning

Skill-discriminator accuracy measures how well an intended state factor reveals its latent label. Across three random seeds, decomposed critics achieve higher prediction accuracy than a monolithic critic in both virtual multi-robot and Tiago pretraining.

This indicates more distinguishable state effects under the diagnostic, not direct proof of safety or success on every downstream task. The experiment compares these architectures rather than establishing that no alternative can learn useful skills.

6 Related Work

The paper places SLAC between full-policy simulation transfer and direct physical RL, then relates it to whole-body planning, control and imitation learning.

6.1 Sim-to-Real Reinforcement Learning

Simulation-trained task policies can use cheap parallel rollouts but depend on suitable assets and dynamics. SLAC transfers a reusable action decoder and learns task selection in reality. This relaxes the need to simulate the complete task, while retaining dependence on shared control interfaces and relevant affordances.

6.2 Reinforcement Learning in the Real World

Direct real-world methods such as SERL motivate strong data reuse. High-dimensional mobile manipulation adds coordinated motion and exploration risks; earlier systems often use demonstrations or explicit behavior priors. SLAC avoids demonstrations and hand-authored motion skills, while still using designed state factors, observations and reward terms.

6.3 Whole-body Mobile Manipulation

Whole-body planners and prioritized controllers depend on models, localization and choices about competing objectives. Demonstration learning avoids some online exploration but requires suitable coordinated teleoperation data. SLAC instead trains closed-loop policies through interaction; its results do not imply that planning or demonstrations are universally inferior.

7 Conclusion

The central result is that coarse simulated affordances can support real-world RL when transferred as structured actions. A frozen decoder limits which fine-grained behaviors the task policy can express. Smaller skill vocabularies and longer execution intervals improve learning efficiency at the cost of control resolution; the best choices are task-dependent.

Decoder finetuning is proposed for finer tasks. Automated resets and reward generation remain future work, so autonomous policy updates should not be read as eliminating every human setup or reset requirement. Reported safety penalties reduce measured violations but do not eliminate them.

Support includes NSF, ONR, ARO, DARPA, Lockheed Martin and UT Austin’s Good Systems program; Jiaheng Hu also reports a Two Sigma fellowship. Peter Stone discloses compensated service as Sony AI Chief Scientist, reviewed under UT Austin’s research-objectivity policy.

Appendix A Appendix

The appendix specifies the reusable action space, continuous-action extension, safety reward, training settings and all task environments.

A.1 Policy and Training Videos

The project website provides real-world training and policy videos. The edition retains the paper’s complete still-image motion sequences in Figures 5 and 6.

A.2 Properties of the Learned Latent Action Space

The skill repertoire is task-agnostic within its training scene, but environment-aware rather than universal. The table decoder is reused for tray and bag tasks. The board repertoire tolerates the tested addition of marks and an obstacle, which were absent from its skill-training scene; broader geometric changes are not evaluated here.

A.3 Discrete vs Continuous Latent Actions

Experiments use categorical latents as a compact vocabulary of distinguishable behaviors. A proposed continuous version would give each factor a k-dimensional continuous prior and replace categorical discriminators with density estimators such as diagonal-covariance Gaussians. The mutual-information objective remains defined, but this appendix does not supply a continuous-versus-discrete experiment.

A.4 Universal Safety Reward

The universal reward penalizes squared action magnitude, squared change from the previous action, collision indicators and excessive-force indicators. Coefficients are respectively 0.01, 0.1, 0.2 and 0.05. They specify a soft penalty; physical force thresholds and a formal safety guarantee are not provided by this equation.

Equation 13
r𝑠𝑎𝑓𝑒=−λ1​‖a‖2−λ2​‖a−aprev‖2−λ3⋅𝕀collision−λ4⋅𝕀force\begin{split}r_{\mathit{safe}}&=-\lambda_{1}\|a\|^{2}-\lambda_{2}\|a-a_{\text{prev}}\|^{2}\\ &\quad-\lambda_{3}\cdot\mathbb{I}_{\text{collision}}-\lambda_{4}\cdot\mathbb{I}_{\text{force}}\end{split} (13)

A.5 Hyperparameter

Each latent action executes for 50 low-level steps—five seconds at the reported 10 Hz. Decoder pretraining uses batch size 1,024, two 1,024-unit layers, 16 environments and learning rate 10⁻⁴. The table labels 4⁵ latent combinations as the latent action dimension; it is not a 1,024-coordinate vector.

Downstream learning uses batch size 64, two 256-unit layers, learning rate 4×10⁻⁴, UTD ratio 10, 60 warm-start samples and Gumbel temperature 1.0. Both complete hyperparameter tables are retained. Their single listed latent configuration does not separately enumerate every multi-robot setting.

Table 2: Hyperparameters of Latent Action Decoder Learning.
Name Value
SAC optimizer Adam
activation functions ReLU
learning rate 1×10−41\times 10^{-4}
batch size 1024
critic target τ\tau 0.01
MLP size [1024, 1024]
n updates 2
# of environments 16
entropy coefficient α\alpha 0.0
log std bounds [-10, 2]
warmup samples 24000
latent action dimension 454^{5}
Table 3: Hyperparameters of Downstream Learning.
Name Value
FLA-SAC optimizer Adam
activation functions ReLu
learning rate 4×10−44\times 10^{-4}
batch size 64
critic target τ\tau 0.05
MLP size [256, 256]
utd ratio 10
# of environments 1
entropy coefficient α\alpha 0.1
log std bounds [-10, 2]
warmup samples 60
gumbel temperature 1.0

A.6 Mobile Manipulation Environment Description

Tiago’s 17 native controls comprise three base velocities, two head joints and two six-dimensional end-effector pose increments. The task policy receives a segmented 320×240 RGB-D observation reduced to 50 points plus seven proprioceptive values. Reward is delivered at the end of each high-level step.

A.6.1 Board Environment

The board decoder observes proprioception, previous actions, board-relative distance/orientation and contact history. RANSAC estimates board geometry in reality. Skill factors cover contact history, base position, camera view and contact force.

Clean Whiteboard rewards looking at, approaching and removing the target mark while penalizing collisions or excessive force. Wipe Board over Obstacles adds an intervening obstacle with the same reward formulation. The stated success criterion requires all four conditions, not just visible wiping.

A.6.2 Table Environment

The table decoder uses proprioception, previous actions and RANSAC-estimated table-relative geometry. Its factors cover both end-effectors, base location and camera view; table collision is penalized.

Push to Tray rewards looking at and approaching the garbage, pushing it into the tray and avoiding dangerous contact. Sweep to Bag starts with a bag already in the left gripper; it rewards pushing garbage off the table while keeping the bag nearby. This setup does not learn bag pickup or specify an explicit garbage-in-bag reward term.

A.7 Multi-Robot Environment Description

The simulated multi-robot system has ten drones and ten uniquely matching stations, a 70-dimensional observation and 50-dimensional action space. Five controls per robot govern acceleration and interaction.

Parallel Match requires simultaneous compliance with binary interaction indicators. Sequential Match requires visiting matching stations in a prescribed episode-level order. Incorrect interactions are penalized; both tasks use a centralized controller.

Concept demonstration

Open image ↗