Literature
Display
← All papers

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Charles Xu · Jost Tobias Springenberg · Michael Equi · Ali Amin · Adnan Esmail · Sergey Levine · Liyiming Ke

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers all thirteen pages of arXiv v2, dated 30 April 2026, including the full experiment and baseline appendices. Michael Equi’s name follows the printed title page. All nine figures and the complete algorithm are preserved, including Figure 1, which is absent from the HTML source.

Complete original figure with every panel, label and legend.
Fig. 1: Our method introduces an “RL token” into the VLA by training an encoder and decoder to produce a compact and meaningful representation from a VLA’s internal features. The extracted representation is then used to train lightweight actor-critic networks with sample-efficient online RL, enabling very precise tasks to be fine-tuned with a few hours or even minutes of robot experience.

At a glance

RL Token (RLT) adapts a pretrained vision-language-action model to expose a compact state representation, then freezes it. A small online actor-critic uses this token, proprioception and a sampled VLA action chunk to refine behavior near a useful prior.

On screw installation, zip tie fastening, Ethernet insertion and charger insertion, it improves precision-critical phases under sparse human supervision. Reported improvements concern selected phases and task-specific adaptation; the short online interaction budgets exclude demonstrations, resets and other overhead.

I Introduction

Generalist imitation policies can grasp and transport effectively yet pause, retry or fail near precise contact. Online RL can optimize those bottlenecks beyond demonstrated behavior, but physical experience is expensive and updating a large VLA online is costly.

RLT separates frozen perception/behavioral knowledge from a small adaptive controller. First train a compressed readout on task demonstrations, optionally fine-tuning the VLA; then learn chunk-level actions online while conditioning on and regularizing toward its reference proposal.

The reported screw critical-phase success improvement is 20%→65%. Selected Ethernet insertions become faster than expert teleoperation. These are specialized task results, not evidence that broad VLA generalization is preserved after unrestricted online training.

II Related Work

OpenVLA, π₀ and related VLAs combine pretrained visual-language knowledge with expressive action generation. Action chunking reduces control decision frequency; diffusion and autoregressive models can represent multiple valid action modes.

HIL-SERL and SERL make small off-policy policies practical on robots using demonstrations and interventions. RECAP instead updates a full large policy through collected experience. RLT retains a frozen VLA during online learning but explicitly trains a compact readout first.

Probe-Learn-Distill learns single-step residual actions; DSRL steers a frozen policy through latent noise. RLT directly predicts full action chunks conditioned on sampled VLA proposals. Its squared-distance anchor is a chosen local refinement objective, not a hard bound on every action.

III Preliminaries

The π₀.₆ base model predicts H=50 actions, one second at 50 Hz, from images, language and robot state. RLT instead acts over C=10 steps (0.2 seconds), making a 140-dimensional chunk from fourteen action dimensions per step.

Use a continuous-control MDP and sparse terminal human labels: +1 for success, zero for failure. Chunk-level TD targets accumulate rewards over C steps and bootstrap with γᶜ. Replay can mix VLA, learner and human-intervention trajectories because learning is off-policy.

The exact return, action-value and chunk definitions are retained below. Chunk transitions shorten temporal credit assignment, but intermediate overlapping windows are correlated training samples, not independent new robot experience.

Equations
𝐚~t:t+H−1=(𝐚~t,…,𝐚~t+H−1)∈ℝH×d\tilde{\mathbf{a}}_{t:t+H-1}=(\tilde{\mathbf{a}}_{t},\ldots,\tilde{\mathbf{a}}_{t+H-1})\in\mathbb{R}^{H\times d}
J⁡(π)=𝔼τ∼ρπ​[∑t=0Tγt​rt]J(\pi)=\mathbb{E}_{\tau\sim\rho_{\pi}}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right]
Qπ(𝐬t,𝐚t)=𝔼τ∼ρπ[∑t′=tTγt′−trt′|𝐬t,𝐚t].Q^{\pi}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbb{E}_{\tau\sim\rho_{\pi}}\left[\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}\;\middle|\;\mathbf{s}_{t},\mathbf{a}_{t}\right].
𝐚t:t+C−1=(𝐚t,…,𝐚t+C−1)∈ℝC×d,\mathbf{a}_{t:t+C-1}=(\mathbf{a}_{t},\dots,\mathbf{a}_{t+C-1})\in\mathbb{R}^{C\times d},
Qπ(𝐬t,𝐚t:t+C−1)=∑t′=tt+C−1γt′−trt′+γC𝔼𝐚′∼π|𝐬t+C[Qπ(𝐬t+C,𝐚′)].Q^{\pi}(\mathbf{s}_{t},\mathbf{a}_{t:t+C-1})=\sum_{t^{\prime}=t}^{t+C-1}\gamma^{t^{\prime}-t}r_{t^{\prime}}+\gamma^{C}\mathbb{E}_{\mathbf{a}^{\prime}\sim\pi|\mathbf{s}_{t+C}}\left[Q^{\pi}(\mathbf{s}_{t+C},\mathbf{a}^{\prime})\right].

IV Reinforcement Learning from RL Token

The two-stage recipe first builds an RL interface from task demonstrations, then trains only the lightweight actor and critic online. During online interaction, both the VLA and readout encoder remain frozen; the VLA supplies perceptual features and sampled reference actions.

IV-A Adapting the VLA to expose an RL interface

Take final-layer VLA embeddings, append a learned <rl> readout embedding, and pass the sequence through a small encoder transformer. Its final-position output is the RL token (equation 1). Figure 2 retains the exact representation diagram and dimensions.

Train a decoder to autoregressively reconstruct the original embeddings from the token and preceding stop-gradient target embeddings (equation 2). This bottleneck objective compresses existing features without backpropagating reconstruction loss into the VLA. Optional supervised VLA fine-tuning is a separate loss.

After adaptation, freeze both models. Each experiment uses one fixed language instruction and drops language embeddings from readout training; flexible instruction-conditioned transfer of the compressed token is not tested. Reconstruction sufficiency does not itself prove that every control-relevant detail survives compression.

Complete original figure with every panel, label and legend.
Fig. 2: Details on RL token extraction. RLT adds an encoder-decoder transformer to a pretrained VLA. It produces a compressed embedding of the VLA representation (the RL token). This representation then enables data and parameter efficient fine-tuning during online RL.
Equations 1–2
𝐳rl=gϕ([𝐳1:M,𝐞rl])M+1.\mathbf{z}_{\text{rl}}\;=\;g_{\phi}\!\bigl([\mathbf{z}_{1:M},\;\mathbf{e}_{\texttt{rl}}]\bigr)_{M+1}\,. (1)
ℒro=𝔼𝒟[∑i=1M∥hϕ(dϕ([𝐳rl,𝐳¯1:i−1]))i−𝐳¯i∥2].\mathcal{L}_{\text{ro}}=\mathbb{E}_{\mathcal{D}}\!\Bigl[\,\sum_{i=1}^{M}\bigl\lVert h_{\phi}\bigl(d_{\phi}([\mathbf{z}_{\text{rl}},\,\bar{\mathbf{z}}_{1:i-1}])\bigr)_{\!i}-\bar{\mathbf{z}}_{i}\bigr\rVert^{2}\,\Bigr]. (2)
𝐳=f⁡(s,ℓ,θvla)\mathbf{z}=f(s,\ell;\theta_{\text{vla}})
𝐳1:M={𝐳1,…,𝐳M}\mathbf{z}_{1:M}=\{\mathbf{z}_{1},\dots,\mathbf{z}_{M}\}
𝐞rl=𝐞ϕ​(<rl>)\mathbf{e}_{\texttt{rl}}=\mathbf{e}_{\phi}(\texttt{<rl>})
𝐳¯i=sg⁡(𝐳i)\bar{\mathbf{z}}_{i}=\mathrm{sg}(\mathbf{z}_{i})

IV-B Online RL to refine VLA action chunks

The critic sees the RL token, proprioception and executed C-step action chunk. Equation 3 fits a squared TD error with discounted chunk rewards and a target critic. Implementation uses two critics and their minimum target, following TD3.

The actor additionally sees a sampled VLA reference chunk and outputs a Gaussian with fixed small standard deviation (equation 4). Conditioning exposes which action mode the VLA proposed, which a single Gaussian conditioned only on state would struggle to reproduce.

Equation 5 trades predicted value against β times squared distance to the reference action. This is action-space regularization, not residual output or a direct KL divergence between the complete VLA and actor distributions. The actor can move beyond the proposal when the learned value favors it.

Mask the reference input to zeros for 50% of training examples, while always providing it at inference, to discourage copying before the critic is informative. Algorithm 1 retains every stage, case, intervention override and update; the paper does not specify numerical β, fixed noise scale or every optimizer/target-network hyperparameter.

Equations 3–5
ℒQ\displaystyle\mathcal{L}_{Q} =𝔼(𝐱,𝐚1:C,𝐱′)∼ℬ[(Q^−Qψ(𝐱,𝐚1:C))2],\displaystyle=\mathbb{E}_{(\mathbf{x},\mathbf{a}_{1:C},\mathbf{x}^{\prime})\sim\mathcal{B}}\Big[\big(\hat{Q}-Q_{\psi}(\mathbf{x},\mathbf{a}_{1:C})\big)^{2}\Big], (3)
Q^\displaystyle\hat{Q} =∑t′=1Cγt′−1​rt′+γC​𝔼𝐚′∼πθ​[Qψ′​(𝐱′,𝐚′)].\displaystyle=\sum_{t^{\prime}=1}^{C}\gamma^{t^{\prime}-1}r_{t^{\prime}}+\gamma^{C}\mathbb{E}_{\mathbf{a}^{\prime}\sim\pi_{\theta}}\Big[Q_{\psi^{\prime}}(\mathbf{x}^{\prime},\mathbf{a}^{\prime})\Big].
πθ(𝐚1:C∣𝐱,𝐚~1:C)=𝒩(μθ(𝐱,𝐚~1:C),σ2𝐈),\pi_{\theta}\bigl(\mathbf{a}_{1:C}\mid\mathbf{x},\tilde{\mathbf{a}}_{1:C}\bigr)=\mathcal{N}\Bigl(\mu_{\theta}\bigl(\mathbf{x},\tilde{\mathbf{a}}_{1:C}\bigr),\sigma^{2}\mathbf{I}\Bigr), (4)
Exact original PDF equation 5, including every term and delimiter.
Selectable original MathML
ℒπ​(θ)\displaystyle\mathcal{L}_{\pi}(\theta) =𝔼𝐬∼ℬ𝐚1:C∼πθ[−Qψ(𝐱,𝐚1:C)+β‖𝐚1:C−𝐚~1:C‖22],\displaystyle=\mathbb{E}_{\begin{subarray}{c}\mathbf{s}\sim\mathcal{B}\\ \mathbf{a}_{1:C}\sim\pi_{\theta}\end{subarray}}\left[-\,Q_{\psi}(\mathbf{x},\mathbf{a}_{1:C})+\beta\left\|\mathbf{a}_{1:C}-\tilde{\mathbf{a}}_{1:C}\right\|_{2}^{2}\right], (5)
𝐚~1:C∼πvla(⋅∣𝐬,ℓ),\displaystyle\tilde{\mathbf{a}}_{1:C}\sim\pi_{\mathrm{vla}}(\cdot\mid\mathbf{s},\ell),
Algorithm 1 RLT
Complete original RLT algorithm, every input, nineteen steps, cases and nested updates.
Selectable pseudocode
0:  Frozen VLA backbone fθvlaf_{\theta_{\text{vla}}} and VLA action distribution πvla\pi_{\text{vla}}; demo data 𝒟\mathcal{D}, chunk length CC, replay buffer ℬ\mathcal{B}, warmup steps NwarmN_{\text{warm}}, ratio GG, VLA fine-tuning weight α\alpha, policy constraint β\beta.
1:  Train RL token and (optionally) fine-tune the VLA
2:  Train ϕ\phi using 𝐳i=fi​(𝐬,ℓ,θvla)\mathbf{z}_{i}=f_{i}(\mathbf{s},\ell,\theta_{\text{vla}}), 𝐳rl=gϕ([𝐳1:M,𝐞rl])M+1\mathbf{z}_{\text{rl}}=g_{\phi}([\mathbf{z}_{1:M},\mathbf{e}_{\text{rl}}])_{M+1}, and θvla\theta_{\text{vla}} (only if α>0\alpha>0).
Equation
ℒro(ϕ)=𝔼𝒟[∑i=1M∥hϕ(dϕ([𝐳rl,𝐳¯1:i−1]))i−𝐳¯i∥2].\mathcal{L}_{\text{ro}}(\phi)=\mathbb{E}_{\mathcal{D}}\!\Bigl[\,\sum_{i=1}^{M}\bigl\lVert h_{\phi}\bigl(d_{\phi}([\mathbf{z}_{\text{rl}},\,\bar{\mathbf{z}}_{1:i-1}])\bigr)_{\!i}-\bar{\mathbf{z}}_{i}\bigr\rVert^{2}\,\Bigr].
3:  
Equation
ϕ,θvla=arg⁡minϕ,θvla​ℒro​(ϕ)+α​ℒvla​(θvla)\phi,\theta_{\text{vla}}=\arg\min_{\phi,\theta_{\text{vla}}}\mathcal{L}_{\text{ro}}(\phi)+\alpha\mathcal{L}_{\text{vla}}(\theta_{\text{vla}})
4:  Train RL actor and critic
5:  Initialize critic QψQ_{\psi} and RL Policy πθ\pi_{\theta}.
6:  for environment steps t=0,C,2​C​…t=0,C,2C\dots do
7:   Sample VLA reference chunk 𝐚~t:t+C−1∼πvla(𝐬t)\tilde{\mathbf{a}}_{t:t+C-1}\sim\pi_{\text{vla}}(\mathbf{s}_{t}).
8:   Form RL state 𝐱t=(𝐳rl​(𝐬t),𝐬tp)\mathbf{x}_{t}=(\mathbf{z}_{\text{rl}}(\mathbf{s}_{t}),\mathbf{s}^{p}_{t}).
9:   𝐚t:t+C−1←{𝐚humanif intervention𝐚~t:t+C−1if ​t<Nwarm∼πθ(⋅∣𝐱t,𝐚~)otherwise\mathbf{a}_{t:t+C-1}\leftarrow\begin{cases}\mathbf{a}^{\text{human}}&\text{if intervention}\\ \tilde{\mathbf{a}}_{t:t+C-1}&\text{if }t<N_{\text{warm}}\\ \sim\pi_{\theta}(\,\cdot\mid\mathbf{x}_{t},\tilde{\mathbf{a}}\,)&\text{otherwise}\end{cases}
10:   Execute 𝐚t:t+C−1\mathbf{a}_{t:t+C-1} and observe rtr_{t}, 𝐬t+1\mathbf{s}_{t+1}, 𝐬t+1p\mathbf{s}^{p}_{t+1}
11:   𝐚~t:t+C−1←𝐚humanif intervention\tilde{\mathbf{a}}_{t:t+C-1}\leftarrow\mathbf{a}^{\text{human}}\quad\text{if intervention}
12:   Store transition in ℬ\mathcal{B}: ⟨𝐱t,𝐚t:t+C−1,𝐚~,rt,𝐱t+1⟩\langle\mathbf{x}_{t},\mathbf{a}_{t:t+C-1},\tilde{\mathbf{a}},r_{t},\mathbf{x}_{t+1}\rangle
13:   for g=1,…,Gg=1,\dots,G do
14:    Sample batch of data b∼ℬ\mathrm{b}\sim\mathcal{B}.
15:    Compute target Q values
Equation
Q^=∑t′=1Cγt′−1​rt′+γC​𝔼𝐚′∼πθ​[Qψ′​(𝐱′,𝐚′)]\hat{Q}=\sum_{t^{\prime}=1}^{C}\gamma^{t^{\prime}-1}r_{t^{\prime}}+\gamma^{C}\mathbb{E}_{\mathbf{a}^{\prime}\sim\pi_{\theta}}\big[Q_{\psi^{\prime}}(\mathbf{x}^{\prime},\mathbf{a}^{\prime})\big]
16:    Train Critic with TD backup (Eq. (3))
Equation
ℒQ​(ψ)=𝔼b​[(Q^−Qψ​(𝐱,𝐚))2]\mathcal{L}_{Q}(\psi)=\mathbb{E}_{\mathrm{b}}\Big[\big(\hat{Q}-Q_{\psi}(\mathbf{x},\mathbf{a})\big)^{2}\Big]
17:    Train Policy 𝐚∼πθ(⋅∣𝐬,𝐚~)\mathbf{a}\sim\pi_{\theta}(\cdot\mid\mathbf{s},\tilde{\mathbf{a}}) (Eq. (5))
Equation
ℒπ​(θ)=𝔼b​[−Qψ​(𝐱,𝐚)+β​‖𝐚−𝐚~‖22]\mathcal{L}_{\pi}(\theta)=\mathbb{E}_{\mathrm{b}}\Bigl[-Q_{\psi}(\mathbf{x},\mathbf{a})+\beta\|\mathbf{a}-\tilde{\mathbf{a}}\|_{2}^{2}\Bigr]
18:   end for
19:  end for
Equations
Qψ(𝐱,𝐚1:C)∈ℝQ_{\psi}(\mathbf{x},\mathbf{a}_{1:C})\in\mathbb{R}
𝐱=(𝐳rl,𝐬p)\mathbf{x}=(\mathbf{z}_{\text{rl}},\mathbf{s}^{\text{p}})

V The Complete System

Collect initial replay with the base VLA, then alternate asynchronous robot rollouts and replay updates. At each chunk boundary, compute the token and VLA proposal; execute the actor chunk unless a human intervenes. Store the executed human action as the reference during interventions so the actor can learn those corrections.

Save overlapping C-step transitions every two control steps, yielding about 25 samples per second. Use update-to-data ratio 5 and two critic updates per actor update. These ratios improve reuse; they do not count resets or model computation as robot interaction.

Train the difficult phase while the base VLA handles earlier task stages. A human selects the handoff and labels success/failure. The paper proposes a final short supervised fine-tuning pass to predict the handoff from these labels, but does not isolate its switching accuracy or provide a fully autonomous training pipeline.

Complete original figure with every panel, label and legend.
Fig. 3: The tasks in our experiments: each task contains a critical phase that requires high precision: (top) using a screwdriver to install a screw, (middle) fastening a zip tie, (bottom) plugging in an Ethernet cable and plugging in a charger.

VI Real-World Experiments

Experiments ask whether the method improves the base VLA, beats alternative experience-based methods, benefits from each component and discovers faster behavior. All four tasks have critical-phase tests; full-task tests cover screw installation and zip tie fastening.

VI-A Tasks and Setup

Screw installation aligns an electric screwdriver to an M3 screw, with grasp-to-tip distance amplifying angular errors. Zip tie fastening coordinates two arms around a deformable tail and small slot. Ethernet insertion requires connector alignment and decisive contact; charger insertion has obscured prongs and is described as centimeter-level alignment, unlike the tighter tasks.

Complete tasks last roughly 30–120 seconds; difficult phases typically last 5–20 seconds. Controlled evaluation begins from a partially completed task with slight randomization and uses fifty episodes per agent/task. Full-task evaluation starts from home and inherits state variation from earlier base-policy actions; its trial count is not separately stated.

Use two wrist RGB cameras and one base camera, plus joint position for screw or end-effector pose for the other tasks, with additional position/velocity information described in the appendix. Robot control is 50 Hz, while the ten-step RL chunk executes for 0.2 seconds before replanning.

VI-B Baselines and Ablations

Collect 1–10 hours of teleoperation demonstrations per task and fine-tune the VLA while training the readout. This adapted model is the base policy in comparisons. Then collect 400–1,000 online episodes, about fifteen minutes to five hours of actual robot data after excluding resets and other overhead.

Measure binary human-judged success and successful completions per ten-minute interval. The four alternative-method comparison is on Ethernet: HIL-SERL, PLD, DSRL and DAgger. Reported matching of data budget does not remove differences in pretraining, action interface, control frequency or exploration restrictions.

Ablate the RL token, action chunking, the β anchor and reference input. The single-step C=1 variant also substitutes ResNet-10 for the RL token because VLA queries cannot run at 50 Hz; its loss therefore combines horizon and representation changes. The no-token ablation separately tests representation with chunking retained.

Complete original figure with every panel, label and legend.
Fig. 4: RLT increases throughput significantly over the base VLA policy, improving both the speed and consistency of the critical phase of each task. The improvement is especially pronounced for the harder tasks where the VLA policy is prone to making mistakes.
Complete original figure with every panel, label and legend.
Fig. 5: RLT can boost success rates across multiple tasks. Where the VLA is already competent (e.g., the Ethernet task) it maintains success rate and increases throughput. For tasks that are challenging for the base VLA policy (screwdriver and zip tie) RLT leads to a significant improvement in success.

VI-C Experimental Results

Figures 4–5 retain every throughput/success bar, error bar and full-task grouping. Critical-phase screw success rises from the reported 20% to 65%; near-ceiling Ethernet reliability is retained while execution speeds up. Full-task success is lower, with reported absolute gains of about forty percentage points for screw and sixty for zip tie.

On Ethernet, HIL-SERL and PLD learn poorly under this long sparse-reward, 50-Hz setup. DSRL and DAgger achieve high reliability but lower throughput. The paper attributes differences partly to chunked credit assignment and less constrained action refinement; it does not independently vary every baseline factor.

Removing the RL token roughly halves throughput. Removing the anchor is particularly harmful. Removing reference pass-through slows learning and causes more failures, but eventually reaches the same final performance on this simpler task. This qualifies the Figure 7 caption’s broad claim that the full system performs best at the end.

The five-minute point in Figure 7 is interaction data, about forty minutes of total experiment time. The paper does not report a distribution across independent training seeds or define the plotted error bars clearly; these curves do not establish universal minutes-to-success.

Figure 9 gives Ethernet critical-phase median lengths 66 steps for RLT, 146 for teleoperation and 228 for the base policy: 1.32, 2.92 and 4.56 seconds at 50 Hz. This median comparison is separate from the text’s roughly 2× mean-step reduction. RLT often inserts directly or uses pressure and small wiggles after initial failure; selected behavior/histogram evidence does not quantify a universal strategy advantage.

Complete original figure with every panel, label and legend.
Fig. 6: Comparison to other RL algorithms. We compare RLT against several baselines from the recent RL literature. Methods that consider only single actions, rather than action chunks, (HIL-SERL, PLD) perform poorly. DSRL leads to high success but significantly lags behind in throughput.
Complete original figure with every panel, label and legend.
Fig. 7: Throughput at different points in training for Ethernet task. The ablation study shows that each part of our method is important for good performance, and the full system learns fastest and performs best at the end. Notably, RLT outperforms the alternative policy after consuming only 5 minutes of data on the critical part of the task (total experiment time ∼\sim 40mins). Dropping the reference action from the actor input (“w/o Pass-Through”) still allows reaching the best final performance, but at the cost of slower learning and significantly more failures over the course of training.
Complete original figure with every panel, label and legend.
Fig. 8: Success rate evaluation during training for the Ethernet task. RLT quickly matches the success rate of the VLA policy on the Ethernet insertion task, while boosting throughput. Not using the reference-action pass-through or not using the RL token leads to slower learning.
Complete original figure with every panel, label and legend.
Fig. 9: Speed on the Ethernet task. RLT significantly improves the speed of the Ethernet task. The final policy is faster even than the demonstrations produced by expert teleoperators, and significantly faster than the base VLA model. Half of the RL episodes during the critical insertion phase (yellow) are faster than all of the teleoperated demonstrations (green).

VII Conclusion

A frozen VLA can support a small controller that improves narrow contact-rich phases efficiently. Task-specific demonstration adaptation, compressed observations, chunked updates and action anchoring jointly make the reported results possible.

The method still depends on humans for reward labels, corrective interventions and training-time handoffs. Reset/operational cost, broad instruction transfer, generalization beyond four tasks and fully autonomous improvement remain unresolved. Proposed reward/progress automation is future work, not an evaluated replacement for supervision.

Acknowledgments

The authors acknowledge Physical Intelligence colleagues and operators for hardware, grippers, data collection, robot operations, infrastructure, annotation, figures, proofreading and media production.

-A Contributions

Charles Xu and Liyiming Ke initiated the project and designed system implementation, tasks and experiments. Xu built online-RL infrastructure; Jost Tobias Springenberg designed/trained the token; Michael Equi built the intervention interface; Ali Amin and Adnan Esmail built gripper/robot hardware. Sergey Levine and Ke advised; Xu, Springenberg, Equi, Levine and Ke contributed writing and presentation.

-B Additional experiment details

Train the token and fine-tune the base VLA for 2,000–10,000 gradient steps on single-task demonstrations, then freeze both. Initialize small actor/critic networks from scratch: two hidden layers of width 256 for zip tie/Ethernet/charger, or three layers of width 512 for screw.

Use twin critics with minimum target, token plus proprioceptive position/velocity, Gaussian action chunks with C=10 and fixed small noise, and 50% reference masking only during training. Save windows two steps apart and obtain sparse +1 success labels from the operator.

For screw and zip tie, first learn from controlled critical-phase resets, then collect full-task experience with the base VLA handling early stages. Reported harder-task policies use about five hours of interaction data. Token architecture details beyond Figure 2, warmup duration and several numeric RL hyperparameters are not fully specified.

Equation
𝐚t:t+C−1∈ℝC×d\mathbf{a}_{t:t+C-1}\in\mathbb{R}^{C\times d}

-C Additional experimental details for the baselines

All methods execute delta actions at 50 Hz. PLD first pretrains its critic on fifty base-policy rollouts using Cal-QL. DSRL predicts a 1×32 latent vector repeated fifty times to match the VLA noise interface.

HIL-SERL starts from twenty demonstration episodes and uses interventions, but runs at 50 Hz instead of the original system’s 10 Hz and lacks an action-space bounding box. The authors identify those changes as causes of difficulty, so the comparison is specific to this setup.

DAgger fine-tunes the VLA using demonstrations and the same intervention data collected during RLT. Its lower speed is consistent with imitation of those trajectories; it is not an independently optimized online-RL baseline.

Concept demonstration

Open image ↗