Overview
This edition covers all thirteen pages of arXiv v2, dated 30 April 2026, including the full experiment and baseline appendices. Michael Equi’s name follows the printed title page. All nine figures and the complete algorithm are preserved, including Figure 1, which is absent from the HTML source.
At a glance
RL Token (RLT) adapts a pretrained vision-language-action model to expose a compact state representation, then freezes it. A small online actor-critic uses this token, proprioception and a sampled VLA action chunk to refine behavior near a useful prior.
On screw installation, zip tie fastening, Ethernet insertion and charger insertion, it improves precision-critical phases under sparse human supervision. Reported improvements concern selected phases and task-specific adaptation; the short online interaction budgets exclude demonstrations, resets and other overhead.
I Introduction
Generalist imitation policies can grasp and transport effectively yet pause, retry or fail near precise contact. Online RL can optimize those bottlenecks beyond demonstrated behavior, but physical experience is expensive and updating a large VLA online is costly.
RLT separates frozen perception/behavioral knowledge from a small adaptive controller. First train a compressed readout on task demonstrations, optionally fine-tuning the VLA; then learn chunk-level actions online while conditioning on and regularizing toward its reference proposal.
The reported screw critical-phase success improvement is 20%→65%. Selected Ethernet insertions become faster than expert teleoperation. These are specialized task results, not evidence that broad VLA generalization is preserved after unrestricted online training.
II Related Work
OpenVLA, π₀ and related VLAs combine pretrained visual-language knowledge with expressive action generation. Action chunking reduces control decision frequency; diffusion and autoregressive models can represent multiple valid action modes.
HIL-SERL and SERL make small off-policy policies practical on robots using demonstrations and interventions. RECAP instead updates a full large policy through collected experience. RLT retains a frozen VLA during online learning but explicitly trains a compact readout first.
Probe-Learn-Distill learns single-step residual actions; DSRL steers a frozen policy through latent noise. RLT directly predicts full action chunks conditioned on sampled VLA proposals. Its squared-distance anchor is a chosen local refinement objective, not a hard bound on every action.
III Preliminaries
The π₀.₆ base model predicts H=50 actions, one second at 50 Hz, from images, language and robot state. RLT instead acts over C=10 steps (0.2 seconds), making a 140-dimensional chunk from fourteen action dimensions per step.
Use a continuous-control MDP and sparse terminal human labels: +1 for success, zero for failure. Chunk-level TD targets accumulate rewards over C steps and bootstrap with γᶜ. Replay can mix VLA, learner and human-intervention trajectories because learning is off-policy.
The exact return, action-value and chunk definitions are retained below. Chunk transitions shorten temporal credit assignment, but intermediate overlapping windows are correlated training samples, not independent new robot experience.
Equations
IV Reinforcement Learning from RL Token
The two-stage recipe first builds an RL interface from task demonstrations, then trains only the lightweight actor and critic online. During online interaction, both the VLA and readout encoder remain frozen; the VLA supplies perceptual features and sampled reference actions.
IV-A Adapting the VLA to expose an RL interface
Take final-layer VLA embeddings, append a learned <rl> readout embedding, and pass the sequence through a small encoder transformer. Its final-position output is the RL token (equation 1). Figure 2 retains the exact representation diagram and dimensions.
Train a decoder to autoregressively reconstruct the original embeddings from the token and preceding stop-gradient target embeddings (equation 2). This bottleneck objective compresses existing features without backpropagating reconstruction loss into the VLA. Optional supervised VLA fine-tuning is a separate loss.
After adaptation, freeze both models. Each experiment uses one fixed language instruction and drops language embeddings from readout training; flexible instruction-conditioned transfer of the compressed token is not tested. Reconstruction sufficiency does not itself prove that every control-relevant detail survives compression.
Equations 1–2
| (1) |
| (2) |
IV-B Online RL to refine VLA action chunks
The critic sees the RL token, proprioception and executed C-step action chunk. Equation 3 fits a squared TD error with discounted chunk rewards and a target critic. Implementation uses two critics and their minimum target, following TD3.
The actor additionally sees a sampled VLA reference chunk and outputs a Gaussian with fixed small standard deviation (equation 4). Conditioning exposes which action mode the VLA proposed, which a single Gaussian conditioned only on state would struggle to reproduce.
Equation 5 trades predicted value against β times squared distance to the reference action. This is action-space regularization, not residual output or a direct KL divergence between the complete VLA and actor distributions. The actor can move beyond the proposal when the learned value favors it.
Mask the reference input to zeros for 50% of training examples, while always providing it at inference, to discourage copying before the critic is informative. Algorithm 1 retains every stage, case, intervention override and update; the paper does not specify numerical β, fixed noise scale or every optimizer/target-network hyperparameter.
Equations 3–5
| (3) | ||||
| (4) |
Selectable original MathML
| (5) | ||||
Selectable pseudocode
Equation
Equation
Equation
Equations
V The Complete System
Collect initial replay with the base VLA, then alternate asynchronous robot rollouts and replay updates. At each chunk boundary, compute the token and VLA proposal; execute the actor chunk unless a human intervenes. Store the executed human action as the reference during interventions so the actor can learn those corrections.
Save overlapping C-step transitions every two control steps, yielding about 25 samples per second. Use update-to-data ratio 5 and two critic updates per actor update. These ratios improve reuse; they do not count resets or model computation as robot interaction.
Train the difficult phase while the base VLA handles earlier task stages. A human selects the handoff and labels success/failure. The paper proposes a final short supervised fine-tuning pass to predict the handoff from these labels, but does not isolate its switching accuracy or provide a fully autonomous training pipeline.
VI Real-World Experiments
Experiments ask whether the method improves the base VLA, beats alternative experience-based methods, benefits from each component and discovers faster behavior. All four tasks have critical-phase tests; full-task tests cover screw installation and zip tie fastening.
VI-A Tasks and Setup
Screw installation aligns an electric screwdriver to an M3 screw, with grasp-to-tip distance amplifying angular errors. Zip tie fastening coordinates two arms around a deformable tail and small slot. Ethernet insertion requires connector alignment and decisive contact; charger insertion has obscured prongs and is described as centimeter-level alignment, unlike the tighter tasks.
Complete tasks last roughly 30–120 seconds; difficult phases typically last 5–20 seconds. Controlled evaluation begins from a partially completed task with slight randomization and uses fifty episodes per agent/task. Full-task evaluation starts from home and inherits state variation from earlier base-policy actions; its trial count is not separately stated.
Use two wrist RGB cameras and one base camera, plus joint position for screw or end-effector pose for the other tasks, with additional position/velocity information described in the appendix. Robot control is 50 Hz, while the ten-step RL chunk executes for 0.2 seconds before replanning.
VI-B Baselines and Ablations
Collect 1–10 hours of teleoperation demonstrations per task and fine-tune the VLA while training the readout. This adapted model is the base policy in comparisons. Then collect 400–1,000 online episodes, about fifteen minutes to five hours of actual robot data after excluding resets and other overhead.
Measure binary human-judged success and successful completions per ten-minute interval. The four alternative-method comparison is on Ethernet: HIL-SERL, PLD, DSRL and DAgger. Reported matching of data budget does not remove differences in pretraining, action interface, control frequency or exploration restrictions.
Ablate the RL token, action chunking, the β anchor and reference input. The single-step C=1 variant also substitutes ResNet-10 for the RL token because VLA queries cannot run at 50 Hz; its loss therefore combines horizon and representation changes. The no-token ablation separately tests representation with chunking retained.
VI-C Experimental Results
Figures 4–5 retain every throughput/success bar, error bar and full-task grouping. Critical-phase screw success rises from the reported 20% to 65%; near-ceiling Ethernet reliability is retained while execution speeds up. Full-task success is lower, with reported absolute gains of about forty percentage points for screw and sixty for zip tie.
On Ethernet, HIL-SERL and PLD learn poorly under this long sparse-reward, 50-Hz setup. DSRL and DAgger achieve high reliability but lower throughput. The paper attributes differences partly to chunked credit assignment and less constrained action refinement; it does not independently vary every baseline factor.
Removing the RL token roughly halves throughput. Removing the anchor is particularly harmful. Removing reference pass-through slows learning and causes more failures, but eventually reaches the same final performance on this simpler task. This qualifies the Figure 7 caption’s broad claim that the full system performs best at the end.
The five-minute point in Figure 7 is interaction data, about forty minutes of total experiment time. The paper does not report a distribution across independent training seeds or define the plotted error bars clearly; these curves do not establish universal minutes-to-success.
Figure 9 gives Ethernet critical-phase median lengths 66 steps for RLT, 146 for teleoperation and 228 for the base policy: 1.32, 2.92 and 4.56 seconds at 50 Hz. This median comparison is separate from the text’s roughly 2× mean-step reduction. RLT often inserts directly or uses pressure and small wiggles after initial failure; selected behavior/histogram evidence does not quantify a universal strategy advantage.
VII Conclusion
A frozen VLA can support a small controller that improves narrow contact-rich phases efficiently. Task-specific demonstration adaptation, compressed observations, chunked updates and action anchoring jointly make the reported results possible.
The method still depends on humans for reward labels, corrective interventions and training-time handoffs. Reset/operational cost, broad instruction transfer, generalization beyond four tasks and fully autonomous improvement remain unresolved. Proposed reward/progress automation is future work, not an evaluated replacement for supervision.
Acknowledgments
The authors acknowledge Physical Intelligence colleagues and operators for hardware, grippers, data collection, robot operations, infrastructure, annotation, figures, proofreading and media production.
-A Contributions
Charles Xu and Liyiming Ke initiated the project and designed system implementation, tasks and experiments. Xu built online-RL infrastructure; Jost Tobias Springenberg designed/trained the token; Michael Equi built the intervention interface; Ali Amin and Adnan Esmail built gripper/robot hardware. Sergey Levine and Ke advised; Xu, Springenberg, Equi, Levine and Ke contributed writing and presentation.
-B Additional experiment details
Train the token and fine-tune the base VLA for 2,000–10,000 gradient steps on single-task demonstrations, then freeze both. Initialize small actor/critic networks from scratch: two hidden layers of width 256 for zip tie/Ethernet/charger, or three layers of width 512 for screw.
Use twin critics with minimum target, token plus proprioceptive position/velocity, Gaussian action chunks with C=10 and fixed small noise, and 50% reference masking only during training. Save windows two steps apart and obtain sparse +1 success labels from the operator.
For screw and zip tie, first learn from controlled critical-phase resets, then collect full-task experience with the base VLA handling early stages. Reported harder-task policies use about five hours of interaction data. Token architecture details beyond Figure 2, warmup duration and several numeric RL hyperparameters are not fully specified.
Equation
-C Additional experimental details for the baselines
All methods execute delta actions at 50 Hz. PLD first pretrains its critic on fifty base-policy rollouts using Cal-QL. DSRL predicts a 1×32 latent vector repeated fifty times to match the VLA noise interface.
HIL-SERL starts from twenty demonstration episodes and uses interventions, but runs at 50 Hz instead of the original system’s 10 Hz and lacks an action-space bounding box. The authors identify those changes as causes of difficulty, so the comparison is specific to this setup.
DAgger fine-tunes the VLA using demonstrations and the same intervention data collected during RLT. Its lower speed is consistent with imitation of those trajectories; it is not an independently optimized online-RL baseline.