Literature
Display
← All papers

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

Hyun Joe Jeong · Gokul Swamy · Andrea Bajcsy

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

A learned language feedback policy changes instructions to steer a frozen robot controller, while a calibrated improvement predictor decides whether to intervene. All prompt templates, benchmark instructions, result tables, and calibration equations are retained. Project videos.

At a glance

The method narrates robot videos, searches nearby instruction sequences through actual policy rollouts, and distills successful sequences into a feedback policy. A separately trained predictor estimates whether steering helps; calibration raises its intervention threshold using harmful examples from held-out perturbations. Seen-environment success improves by roughly 24.7% relatively in simulation and 65% on hardware. “Harmless” means limiting performance-degrading steering under a matching calibration distribution, not guaranteeing physical safety or success on every trial.

1 Introduction

Natural-language variants can evoke very different actions from the same pretrained controller, and some desired skills cannot be elicited at all. Interactive evaluation discovers useful phrasing; a reusable feedback policy adapts phrasing as the scene changes. A refusal decision keeps the original instruction when predicted benefit is insufficient. The controller remains frozen, but the language policy and benefit predictor are trained with interaction data, and deployment perturbations require calibration examples.

Refer to caption
Figure 1: Overview. Left: robot videos are narrated into per-frame language descriptions, yielding a task-relevant prior without action labels. Center: we generate trajectory-level perturbations around this prior, evaluate them through closed-loop VLA rollouts, and use high-improvement sequences for rejection fine-tuning and improvement-head supervision. Right: at deployment, LFP steers the frozen VLA only when the calibrated improvement head predicts benefit; otherwise, the system refuses steering and falls back to the base instruction.

2 Steering Pre-trained VLA Policies via Natural Language

Language steering is an outer decision process: observations and the original task condition a language action, which the frozen vision-language-action model converts into a low-level action chunk. Only a prefix executes before replanning. The objective is binary task success, not linguistic quality. Open-vocabulary search is large, language-to-action effects are unpredictable, and absent underlying skills cannot be created by wording alone; these motivate local search and refusal.

Equation 1
π⋆∈argmaxπ∈Π𝔼ξ∼πVLA∘π​[r⁡(ξ,ℓtask)].\pi^{\star}\in\argmax_{\pi\in\Pi}\mathbb{E}_{\xi\sim\pi^{\textsf{VLA}}\circ\pi}\left[r(\xi;\ell^{\textsf{task}})\right]. (1)

3 Learning Mostly Harmless Language Feedback Policies via Interaction

The method restricts search to semantic variants of narrated behavior, evaluates their closed-loop effect, and distills the selected sequences. Improvement is defined as expected success with the candidate sequence minus success with the original static instruction, under the same initial-state distribution. In practice, finite independent rollouts estimate both terms, so selected improvements are noisy estimates rather than known expected gains.

Equation 2
Δ(ℓtask,e,ℓ0:T)=\displaystyle\Delta(\ell^{\textsf{task}},e,\ell_{0:T})= 𝔼o0∼p⁡(o0∣ℓtask,e)𝔼ξ∼(πVLA∘ℓ0:T)(o0)[r(ξ;ℓtask)]\displaystyle\mathbb{E}_{o_{0}\sim p(o_{0}\mid\ell^{\textsf{task}},e)}\mathbb{E}_{\xi\sim(\pi^{\textsf{VLA}}\circ\ell_{0:T})(o_{0})}\left[r(\xi;\ell^{\textsf{task}})\right] (2)
−𝔼o0∼p⁡(o0∣ℓtask,e)​𝔼ξ∼πVLA​(o0,ℓtask)​[r⁡(ξ,ℓtask)].\displaystyle-\mathbb{E}_{o_{0}\sim p(o_{0}\mid\ell^{\textsf{task}},e)}\mathbb{E}_{\xi\sim\pi^{\textsf{VLA}}(o_{0},\ell^{\textsf{task}})}\left[r(\xi;\ell^{\textsf{task}})\right].

3.1 A Three-Phase Approach to Mostly Harmless Language Steering

Narration. A vision-language teacher decomposes and labels robot videos. Supervised language finetuning learns a proposal policy conditioned on images and the task. Interaction. A language teacher paraphrases whole candidate sequences; robot rollouts estimate their improvement. Selected positive-improvement sequences train the feedback policy by rejection finetuning, while a small head regresses improvement from the initial observation and task.

Calibration. A zero threshold can wrongly steer harmful tasks. Class-conditional calibration instead uses the upper score quantile among examples whose empirical improvement is negative. Deployment steers only above that threshold and otherwise repeats the original instruction. The stated false-positive guarantee requires calibration/test exchangeability and concerns harmful-class intervention probability, not the fraction of interventions that are harmful. Appendix 6.3 gives the finite-sample convention and important qualification about the written threshold rule.

Equations 3–7
πSFT=argmaxπ∈Π𝔼(o,ℓtask,ℓ)∼𝒟nar​[log⁡π⁡(ℓ∣o,ℓtask)].\pi^{\textsf{SFT}}=\argmax_{\pi\in\Pi}\mathbb{E}_{(o,\ell^{\textsf{task}},\ell)\sim\mathcal{D}_{\text{nar}}}\left[\log\pi(\ell\mid o,\ell^{\textsf{task}})\right]. (3)
ℓ0:T⋆(ℓtask,e)=argmaxℓ(n)0:T∈ℒlocal(ℓ¯0:T)Δ^(ℓtask,e,ℓ0:T(n)).\ell^{\star}_{0:T}(\ell^{\textsf{task}},e)=\argmax_{\ell^{(n)}_{0:T}\in\mathcal{L}_{\text{local}}(\bar{\ell}_{0:T})}\widehat{\Delta}(\ell^{\textsf{task}},e,\ell^{(n)}_{0:T}). (4)
πRFT=argmaxπ∈Π𝔼(o,ℓtask,ℓ⋆)∼𝒟RFT​[log⁡π⁡(ℓ⋆∣o,ℓtask)].\pi^{\textsf{RFT}}=\argmax_{\pi\in\Pi}\mathbb{E}_{(o,\ell^{\textsf{task}},\ell^{\star})\sim\mathcal{D}_{\text{RFT}}}\left[\log\pi(\ell^{\star}\mid o,\ell^{\textsf{task}})\right]. (5)
ψ=argminψ∈Ψ𝔼(o0,ℓtask,e,Δ^)∼𝒟Δ​[(ψ⁡(o0,ℓtask)−Δ^)2].\psi=\argmin_{\psi\in\Psi}\mathbb{E}_{(o_{0},\ell^{\textsf{task}},e,\widehat{\Delta})\sim\mathcal{D}_{\Delta}}\left[\left(\psi(o_{0},\ell^{\textsf{task}})-\widehat{\Delta}\right)^{2}\right]. (6)
ℙ⁡(ψ⁡(X)≥q^α∣Y=0)≤α.\mathbb{P}\left(\psi(X)\geq\hat{q}_{\alpha}\mid Y=0\right)\leq\alpha. (7)

4 Experiments

Experiments examine success, refusal calibration, open- versus closed-loop language updates, sequence-level search, and transfer to a second frozen controller. Simulation covers visual and semantic perturbations; physical tests use a Franka manipulator on a small tabletop suite. Error bars summarize pooled episodes, rather than independently trained-model uncertainty.

4.1 Experimental Setup

The primary controller is π₀.₅, specialized to LIBERO in simulation and DROID for the robot. A Qwen3-VL-4B language policy uses LoRA, Molmo2-8B narrations, and sixteen GPT-5.4 sequence variants. Fifty base-policy videos per task initialize narration. At deployment, language is greedily decoded from wrist and external camera views.

The ten-task simulation suite combines four visual environments with five instruction versions, giving two hundred combinations plus thirteen novel behavior compositions. “In distribution” is relative to language-policy training in Visual-Viewpoint; that scene is itself a shift for the original LIBERO controller. Four physical training tasks receive nominal, distractor, and paraphrase tests, with thirty rollouts per condition; MarkerBowl and ChipsCup are novel tasks. These tasks and their exact instructions remain in the appendix tables.

4.2 Q1: Why Should We Do Language Steering of VLAs?

Simulation uses fifty rollouts per task/perturbation combination. Narration alone can improve the training scene while hurting deployment. Interactive language finetuning reaches 63.9% training-scene success versus 51.2% for the base controller; calibration gives 63.8%. The approximately 24.7% headline is a relative improvement, about 12.6 percentage points using rounded table values. Physical nominal-task averages rise from 50% to 82.5% with calibration, a 65% relative gain.

The ten-versus-fifty successful-rollout ablation favors language finetuning over direct action-policy LoRA at these settings. It measures the final successful finetuning dataset, not total interaction or compute: the language method also performs candidate search, narration, and calibration. Novel composition transfer is limited—seven of thirteen tasks score zero for every method, and language steering worsens task fifteen. Table 11 also shows that uncalibrated steering can degrade a held-out scene.

Refer to caption
Figure 4: 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} steers 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} in unseen behavior Compose tasks. Error bars are standard error over episodes across all perturbations.
Refer to caption
Figure 5: 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} learns more from same on-policy rollout data compared to 𝝅VLA-SFT{\color[rgb]{0.2617,0.5742,0.5586}\boldsymbol{\pi}^{\textsf{VLA-SFT}}}. Error bars are standard error over episodes across all perturbation combinations.
Refer to caption
Figure 2: Simulation. Left: Baseline comparisons are shown in the language-policy training environment vs. held-out deployment environments; error bars are standard error over pooled episodes across perturbation combinations. 𝝅SFT{\color[rgb]{0.418,0.1016,0.6523}\boldsymbol{\pi}^{\textsf{SFT}}} improves ID performance but degrades OOD performance relative to 𝝅Base{\color[rgb]{0.7344,0.1563,0.4023}\boldsymbol{\pi}^{\textsf{Base}}}, showing that narration is not enough for robustness. Interactive search and conformalization recover OOD robustness, with 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} and 𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}} outperforming all baselines. Note that absolute success is higher in deployment environments because LIBERO-10 is near-saturated for the base VLA. Right: training and deployment environments are visualized for selected tasks.
Refer to caption
Figure 3: Hardware. We have four training tasks (Nom.) and their test-time visual (VOOD) and semantic (SOOD) perturbations, and two novel tasks (right). 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} usually improves task performance, and conformalization reduces harmful interventions on tasks such as Microwave.

4.3 Q2: When Should We (Not) Do Language Steering?

Always steering achieves 70.93% over ten thousand pooled simulation rollouts; an uncalibrated refusal head raises this to 74.96%, and calibration to 75.96%. At target false-positive rate 0.1, measured harmful-class intervention falls from 38.92% to 9.31% in simulation and 61.11% to 2.22% physically. Simulation refusal rises from 43.42% to 57.66%, trading some helpful interventions for fewer harmful ones.

A reliability sweep uses five randomized calibration/test splits and ten test episodes per condition. The result supports calibration on those perturbations; it does not establish robustness to an unrelated deployment distribution. “Accuracy” in Table 1 classifies intervention decisions, not robot task completion or physical safety.

Accuracy (↑\uparrow) TPR (↑\uparrow) FPR (↓\downarrow) TNR (↑\uparrow) FNR (↓\downarrow) Refusal Rate
Simulation 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 66.22%66.22\% 70.18%\mathbf{70.18\%} 38.92%38.92\% 61.08%61.08\% 29.82%\mathbf{29.82\%} 43.42%43.42\%
𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}} 77.74%\mathbf{77.74\%} 67.77%67.77\% 9.31%\mathbf{9.31\%} 90.69%\mathbf{90.69\%} 32.23%32.23\% 57.66%57.66\%
Hardware 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 84.72%84.72\% 100.00%\mathbf{100.00\%} 61.11%61.11\% 38.89%38.89\% 0.00%\mathbf{0.00\%} 9.72%9.72\%
𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}} 99.17%\mathbf{99.17\%} 99.63%99.63\% 2.22%\mathbf{2.22\%} 97.78%\mathbf{97.78\%} 0.37%0.37\% 24.72%24.72\%
Table 1: Confusion Matrix of πRFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}}’s Language Interventions. After conformalization done at α=0.1\alpha=0.1, FPR decreases from 38.92%38.92\% to 9.31%9.31\% in simulation and from 61.11%61.11\% to 2.22%2.22\% on hardware. This shows that conformalization controls FPR at or below the target level, mitigating harmful steering while only slightly decreasing TPR.
Refer to caption
Figure 6: Simulation Refusal Results. Left: success rate vs. FPR for no refusal 𝝅NR{\color[rgb]{0.7461,0.1914,0.1719}\boldsymbol{\pi}^{\textsf{NR}}}, 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}}, and 𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}}. Both refusal and conformalization help performance. Right: reliability diagram shows that empirical FPR tracks the target α\alpha across 5 randomized splits.

4.4 Q3: How Does Open- vs. Closed-Loop Language Feedback Influence VLA Performance?

Closed-loop sequence-level steering averages 75.0% success, compared with 71.3% for a single static rewrite and 70.4% for independently perturbed phrases. Refusal rates remain around 43–46%. A qualitative physical example moves a correctly sorted block back into the scene; updated language prompts recover its intended bin while the base controller misplaces it. This demonstrates one recovery behavior, not a quantified guarantee against arbitrary mid-task disturbances.

Method SR (%) Refusal (%)
OL 71.3±0.571.3\pm 0.5 43.543.5
Phrase (CL) 70.4±0.570.4\pm 0.5 45.945.9
Traj. (CL) 75.0±0.4\mathbf{75.0\pm 0.4} 43.4\mathbf{43.4}
Table 2: Simulation Search Ablations. OL: open-loop; CL: closed-loop. Mean success rate with standard error over episodes across all perturbation combinations are shown.
Refer to caption
Figure 7: Closed-loop Language Steering vs. Base VLA. After the robot initially places the green cube correctly, a human moves the cube back into the scene. 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} continues executing from the original task instruction and misplaces the cube, whereas 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} observes the changed state, outputs an updated language action ℓt\ell_{t}, and steers the frozen VLA to recover from the perturbation.

4.5 Q4: Can We Learn Language Feedback Policies for Different Pre-Trained VLA’s?

A separately trained language policy also steers MolmoAct2: simulation success rises from 32.5% to 40.6% over ten tasks, and physical CubeSort from 40.0% to 76.7% over thirty trials. This supports applicability to more than one frozen controller; it does not show one unchanged language policy transfers freely between controllers.

𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}}
Sim. SR(%) 32.5±1.032.5\pm 1.0 40.6±1.0\mathbf{40.6\pm 1.0}
HW SR(%) 40.0±8.940.0\pm 8.9 76.7±7.7\mathbf{76.7\pm 7.7}
Table 3: 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} deployed on MolmoAct2 in simulation (Visual-Viewpoint; 50 trials per task, 10 tasks) and hardware (CubeSort; 30 trials).

5 Limitations

Language has limited control authority when the underlying controller ignores wording or lacks a needed skill. Search covers local paraphrases around a narration, not the full linguistic space. Calibration assumes deployment perturbations match the calibration distribution. Other observation, latent, or action interventions may complement language steering; general robustness outside calibrated distributions remains unresolved.

Acknowledgments

The authors thank Junwon Seo for calibration discussions and acknowledge DARPA SAFRON and an STTR grant. Their conclusions are not official sponsor policy.

6 Appendix

The appendix preserves related work, the four complete prompt templates, training hyperparameters, the calibration argument, exact benchmark instructions, full per-task results, latency, and data-size ablations.

6.1 Related Work

The approach complements action-token and continuous-action robot models, hierarchical language control, prompt optimization, latent steering, action guardrails, and failure prediction. Its distinguishing interface is a distilled closed-loop language policy over an unchanged controller, with selective refusal calibrated using class-conditional conformal prediction. This does not replace lower-level safety constraints.

6.2 Method and Implementation Details

Molmo2 decomposes each video, then uses a common frequent decomposition to assign contiguous step labels. Four original prompt figures specify decomposition, frame narration, sequence perturbation, and next-step language generation. Narration and rejection finetuning each use one epoch, batch size one, bfloat16, AdamW at 10⁻⁵; narration uses LoRA rank sixty-four and weight decay 0.01, while rejection finetuning uses zero decay. The improvement head uses standardized frozen-policy features, a sixty-four-unit hidden layer, mean squared error, and early stopping with 20% validation.

Sixteen candidates initially receive twenty-five simulated rollouts each; eight finalists receive seventy-five more, totaling one thousand candidate evaluations per task before base comparisons and final successful demonstrations. Hardware uses ten initial and twenty additional rollouts, totaling 320 candidate evaluations. Fifty successful selected-policy trajectories per task train the final language policy. The base comparison is independently initialized from the same distribution, not paired by identical rollout noise.

Calibration collects twenty samples per perturbation combination. The initial-observation head gates the intervention; when allowed, language updates at every controller replan—five simulation steps or eight DROID steps. Teachers generate training labels and variants offline; they are not the deployment feedback model. Original hyperparameter Tables 5 and 6 are separated here for legibility.

Original figure
Figure 8: Prompt template for task decomposition.
Original figure
Figure 9: Prompt template for narration.
Original figure
Figure 10: Prompt template for candidate sequence generation.
Original figure
Figure 11: Prompt template for closed-loop steering.
Category Value
Model
Base model Qwen3-VL-4B-Instruct
Adapter LoRA (r=64, α=128\alpha=128, dropout=0)
Target modules q, k, v, o, gate, up, down_proj
Data
Inputs Wrist + agent view images
Supervision Per-step (ot,ℓt)(o_{t},\ell_{t}) demonstrations
Loss Cross-entropy
Optimization
Optimizer AdamW
Learning rate 1×10−51\times 10^{-5}
Weight decay 0.010.01
Batch size 1 (no accumulation)
Epochs 1
Precision bfloat16
Table 4: Narrated fine-tuning hyperparameters.
Category Value
Model
Base model Qwen3-VL-4B-Instruct
Initialization SFT LoRA checkpoint (frozen base)
Data
Inputs Wrist + agent view images
Supervision Per-step (ot,ℓt)(o_{t},\ell_{t}) demonstrations
Loss Cross-entropy
Optimization
Optimizer AdamW
Learning rate 1×10−51\times 10^{-5}
Weight decay 0
Batch size 1
Epochs 1
Precision bfloat16
Table 5: Expert Iteration hyperparameters.
Category Value
Model
Backbone Frozen 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}}
Feature Final hidden state
Head MLP (d→64→1d\rightarrow 64\rightarrow 1, ReLU, dropout=0.1)
Feature norm Per-dimension z-score
Data
Inputs Hidden state features
Supervision Task improvement
Target Δ^i\hat{\Delta}_{i}
Optimization
Optimizer Adam
Learning rate 1×10−31\times 10^{-3}
Weight decay 1×10−31\times 10^{-3}
Loss MSE
Epochs up to 200 (early stopping)
Validation split 20%
Table 6: improvement predictor hyperparameters.

6.3 Conformal Guarantee for Calibrated Refusal

The harmful class is defined by a negative Monte Carlo improvement estimate. Sort its m calibration scores and choose order statistic ceil((m+1)(1−α)); if that index exceeds m, the threshold is positive infinity and steering is refused. The argument assumes calibration and deployment scores are exchangeable conditional on this empirical harmful label. It does not account explicitly for uncertainty in the estimated improvement label or guarantee the sign of true expected improvement on every case.

The source states a bound on the chance of steering conditional on harmfulness. That differs from a bound conditional on having steered, and from a robot injury or collision probability. As written, the source uses a non-strict ≥ threshold while mentioning ties without specifying their treatment; tied scores require additional care for the claimed finite-sample bound. All original equations are retained unchanged rather than silently replacing that rule.

Equations 8–9
q^α=s(k)0,k=⌈(m+1)​(1−α)⌉,\hat{q}_{\alpha}=s^{0}_{(k)},\qquad k=\left\lceil(m+1)(1-\alpha)\right\rceil, (8)
{si:i∈ℐ0}∪{s}\{s^{i}:i\in\mathcal{I}^{0}\}\cup\{s\}
s=ψ⁡(X)≥q^α.s=\psi(X)\geq\hat{q}_{\alpha}.
ℙ⁡(ψ⁡(X)≥q^α∣Y=0)≤α.\mathbb{P}\left(\psi(X)\geq\hat{q}_{\alpha}\mid Y=0\right)\leq\alpha. (9)
Refer to caption
Figure 12: 10 LIBERO-OOD tasks (left to right), as well as their perturbations (top to bottom).
Task Nominal Task Instruction
0 put both the alphabet soup and the tomato sauce in the basket
1 put both the cream cheese box and the butter in the basket
2 turn on the stove and put the moka pot on it
3 put the black bowl in the bottom drawer of the cabinet and close it
4 put the white mug on the left plate and put the yellow and white mug on the right plate
5 pick up the book and place it in the back compartment of the caddy
6 put the white mug on the plate and put the chocolate pudding to the right of the plate
7 put both the alphabet soup and the cream cheese box in the basket
8 put both moka pots on the stove
9 put the yellow and white mug in the microwave and close it
Table 7: LIBERO-10 task instructions.
Task Perturbed Instructions
0 SOOD 1: place the alphabet soup in the basket and the tomato sauce as well
SOOD 2: put both the can of soup and can of sauce in the basket
SOOD 3: put away the alphabet soup and tomato sauce in the basket
SOOD 4: place the can of soup in the basket and the can of sauce in the basket
1 SOOD 1: pick up both the cream cheese box and the butter and place them in the basket
SOOD 2: put both the box of cheese and the box of butter in the basket
SOOD 3: put away the cheese box and butter box in the basket
SOOD 4: stash the cream cheese and butter
2 SOOD 1: switch the stove on and set the moka pot on the stove
SOOD 2: turn on the cooktop and place the moka machine on it
SOOD 3: put the moka pot over the heat
SOOD 4: heat up the stove and put the moka pot on it
3 SOOD 1: move the black bowl to the bottom drawer and close the drawer of the cabinet
SOOD 2: put the middle bowl to the lowest drawer and close it
SOOD 3: put away the black bowl in the bottom drawer
SOOD 4: stash away the middle bowl in the lower cabinet
4 SOOD 1: place two mugs on the plates, left one is the white mug and the right one is the yellow and white mug
SOOD 2: put the pure white cup on the left plate and put the other one with the yellow handle on the right plate
SOOD 3: serve the cups
SOOD 4: serve the mugs
5 SOOD 1: grab the standing book and transfer it to the back compartment of the caddy
SOOD 2: pick up the right book and put it in the rear part of the caddy
SOOD 3: place the book in the back of the caddy
SOOD 4: put away the book in the back part
6 SOOD 1: set the white mug on the plate with chocolate pudding to be placed to the right
SOOD 2: put the pure white cup on the middle plate and put the brown chocolate to the right of the plate
SOOD 3: place the white cup on the plate and the chocolate to the right
SOOD 4: set the white mug on the plate and the chocolate pudding set on the right
7 SOOD 1: put both the can of soup and the box of cheese in the basket
SOOD 2: move the two objects, alphabet soup and cream cheese box, to the basket
SOOD 3: add the cream cheese box and the alphabet soup to the collection
SOOD 4: grab the alphabet soup and the cream cheese box and throw them in the basket
8 SOOD 1: transfer both moka pots from the table to the stove
SOOD 2: put both the moka coffee makers on the cooktop
SOOD 3: put the coffee makers where they can heat up
SOOD 4: heat up both moka pots
9 SOOD 1: place the yellow and white mug inside the microwave, then shut the door
SOOD 2: put the middle mug inside the microwave and close the door
SOOD 3: heat up the yellow and white mug by placing it and shutting the door
SOOD 4: heat up the yellow and white mug in the microwave
Table 8: LIBERO-OOD semantic perturbations (SOOD1-2) and newly added semantic perturbations (SOOD 3-4).

6.4 Benchmark Details

The appendix enumerates ten nominal tasks, forty paraphrases, thirteen behavior compositions, and six physical tasks. Some paraphrases omit relations or change references, making the exact tables important for interpreting the “semantic” stress test. Visual-Viewpoint is the language training scene; original LIBERO is a different scene for that policy but familiar to the underlying controller.

In the most shifted visual condition, uncalibrated success drops from the base’s 89.8% to 88.5%; calibration reaches 90.9%. Composition task thirteen improves from 36% to 88%, while task fifteen falls from 40% to 30%; tasks sixteen through twenty-two remain unsolved. The roughly 38% composition bar in Figure 4 matches averaging tasks ten through fifteen; including all thirteen Table 12 tasks gives about 17.7% for language steering. The narrative’s broad claim of success on most compositions is not supported by these full thirteen-task results.

For physical Microwave under a paraphrase, steering drops success from 66.7% to 46.7%, and calibration yields 70%. Novel MarkerBowl and ChipsCup reach 80% and 73.3% with uncalibrated steering; the table does not report calibrated scores for either. These details distinguish targeted transfer from broad calibrated generalization.

Refer to caption
Figure 13: LIBERO-OOD Novel Behavior Composition tasks. Tasks 10-16 correspond to the 1st environment, tasks 17-18 to the 2nd, task 19 to the 3rd, task 20-21 to the 4th, and task 22 to the 5th.
Task Instruction
10 put both the cream cheese box and the tomato sauce in the basket
11 put both the milk and the tomato sauce in the basket
12 put both the orange juice and the tomato sauce in the basket
13 put both the alphabet soup and the butter in the basket
14 put both the orange juice and the butter in the basket
15 put both the tomato sauce and the butter in the basket
16 put both the milk and the butter in the basket
17 put both the ketchup and the cream cheese box in the basket
18 put both the tomato sauce and the cream cheese box in the basket
19 put the wine bottle in the bottom drawer of the cabinet and close it
20 put the red mug on the plate and put the chocolate pudding to the right of the plate
21 put the red mug on the plate and put the chocolate pudding to the left of the plate
22 put the red mug on the left plate and put the yellow and white mug on the right plate
Table 9: BOOD (behavior composition) task instructions for LIBERO-10 tasks.
Refer to caption
Figure 14: Hardware Environments. Left: training environments and their visual perturbations. CubeSort, CubeMug, Microwave, and MarkerBlock shown left to right. Right: novel environments MarkerBowl (top) and ChipsCup (bottom).
Task Nominal Semantic Perturbation
CubeSort put the green cube in the blue basket and the red cube in the orange basket put away the green cube and the red cube in similar color baskets
CubeMug place the green cube in the blue mug then the blue mug in the orange basket put away the cube in the mug and the mug in the bin
Microwave place the orange bowl on top of the microwave and close the microwave door put the bowl on top of the oven and close the door
MarkerBlock place the marker and green block on the plate put away the pen and green block in the bowl
MarkerBowl put the marker in the blue bowl and put the blue bowl on the plate —
ChipsCup put the chips and green cup in the basket —
Table 10: Hardware task descriptions (nominal and semantic perturbation).
Condition 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 𝝅VLA-SFT{\color[rgb]{0.2617,0.5742,0.5586}\boldsymbol{\pi}^{\textsf{VLA-SFT}}} 𝝅Base{\color[rgb]{0.7344,0.1563,0.4023}\boldsymbol{\pi}^{\textsf{Base}}} 𝝅SFT{\color[rgb]{0.418,0.1016,0.6523}\boldsymbol{\pi}^{\textsf{SFT}}} 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}}
Train Env 51.2 60.6 53.1 55.0 63.9 63.8
ID Scene, OOD Distractor 46.7 50.7 45.7 47.3 55.0 54.2
OOD Scene, ID Distractor 92.4 88.2 92.4 89.1 92.5 95.0
OOD Scene, OOD Distractor 89.8 82.6 89.5 81.2 88.5 90.9
Table 11: Baseline success rates (%) by visual perturbation condition, pooled over tasks and semantic perturbations.
Method 10 11 12 13 14 15 16 17 18 19 20 21 22
𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 88.0 0.0 2.0 36.0 4.0 40.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
𝝅VLA-SFT{\color[rgb]{0.2617,0.5742,0.5586}\boldsymbol{\pi}^{\textsf{VLA-SFT}}} 94.0 2.0 10.0 10.0 0.0 46.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 98.0 4.0 10.0 88.0 0.0 30.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Table 12: novel task composition per-task success rates (%). Tasks 16–22 have 0% SR (no method succeeds).
Task Condition 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 𝝅SFT{\color[rgb]{0.418,0.1016,0.6523}\boldsymbol{\pi}^{\textsf{SFT}}} 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 𝝅CP{\color[rgb]{1,0.4961,0.0547}\boldsymbol{\pi}^{\textsf{CP}}}
CubeSort Nominal 56.7 80.0 93.3 90.0
VOOD 13.3 43.3 70.0 70.0
SOOD 6.7 26.7 60.0 63.3
CubeMug Nominal 36.7 63.3 83.3 76.7
VOOD 30.0 40.0 66.7 73.3
SOOD 16.7 36.7 70.0 70.0
Microwave Nominal 73.3 66.7 76.7 73.3
VOOD 46.7 16.7 50.0 50.0
SOOD 66.7 40.0 46.7 70.0
MarkerBlock Nominal 33.3 80.0 90.0 90.0
VOOD 36.7 60.0 76.7 76.7
SOOD 30.0 43.3 63.3 70.0
MarkerBowl Nominal 30.0 46.7 80.0 —
ChipsCup Nominal 33.3 46.7 73.3 —
Table 13: Hardware success rates (%) by task and perturbation condition.

6.5 Additional Results

On the reported A6000 Ada platform, a replan grows from 68 ms for the controller alone to 274 ms when language generation adds 204 ms, reducing measured frequency from 14.7 to 3.6 Hz. Five environment steps at 20 Hz span 250 ms, so these timings do not by themselves establish a strict four-hertz real-time deadline. Caching, asynchronous updates, and distillation are proposed improvements.

Trajectory-length means include successful hardware trials only. Steering shortens several tasks, including ChipsCup from 870.8 to 562.2 steps, but lengthens MarkerBowl from 631.5 to 657.0; these are neither unconditional episode costs nor wall-clock savings. With ten, thirty, and fifty narration videos per task, final simulation success is 60.8%, 59.7%, and 63.9% versus the base’s 51.2%. This supports useful interaction-stage gains in this range, without eliminating narration or search cost.

Config 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} (ms) 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} (ms) Total (ms) Replan Hz
𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} only 6868 — 6868 14.714.7
𝝅VLA∘𝝅RFT{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}}\circ{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} 7070 204204 274274 3.63.6
Table 14: Inference latency per replanning step, measured over 50 replanning steps on a single NVIDIA A6000 Ada GPU. Each replanning step corresponds to 5 environment steps at 20 Hz.
Task 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 𝝅SFT{\color[rgb]{0.418,0.1016,0.6523}\boldsymbol{\pi}^{\textsf{SFT}}} 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}}
CubeSort 709.5 587.5 627.4
CubeMug 680.9 578.3 542.1
Microwave 499.0 469.2 425.2
MarkerBlock 775.6 696.1 596.7
MarkerBowl 631.5 578.1 657.0
ChipsCup 870.8 728.4 562.2
Table 15: Mean successful trajectory length (steps) on nominal hardware tasks.
Model (Data) 𝝅VLA{\color[rgb]{0.4961,0.4961,0.4961}\boldsymbol{\pi}^{\textsf{VLA}}} 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} (10) 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} (30) 𝝅RFT{\color[rgb]{0.1211,0.4648,0.707}\boldsymbol{\pi}^{\textsf{RFT}}} (50)
SR(%) 51.2 ±\pm 1.0 60.8 ±\pm 1.0 59.7 ±\pm 1.0 63.9 ±\pm 1.0
Table 16: Narration data size ablation on Visual-Viewpoint tasks.
Concept demonstration

Open image ↗