Overview
A learned language feedback policy changes instructions to steer a frozen robot controller, while a calibrated improvement predictor decides whether to intervene. All prompt templates, benchmark instructions, result tables, and calibration equations are retained. Project videos.
At a glance
The method narrates robot videos, searches nearby instruction sequences through actual policy rollouts, and distills successful sequences into a feedback policy. A separately trained predictor estimates whether steering helps; calibration raises its intervention threshold using harmful examples from held-out perturbations. Seen-environment success improves by roughly 24.7% relatively in simulation and 65% on hardware. “Harmless” means limiting performance-degrading steering under a matching calibration distribution, not guaranteeing physical safety or success on every trial.
1 Introduction
Natural-language variants can evoke very different actions from the same pretrained controller, and some desired skills cannot be elicited at all. Interactive evaluation discovers useful phrasing; a reusable feedback policy adapts phrasing as the scene changes. A refusal decision keeps the original instruction when predicted benefit is insufficient. The controller remains frozen, but the language policy and benefit predictor are trained with interaction data, and deployment perturbations require calibration examples.
2 Steering Pre-trained VLA Policies via Natural Language
Language steering is an outer decision process: observations and the original task condition a language action, which the frozen vision-language-action model converts into a low-level action chunk. Only a prefix executes before replanning. The objective is binary task success, not linguistic quality. Open-vocabulary search is large, language-to-action effects are unpredictable, and absent underlying skills cannot be created by wording alone; these motivate local search and refusal.
Equation 1
| (1) |
3 Learning Mostly Harmless Language Feedback Policies via Interaction
The method restricts search to semantic variants of narrated behavior, evaluates their closed-loop effect, and distills the selected sequences. Improvement is defined as expected success with the candidate sequence minus success with the original static instruction, under the same initial-state distribution. In practice, finite independent rollouts estimate both terms, so selected improvements are noisy estimates rather than known expected gains.
Equation 2
| (2) | ||||
3.1 A Three-Phase Approach to Mostly Harmless Language Steering
Narration. A vision-language teacher decomposes and labels robot videos. Supervised language finetuning learns a proposal policy conditioned on images and the task. Interaction. A language teacher paraphrases whole candidate sequences; robot rollouts estimate their improvement. Selected positive-improvement sequences train the feedback policy by rejection finetuning, while a small head regresses improvement from the initial observation and task.
Calibration. A zero threshold can wrongly steer harmful tasks. Class-conditional calibration instead uses the upper score quantile among examples whose empirical improvement is negative. Deployment steers only above that threshold and otherwise repeats the original instruction. The stated false-positive guarantee requires calibration/test exchangeability and concerns harmful-class intervention probability, not the fraction of interventions that are harmful. Appendix 6.3 gives the finite-sample convention and important qualification about the written threshold rule.
Equations 3–7
| (3) |
| (4) |
| (5) |
| (6) |
| (7) |
4 Experiments
Experiments examine success, refusal calibration, open- versus closed-loop language updates, sequence-level search, and transfer to a second frozen controller. Simulation covers visual and semantic perturbations; physical tests use a Franka manipulator on a small tabletop suite. Error bars summarize pooled episodes, rather than independently trained-model uncertainty.
4.1 Experimental Setup
The primary controller is π₀.₅, specialized to LIBERO in simulation and DROID for the robot. A Qwen3-VL-4B language policy uses LoRA, Molmo2-8B narrations, and sixteen GPT-5.4 sequence variants. Fifty base-policy videos per task initialize narration. At deployment, language is greedily decoded from wrist and external camera views.
The ten-task simulation suite combines four visual environments with five instruction versions, giving two hundred combinations plus thirteen novel behavior compositions. “In distribution” is relative to language-policy training in Visual-Viewpoint; that scene is itself a shift for the original LIBERO controller. Four physical training tasks receive nominal, distractor, and paraphrase tests, with thirty rollouts per condition; MarkerBowl and ChipsCup are novel tasks. These tasks and their exact instructions remain in the appendix tables.
4.2 Q1: Why Should We Do Language Steering of VLAs?
Simulation uses fifty rollouts per task/perturbation combination. Narration alone can improve the training scene while hurting deployment. Interactive language finetuning reaches 63.9% training-scene success versus 51.2% for the base controller; calibration gives 63.8%. The approximately 24.7% headline is a relative improvement, about 12.6 percentage points using rounded table values. Physical nominal-task averages rise from 50% to 82.5% with calibration, a 65% relative gain.
The ten-versus-fifty successful-rollout ablation favors language finetuning over direct action-policy LoRA at these settings. It measures the final successful finetuning dataset, not total interaction or compute: the language method also performs candidate search, narration, and calibration. Novel composition transfer is limited—seven of thirteen tasks score zero for every method, and language steering worsens task fifteen. Table 11 also shows that uncalibrated steering can degrade a held-out scene.
4.3 Q2: When Should We (Not) Do Language Steering?
Always steering achieves 70.93% over ten thousand pooled simulation rollouts; an uncalibrated refusal head raises this to 74.96%, and calibration to 75.96%. At target false-positive rate 0.1, measured harmful-class intervention falls from 38.92% to 9.31% in simulation and 61.11% to 2.22% physically. Simulation refusal rises from 43.42% to 57.66%, trading some helpful interventions for fewer harmful ones.
A reliability sweep uses five randomized calibration/test splits and ten test episodes per condition. The result supports calibration on those perturbations; it does not establish robustness to an unrelated deployment distribution. “Accuracy” in Table 1 classifies intervention decisions, not robot task completion or physical safety.
| Accuracy () | TPR () | FPR () | TNR () | FNR () | Refusal Rate | ||
|---|---|---|---|---|---|---|---|
| Simulation | |||||||
| Hardware | |||||||
4.4 Q3: How Does Open- vs. Closed-Loop Language Feedback Influence VLA Performance?
Closed-loop sequence-level steering averages 75.0% success, compared with 71.3% for a single static rewrite and 70.4% for independently perturbed phrases. Refusal rates remain around 43–46%. A qualitative physical example moves a correctly sorted block back into the scene; updated language prompts recover its intended bin while the base controller misplaces it. This demonstrates one recovery behavior, not a quantified guarantee against arbitrary mid-task disturbances.
| Method | SR (%) | Refusal (%) |
|---|---|---|
| OL | ||
| Phrase (CL) | ||
| Traj. (CL) |
4.5 Q4: Can We Learn Language Feedback Policies for Different Pre-Trained VLA’s?
A separately trained language policy also steers MolmoAct2: simulation success rises from 32.5% to 40.6% over ten tasks, and physical CubeSort from 40.0% to 76.7% over thirty trials. This supports applicability to more than one frozen controller; it does not show one unchanged language policy transfers freely between controllers.
5 Limitations
Language has limited control authority when the underlying controller ignores wording or lacks a needed skill. Search covers local paraphrases around a narration, not the full linguistic space. Calibration assumes deployment perturbations match the calibration distribution. Other observation, latent, or action interventions may complement language steering; general robustness outside calibrated distributions remains unresolved.
Acknowledgments
The authors thank Junwon Seo for calibration discussions and acknowledge DARPA SAFRON and an STTR grant. Their conclusions are not official sponsor policy.
6 Appendix
The appendix preserves related work, the four complete prompt templates, training hyperparameters, the calibration argument, exact benchmark instructions, full per-task results, latency, and data-size ablations.
6.1 Related Work
The approach complements action-token and continuous-action robot models, hierarchical language control, prompt optimization, latent steering, action guardrails, and failure prediction. Its distinguishing interface is a distilled closed-loop language policy over an unchanged controller, with selective refusal calibrated using class-conditional conformal prediction. This does not replace lower-level safety constraints.
6.2 Method and Implementation Details
Molmo2 decomposes each video, then uses a common frequent decomposition to assign contiguous step labels. Four original prompt figures specify decomposition, frame narration, sequence perturbation, and next-step language generation. Narration and rejection finetuning each use one epoch, batch size one, bfloat16, AdamW at 10⁻⁵; narration uses LoRA rank sixty-four and weight decay 0.01, while rejection finetuning uses zero decay. The improvement head uses standardized frozen-policy features, a sixty-four-unit hidden layer, mean squared error, and early stopping with 20% validation.
Sixteen candidates initially receive twenty-five simulated rollouts each; eight finalists receive seventy-five more, totaling one thousand candidate evaluations per task before base comparisons and final successful demonstrations. Hardware uses ten initial and twenty additional rollouts, totaling 320 candidate evaluations. Fifty successful selected-policy trajectories per task train the final language policy. The base comparison is independently initialized from the same distribution, not paired by identical rollout noise.
Calibration collects twenty samples per perturbation combination. The initial-observation head gates the intervention; when allowed, language updates at every controller replan—five simulation steps or eight DROID steps. Teachers generate training labels and variants offline; they are not the deployment feedback model. Original hyperparameter Tables 5 and 6 are separated here for legibility.
| Category | Value |
|---|---|
| Model | |
| Base model | Qwen3-VL-4B-Instruct |
| Adapter | LoRA (r=64, , dropout=0) |
| Target modules | q, k, v, o, gate, up, down_proj |
| Data | |
| Inputs | Wrist + agent view images |
| Supervision | Per-step demonstrations |
| Loss | Cross-entropy |
| Optimization | |
| Optimizer | AdamW |
| Learning rate | |
| Weight decay | |
| Batch size | 1 (no accumulation) |
| Epochs | 1 |
| Precision | bfloat16 |
| Category | Value |
|---|---|
| Model | |
| Base model | Qwen3-VL-4B-Instruct |
| Initialization | SFT LoRA checkpoint (frozen base) |
| Data | |
| Inputs | Wrist + agent view images |
| Supervision | Per-step demonstrations |
| Loss | Cross-entropy |
| Optimization | |
| Optimizer | AdamW |
| Learning rate | |
| Weight decay | 0 |
| Batch size | 1 |
| Epochs | 1 |
| Precision | bfloat16 |
| Category | Value |
|---|---|
| Model | |
| Backbone | Frozen |
| Feature | Final hidden state |
| Head | MLP (, ReLU, dropout=0.1) |
| Feature norm | Per-dimension z-score |
| Data | |
| Inputs | Hidden state features |
| Supervision | Task improvement |
| Target | |
| Optimization | |
| Optimizer | Adam |
| Learning rate | |
| Weight decay | |
| Loss | MSE |
| Epochs | up to 200 (early stopping) |
| Validation split | 20% |
6.3 Conformal Guarantee for Calibrated Refusal
The harmful class is defined by a negative Monte Carlo improvement estimate. Sort its m calibration scores and choose order statistic ceil((m+1)(1−α)); if that index exceeds m, the threshold is positive infinity and steering is refused. The argument assumes calibration and deployment scores are exchangeable conditional on this empirical harmful label. It does not account explicitly for uncertainty in the estimated improvement label or guarantee the sign of true expected improvement on every case.
The source states a bound on the chance of steering conditional on harmfulness. That differs from a bound conditional on having steered, and from a robot injury or collision probability. As written, the source uses a non-strict ≥ threshold while mentioning ties without specifying their treatment; tied scores require additional care for the claimed finite-sample bound. All original equations are retained unchanged rather than silently replacing that rule.
Equations 8–9
| (8) |
| (9) |
| Task | Nominal Task Instruction |
|---|---|
| 0 | put both the alphabet soup and the tomato sauce in the basket |
| 1 | put both the cream cheese box and the butter in the basket |
| 2 | turn on the stove and put the moka pot on it |
| 3 | put the black bowl in the bottom drawer of the cabinet and close it |
| 4 | put the white mug on the left plate and put the yellow and white mug on the right plate |
| 5 | pick up the book and place it in the back compartment of the caddy |
| 6 | put the white mug on the plate and put the chocolate pudding to the right of the plate |
| 7 | put both the alphabet soup and the cream cheese box in the basket |
| 8 | put both moka pots on the stove |
| 9 | put the yellow and white mug in the microwave and close it |
| Task | Perturbed Instructions |
|---|---|
| 0 | SOOD 1: place the alphabet soup in the basket and the tomato sauce as well |
| SOOD 2: put both the can of soup and can of sauce in the basket | |
| SOOD 3: put away the alphabet soup and tomato sauce in the basket | |
| SOOD 4: place the can of soup in the basket and the can of sauce in the basket | |
| 1 | SOOD 1: pick up both the cream cheese box and the butter and place them in the basket |
| SOOD 2: put both the box of cheese and the box of butter in the basket | |
| SOOD 3: put away the cheese box and butter box in the basket | |
| SOOD 4: stash the cream cheese and butter | |
| 2 | SOOD 1: switch the stove on and set the moka pot on the stove |
| SOOD 2: turn on the cooktop and place the moka machine on it | |
| SOOD 3: put the moka pot over the heat | |
| SOOD 4: heat up the stove and put the moka pot on it | |
| 3 | SOOD 1: move the black bowl to the bottom drawer and close the drawer of the cabinet |
| SOOD 2: put the middle bowl to the lowest drawer and close it | |
| SOOD 3: put away the black bowl in the bottom drawer | |
| SOOD 4: stash away the middle bowl in the lower cabinet | |
| 4 | SOOD 1: place two mugs on the plates, left one is the white mug and the right one is the yellow and white mug |
| SOOD 2: put the pure white cup on the left plate and put the other one with the yellow handle on the right plate | |
| SOOD 3: serve the cups | |
| SOOD 4: serve the mugs | |
| 5 | SOOD 1: grab the standing book and transfer it to the back compartment of the caddy |
| SOOD 2: pick up the right book and put it in the rear part of the caddy | |
| SOOD 3: place the book in the back of the caddy | |
| SOOD 4: put away the book in the back part | |
| 6 | SOOD 1: set the white mug on the plate with chocolate pudding to be placed to the right |
| SOOD 2: put the pure white cup on the middle plate and put the brown chocolate to the right of the plate | |
| SOOD 3: place the white cup on the plate and the chocolate to the right | |
| SOOD 4: set the white mug on the plate and the chocolate pudding set on the right | |
| 7 | SOOD 1: put both the can of soup and the box of cheese in the basket |
| SOOD 2: move the two objects, alphabet soup and cream cheese box, to the basket | |
| SOOD 3: add the cream cheese box and the alphabet soup to the collection | |
| SOOD 4: grab the alphabet soup and the cream cheese box and throw them in the basket | |
| 8 | SOOD 1: transfer both moka pots from the table to the stove |
| SOOD 2: put both the moka coffee makers on the cooktop | |
| SOOD 3: put the coffee makers where they can heat up | |
| SOOD 4: heat up both moka pots | |
| 9 | SOOD 1: place the yellow and white mug inside the microwave, then shut the door |
| SOOD 2: put the middle mug inside the microwave and close the door | |
| SOOD 3: heat up the yellow and white mug by placing it and shutting the door | |
| SOOD 4: heat up the yellow and white mug in the microwave |
6.4 Benchmark Details
The appendix enumerates ten nominal tasks, forty paraphrases, thirteen behavior compositions, and six physical tasks. Some paraphrases omit relations or change references, making the exact tables important for interpreting the “semantic” stress test. Visual-Viewpoint is the language training scene; original LIBERO is a different scene for that policy but familiar to the underlying controller.
In the most shifted visual condition, uncalibrated success drops from the base’s 89.8% to 88.5%; calibration reaches 90.9%. Composition task thirteen improves from 36% to 88%, while task fifteen falls from 40% to 30%; tasks sixteen through twenty-two remain unsolved. The roughly 38% composition bar in Figure 4 matches averaging tasks ten through fifteen; including all thirteen Table 12 tasks gives about 17.7% for language steering. The narrative’s broad claim of success on most compositions is not supported by these full thirteen-task results.
For physical Microwave under a paraphrase, steering drops success from 66.7% to 46.7%, and calibration yields 70%. Novel MarkerBowl and ChipsCup reach 80% and 73.3% with uncalibrated steering; the table does not report calibrated scores for either. These details distinguish targeted transfer from broad calibrated generalization.
| Task | Instruction |
|---|---|
| 10 | put both the cream cheese box and the tomato sauce in the basket |
| 11 | put both the milk and the tomato sauce in the basket |
| 12 | put both the orange juice and the tomato sauce in the basket |
| 13 | put both the alphabet soup and the butter in the basket |
| 14 | put both the orange juice and the butter in the basket |
| 15 | put both the tomato sauce and the butter in the basket |
| 16 | put both the milk and the butter in the basket |
| 17 | put both the ketchup and the cream cheese box in the basket |
| 18 | put both the tomato sauce and the cream cheese box in the basket |
| 19 | put the wine bottle in the bottom drawer of the cabinet and close it |
| 20 | put the red mug on the plate and put the chocolate pudding to the right of the plate |
| 21 | put the red mug on the plate and put the chocolate pudding to the left of the plate |
| 22 | put the red mug on the left plate and put the yellow and white mug on the right plate |
| Task | Nominal | Semantic Perturbation |
|---|---|---|
| CubeSort | put the green cube in the blue basket and the red cube in the orange basket | put away the green cube and the red cube in similar color baskets |
| CubeMug | place the green cube in the blue mug then the blue mug in the orange basket | put away the cube in the mug and the mug in the bin |
| Microwave | place the orange bowl on top of the microwave and close the microwave door | put the bowl on top of the oven and close the door |
| MarkerBlock | place the marker and green block on the plate | put away the pen and green block in the bowl |
| MarkerBowl | put the marker in the blue bowl and put the blue bowl on the plate | — |
| ChipsCup | put the chips and green cup in the basket | — |
| Condition | ||||||
|---|---|---|---|---|---|---|
| Train Env | 51.2 | 60.6 | 53.1 | 55.0 | 63.9 | 63.8 |
| ID Scene, OOD Distractor | 46.7 | 50.7 | 45.7 | 47.3 | 55.0 | 54.2 |
| OOD Scene, ID Distractor | 92.4 | 88.2 | 92.4 | 89.1 | 92.5 | 95.0 |
| OOD Scene, OOD Distractor | 89.8 | 82.6 | 89.5 | 81.2 | 88.5 | 90.9 |
| Method | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 88.0 | 0.0 | 2.0 | 36.0 | 4.0 | 40.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| 94.0 | 2.0 | 10.0 | 10.0 | 0.0 | 46.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| 98.0 | 4.0 | 10.0 | 88.0 | 0.0 | 30.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Task | Condition | ||||
|---|---|---|---|---|---|
| CubeSort | Nominal | 56.7 | 80.0 | 93.3 | 90.0 |
| VOOD | 13.3 | 43.3 | 70.0 | 70.0 | |
| SOOD | 6.7 | 26.7 | 60.0 | 63.3 | |
| CubeMug | Nominal | 36.7 | 63.3 | 83.3 | 76.7 |
| VOOD | 30.0 | 40.0 | 66.7 | 73.3 | |
| SOOD | 16.7 | 36.7 | 70.0 | 70.0 | |
| Microwave | Nominal | 73.3 | 66.7 | 76.7 | 73.3 |
| VOOD | 46.7 | 16.7 | 50.0 | 50.0 | |
| SOOD | 66.7 | 40.0 | 46.7 | 70.0 | |
| MarkerBlock | Nominal | 33.3 | 80.0 | 90.0 | 90.0 |
| VOOD | 36.7 | 60.0 | 76.7 | 76.7 | |
| SOOD | 30.0 | 43.3 | 63.3 | 70.0 | |
| MarkerBowl | Nominal | 30.0 | 46.7 | 80.0 | — |
| ChipsCup | Nominal | 33.3 | 46.7 | 73.3 | — |
6.5 Additional Results
On the reported A6000 Ada platform, a replan grows from 68 ms for the controller alone to 274 ms when language generation adds 204 ms, reducing measured frequency from 14.7 to 3.6 Hz. Five environment steps at 20 Hz span 250 ms, so these timings do not by themselves establish a strict four-hertz real-time deadline. Caching, asynchronous updates, and distillation are proposed improvements.
Trajectory-length means include successful hardware trials only. Steering shortens several tasks, including ChipsCup from 870.8 to 562.2 steps, but lengthens MarkerBowl from 631.5 to 657.0; these are neither unconditional episode costs nor wall-clock savings. With ten, thirty, and fifty narration videos per task, final simulation success is 60.8%, 59.7%, and 63.9% versus the base’s 51.2%. This supports useful interaction-stage gains in this range, without eliminating narration or search cost.
| Config | (ms) | (ms) | Total (ms) | Replan Hz |
|---|---|---|---|---|
| only | — | |||
| Task | |||
|---|---|---|---|
| CubeSort | 709.5 | 587.5 | 627.4 |
| CubeMug | 680.9 | 578.3 | 542.1 |
| Microwave | 499.0 | 469.2 | 425.2 |
| MarkerBlock | 775.6 | 696.1 | 596.7 |
| MarkerBowl | 631.5 | 578.1 | 657.0 |
| ChipsCup | 870.8 | 728.4 | 562.2 |
| Model (Data) | (10) | (30) | (50) | |
|---|---|---|---|---|
| SR(%) | 51.2 1.0 | 60.8 1.0 | 59.7 1.0 | 63.9 1.0 |