Overview
LIBERO-CF probes whether a robot follows an alternative feasible instruction in a familiar scene, and CAG strengthens the action difference induced by language.
At a glance
Vision–language–action policies can default to a scene’s familiar training task despite an instruction specifying another object or goal. LIBERO-CF tests these alternatives, while Counterfactual Action Guidance extrapolates between language-conditioned and vision-only action chunks. For π0.5, the training-free variant raises average grounding from 30.8% to 40.5% and success from 13.2% to 16.8%; a separately trained vision-only branch reaches 46.3% and 21.7%. Improvements are substantial but far from reliable general instruction following, and individual tasks can regress.
1 Introduction
A scene associated with one demonstrated behavior can become a shortcut for task selection. Empty or alternative instructions often leave that behavior intact, motivating evaluation that changes the requested task while holding the layout fixed. CAG attempts to reveal capabilities already present in the base model by strengthening language-sensitive action differences. It changes neither the conditioned model architecture nor its weights, although the stronger vision-only-prior variant requires separate training.
2 Related Work
Existing VLA work develops model architectures, action representations, and training recipes, while multimodal work studies imbalance between information sources. This paper adds controlled alternative-task evaluation and inference-time guidance inspired by classifier-free guidance [20]. The action-space procedure is compatible with different policy outputs, but that architectural compatibility does not imply equal effectiveness across models.
3 Problem Formulation
The formulation separates the action preference suggested by a scene from the influence of the requested instruction, then tests whether alternative feasible instructions actually change behavior.
3.1 Vision Bias in Datasets and Architectures
Scene-specific demonstrations typically cover few of the feasible tasks, so an object arrangement may predict the training behavior without language. The authors also propose visual-token dominance as a contributor. Dataset bias is supported by the behavioral probes; token counts and architecture imbalance are plausible explanations rather than isolated causal findings.
3.2 Problem Formulation from Bayesian Perspective
Bayes’ rule motivates viewing an instruction-conditioned action distribution as a vision-conditioned prior reweighted by instruction compatibility. Well-observed tasks and under-observed alternatives can coexist in the same scene. The evaluation changes the intended task, not merely the wording of an unchanged task. This probabilistic framing motivates the method but does not derive its final deterministic action extrapolation as an exact posterior computation.
Equation 1
| (1) |
4 The LIBERO-CF Benchmark
Built on LIBERO [36] layouts, the benchmark contains fifteen spatial tasks, ten alternative-object tasks, ten long-horizon tasks, and fifteen tasks involving objects unseen during finetuning. Each method receives fifty trials per task. The reported main average weights the four suites equally rather than weighting every task equally.
Grounding measures contact with the instructed target, while success requires completing the task. Contact is a useful proxy but does not establish full goal understanding, order, or termination. Faithful and biased measures respectively assess the requested and original training tasks; they need not be complementary exclusive outcomes, as a rollout can interact with multiple objects.
4.1 Validating Vision Shortcuts of VLAs
Removing language leaves substantial standard-LIBERO success: 69.1% for OpenVLA-OFT, 18.5% for π0, and 51.7% for π0.5, versus 97.1%, 94.2%, and 96.9% with language. Goal tasks lose more because the same scene supports different goals. Grasp heatmaps also show attraction to training-task objects under alternative or empty instructions. Removing those objects in CF-Focused raises mean grounding from 39.0% to 64.4% and success from 20.0% to 40.0%. This controlled scene intervention supports the shortcut interpretation while also changing scene complexity and available interaction targets.
| Model | Input | Spatial | Object | Goal | Long | Avg. |
| OpenVLA -OFT | V+L | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 |
| V-only | 83.2 | 98.2 | 10.0 | 85.0 | 69.1 | |
| V+L | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 | |
| V-only | 28.0 | 26.8 | 3.4 | 15.6 | 18.5 | |
| V+L | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 | |
| V-only | 68.2 | 56.6 | 10.2 | 71.6 | 51.7 |
5 Method
CAG combines conditional and vision-only action predictions at inference time. Two implementations differ in how the vision-only branch is obtained.
5.1 Counterfactual Action Guidance
Compute an action chunk from each branch, subtract the vision-only prediction from the conditioned one, then scale that difference and add it back to the vision-only action. Scale one reproduces the conditioned policy, zero yields the vision-only policy, and values above one extrapolate beyond the conditioned action. Mixing happens after each policy finishes generating its final chunk, rather than inside diffusion or flow steps. This is a classifier-free-guidance-inspired heuristic in action space, with no guarantee that extrapolated actions remain feasible or improve completion.
Equation 2
| (2) |
5.2 Dual-branch Inference with CAG
TF-CAG runs the existing VLA with and without language, requiring no additional training. VA-CAG instead trains a separate vision-action policy without language and pairs it with the unchanged conditioned model. Both reuse existing demonstrations; only TF-CAG is training-free. A dropped-language VLA is an approximation to a vision-only prior, and the separately trained branch can provide a different, sometimes more useful correction.
| Model | Metric | CAG | CF-Spatial | CF-Object | CF-Long | CF-OOD | Average | |||||
| F. | B. | F. | B. | F. | B. | F. | B. | F. | B. | |||
| OpenVLA-OFT [27] | Grounding | / | 6.8 | 84.7 | 2.0 | 99.6 | 4.0 | 62.9 | 6.0 | 87.3 | 4.7 | 83.6 |
| TF | 11.1 | 78.5 | 1.6 | 100.0 | 4.9 | 61.1 | 5.6 | 88.1 | 5.8 | 81.9 | ||
| VA | 26.5 | 59.5 | 3.8 | 91.8 | 6.9 | 60.0 | 7.9 | 50.1 | 11.3 | 65.4 | ||
| Success | / | 1.1 | 80.7 | 0.0 | 97.6 | 0.2 | 54.4 | 0.1 | 81.7 | 0.4 | 78.6 | |
| TF | 3.3 | 73.3 | 0.0 | 94.4 | 0.2 | 54.0 | 0.1 | 83.5 | 0.9 | 76.3 | ||
| VA | 7.9 | 35.2 | 0.0 | 57.0 | 0.4 | 48.9 | 0.1 | 22.7 | 2.1 | 41.0 | ||
| [4] | Grounding | / | 70.9 | 37.1 | 6.0 | 59.4 | 20.4 | 83.1 | 18.0 | 72.4 | 28.8 | 63.0 |
| TF | 74.5 | 32.5 | 12.2 | 50.6 | 19.6 | 81.8 | 21.1 | 72.4 | 31.9 | 59.3 | ||
| VA | 85.2 | 24.4 | 13.8 | 53.8 | 22.7 | 82.2 | 27.6 | 47.5 | 37.3 | 52.0 | ||
| Success | / | 34.4 | 19.1 | 0.0 | 45.0 | 1.1 | 63.1 | 2.7 | 52.9 | 9.6 | 45.0 | |
| TF | 33.5 | 14.4 | 0.6 | 15.0 | 2.0 | 38.7 | 2.1 | 42.3 | 9.6 | 27.6 | ||
| VA | 36.7 | 6.1 | 0.0 | 29.2 | 2.9 | 48.7 | 2.1 | 19.2 | 10.4 | 25.8 | ||
| [21] | Grounding | / | 39.3 | 61.3 | 11.4 | 68.8 | 51.6 | 58.4 | 20.7 | 74.0 | 30.8 | 65.6 |
| TF | 52.0 | 45.3 | 24.6 | 65.2 | 60.7 | 56.7 | 24.8 | 71.9 | 40.5 | 59.8 | ||
| VA | 50.7 | 45.2 | 34.0 | 49.8 | 64.2 | 61.1 | 36.4 | 52.8 | 46.3 | 52.2 | ||
| Success | / | 24.4 | 56.9 | 5.8 | 66.8 | 15.8 | 50.4 | 6.9 | 69.6 | 13.2 | 60.9 | |
| TF | 27.3 | 33.3 | 9.2 | 41.4 | 20.9 | 36.7 | 9.9 | 62.7 | 16.8 | 43.5 | ||
| VA | 31.6 | 36.0 | 18.0 | 29.8 | 26.7 | 41.8 | 10.3 | 37.1 | 21.7 | 36.2 | ||
6 Simulation Experiments
Simulation evaluates requested-task contact and completion alongside persistence of the original training behavior, then studies training strategy and guidance strength.
6.1 Evaluating Counterfactual Behaviors
Standard benchmark performance masks poor counterfactual results. OpenVLA-OFT averages 4.7% faithful grounding and 0.4% faithful success, while still achieving 83.6% biased grounding and 78.6% biased success. π0.5 averages 30.8%/13.2% faithful grounding/success, compared with 65.6%/60.9% on the original task.
For π0.5, TF-CAG reaches 40.5% grounding and 16.8% success; VA-CAG reaches 46.3% and 21.7%, with biased success falling to 36.2%. The gains are percentage points, not relative percentages. On unseen-object tasks, VA-CAG reaches 36.4% grounding but only 10.3% success. Results are not uniformly improved: π0’s training-free mean success stays at 9.6%, and some suite-specific grounding or success rates decrease. OpenVLA-OFT remains especially limited even with guidance.
6.2 Ablation Studies
Language-dropout training performs poorly in the tested π0.5 ablation, while a separately trained vision-only branch gives stronger results than simply dropping language at inference. Low-rank and full finetuning provide similar grounding; full finetuning improves spatial success from 27.7% to 31.6%, a 3.9-point difference in Table 4 rather than the prose’s 4.3-point statement. Increasing guidance can improve target selection but eventually damages manipulation. The chosen scales are 1.5 for π0/π0.5 and 3.0 for OpenVLA-OFT; the paper does not establish an adaptive or universally optimal setting.
7 Real-world Experiments
Physical experiments create a deliberately skewed demonstration distribution, with one common task and two alternative tasks in each scene.


7.1 Implementation Details
A Franka Research 3 with a Robotiq 2F-85 gripper, external and wrist cameras follows the DROID [26] setup. Each in-domain instruction has twenty demonstrations; under-observed alternatives receive one each, while unseen-object tasks receive none. Both the π0.5 conditioned policy and the separate vision-only branch are fully finetuned for 10,000 steps with batch size 32. Evaluation randomizes object positions and uses ten trials per instruction. Thus most alternative physical tasks are few-shot, with zero-shot claims confined to the designated unseen objects.
7.2 Results
Six main scenes test appearance-only object choice, different geometries, spatial references, placement goals, and unseen objects. Across these scenes, the reported average gains are 9.4 percentage points in grounding and 17.2 in success. Object-recognition scenes gain 13.3 success points each, and the goal-targeting scene gains 36.7 points. These averages include the familiar instruction as well as the two alternatives.
The full breakdown makes the limits clearer: grounding the right object can still fail manipulation, and per-task improvements are not universal. For example, the well-observed tape task falls from ten to nine successes. Each condition has only ten trials, so differences of one trial are ten percentage points; repeated-seed variation or uncertainty intervals are not reported.
7.3 Analysis
Adding zero, one, or two demonstrations for four alternative tasks generally helps, and guidance often adds benefits at the same data budget. It cannot create an absent skill: with zero alternative demonstrations, Left, Right, and Basket stay at zero successes with or without CAG, while Plate improves from three to eight of ten. Benefits are not monotonic for every individual task and data count.
On the stated deployment, a fifteen-step chunk takes 80.0±2.2 ms for π0.5, 126.6±4.4 ms for TF-CAG, and 132.6±1.9 ms for VA-CAG. Memory is 8.5, 8.5, and 16.5 GB respectively. These are chunk-generation latencies, not robot control-step frequencies. VA training is reported as about three hours for LIBERO and one hour for physical data on eight H100 GPUs.
8 Conclusion
CAG can reduce scene-driven default behavior when the base policy already contains useful perception and motor capabilities. It does not supply missing skills, guarantee valid extrapolated actions, or remove the cost of two inference branches; VA-CAG also requires a second trained model. Broader environments, embodiments, long-horizon tests, and adaptive uncertainty-based guidance remain future directions.
Acknowledgments
The acknowledgments credit reviewer and area-chair feedback.
Appendix
The appendix explains practical motivation, adds X-VLA results and training details, and supplies physical task definitions, longer sequences, rollout figures, and complete outcome tables.
Appendix A Practical Significance
The intended setting is adaptation from a small scene-specific demonstration set followed by deployment on new goals or objects. Familiar-layout success alone can conceal failure to use the instruction. The benchmark therefore tests whether pretraining-derived capabilities survive this biased adaptation; it does not assume that every arbitrary unseen task is solvable.
Appendix B Extended Simulation Experiments
Additional simulation material specifies baseline checkpoints, vision-only training, and a fourth policy architecture.
B.1 Additional Details in the Main Paper
Official LIBERO checkpoints are used for the principal models, with fifty trials per task. Vision-only models use the corresponding LIBERO training recipes with language removed. The stronger improvement for π0.5 than for OpenVLA-OFT indicates that the conditional–unconditional action difference depends on the underlying model. These comparisons do not isolate whether architecture, data, or optimization causes that difference.
| Model | Metric | CAG | CF-Spatial | CF-Object | CF-Long | CF-OOD | Avg. | |||||
| F. | B. | F. | B. | F. | B. | F. | B. | F. | B. | |||
| X-VLA [60] | Grounding | / | 22.1 | 72.1 | 1.4 | 70.4 | 98.2 | 23.3 | 27.3 | 58.4 | 37.3 | 56.1 |
| TF | 22.3 | 68.9 | 0.0 | 77.6 | 95.6 | 12.2 | 40.9 | 63.7 | 39.7 | 55.6 | ||
| VA | 24.4 | 68.8 | 2.0 | 59.6 | 96.7 | 17.6 | 40.8 | 60.5 | 41.0 | 51.6 | ||
| Success | / | 3.2 | 58.9 | 0.0 | 50.2 | 47.6 | 9.1 | 4.4 | 38.8 | 13.8 | 39.3 | |
| TF | 8.9 | 62.3 | 0.0 | 56.6 | 59.1 | 3.8 | 10.1 | 47.9 | 19.5 | 42.6 | ||
| VA | 8.7 | 53.7 | 0.0 | 31.6 | 51.1 | 0.9 | 8.7 | 38.1 | 17.1 | 31.1 | ||
B.2 Supplementary Results
For X-VLA, TF-CAG raises average grounding from 37.3% to 39.7% and success from 13.8% to 19.5%; VA-CAG reaches 41.0% and 17.1%. Thus the separately trained prior improves grounding more but has lower faithful success than the training-free variant. TF-CAG also raises biased success from 39.3% to 42.6%, contrary to the appendix prose’s claim that it reduces it. The original table is retained, and these outcomes show that faithful contact, faithful completion, and suppression of training behavior can move differently.
B.3 Implementation Details of VLA Baselines
OpenVLA-OFT uses continuous action chunks and L1 regression; its vision-only version uses empty instructions while preserving prompt structure and low-rank finetuning for 300,000 steps at batch size 64. π0 and π0.5 vision-only branches use full finetuning for 30,000 steps at batch size 32, learning rate 0.00005, and exponential-moving-average decay 0.999. X-VLA is finetuned in absolute end-effector action space for 50,000 iterations at batch size 16, with mixed precision and scheduled backbone freezing. Unlike the broad checkpoint statement, this X-VLA baseline is itself finetuned from a released base model.
Appendix C Extended Real-world Experiments
The physical appendix covers eight scenes in total, adding two long-horizon scenes to the six in the main experiments.
C.1 Details of Our Real-world Datasets
Task definitions specify one well-observed instruction and two alternatives per scene. Demonstrations are collected through teleoperation, usually twenty for the familiar task and one for each alternative; the designated unseen objects remain the stated zero-shot exception. Keeping that distinction prevents conflating under-observed tasks with entirely unseen supervision.

| Dimension | Scene | In-domain Task | Under-observed Tasks |
| Object Recognition | Coke / Sprite / Fanta | Pick up the coke | Pick up the sprite; pick up the fanta |
| Tape / Mustard / Pringles | Pick up the tape | Pick up the mustard; pick up the pringles | |
| Spatial Reasoning | Middle / Left / Right | Pick up the cup in the middle | Pick up the cup on the left; pick up the cup on the right |
| Table / Plate / Bowl | Pick up the corn can on the table | Pick up the corn can on the plate; pick up the corn can in the bowl | |
| Goal Targeting | Stack / Plate / Basket | Stack the cups | Put the cup on the plate; put the cup in the basket |
| OOD Generalization | Cup / Cube / Basketball | Pick up the cup | Pick up the Rubik’s Cube; pick up the basketball |
| Long-Horizon Reasoning | Move and Pour | Move the cup to the right and pour the coke into the cup | Move the cup to the right and pour the fanta into the cup; move the cup to the right and pour the sprite into the cup |
| Apple and Banana | Put the apple on the tray and then put the banana on the tray | Put the banana on the tray and then put the apple on the tray; put the apple on the tray |
C.2 Additional Experiments for Long-Horizon Reasoning
Move-and-Pour changes which drink is used after moving a cup. Apple-and-Banana tests reversing a familiar two-object order and stopping after only the apple. These expose target-choice errors late in a sequence, ordering errors, and unwanted continuation of a learned routine.
Guidance helps some outcomes but does not improve every long-horizon variant as the prose claims. The apple-only task rises from two to seven successes in ten trials, while reversed-order success falls from five to four and the familiar two-object sequence falls from nine to eight. In Move-and-Pour, Sprite and Fanta success remain five and three respectively. The full tables therefore support selective gains in instruction adherence, especially stopping, rather than uniformly better long-horizon completion.
C.3 Detailed Results
Each table row is an instruction and each outcome column records which task or object the robot attempted. A cell reports successful executions divided by attempts in that column; overall success and grounding still use all ten trials as denominator. For example, six successes in nine Fanta attempts represent 60% success and 90% grounding, not a 6/9 overall success rate. The Fail column records idle or unclear behavior. All rollout panels and outcome tables are retained to show both corrected behavior and residual failures; some coarse prose descriptions differ from these detailed tabulations.
| Coke / Sprite / Fanta | |||||
| Model | Instruction | Coke | Sprite | Fanta | Fail |
| “Coke” | 8/10 | ||||
| “Sprite” | 6/10 | ||||
| “Fanta” | 1/1 | 6/9 | |||
| + CAG | “Coke” | 9/10 | |||
| “Sprite” | 8/10 | ||||
| “Fanta” | 7/10 | ||||
| Tape / Mustard / Pringles | |||||
| Model | Instruction | Tape | Mustard | Pringles | Fail |
| “Tape” | 10/10 | ||||
| “Mustard” | 3/3 | 3/6 | 0/1 | ||
| “Pringles” | 1/2 | 2/8 | |||
| + CAG | “Tape” | 9/10 | |||
| “Mustard” | 6/10 | ||||
| “Pringles” | 4/10 | ||||
| Middle / Left / Right | |||||
| Model | Instruction | Middle | Left | Right | Fail |
| “Middle” | 9/10 | ||||
| “Left” | 8/8 | 1/2 | |||
| “Right” | 7/7 | 3/3 | |||
| + CAG | “Middle” | 10/10 | |||
| “Left” | 6/7 | 3/3 | |||
| “Right” | 4/5 | 1/1 | 4/4 | ||
| Table / Plate / Bowl | |||||
| Model | Instruction | Table | Plate | Bowl | Fail |
| “Table” | 7/9 | 1/1 | |||
| “Plate” | 6/8 | 2/2 | |||
| “Bowl” | 4/5 | 4/5 | |||
| + CAG | “Table” | 8/10 | |||
| “Plate” | 7/7 | 3/3 | |||
| “Bowl” | 3/4 | 6/6 | |||
| Stack / Plate / Basket | |||||
| Model | Instruction | Stack | Plate | Basket | Fail |
| “Stack” | 5/9 | 1/1 | |||
| “Plate” | 0/1 | 2/7 | 2 | ||
| “Basket” | 2/10 | ||||
| + CAG | “Stack” | 7/10 | |||
| “Plate” | 7/10 | ||||
| “Basket” | 6/9 | 1 | |||
| Cup / Cube / Basketball | |||||
| Model | Instruction | Cup | Cube | Basketball | Fail |
| “Cup” | 9/10 | ||||
| “Cube” | 0/1 | 5/6 | 3/3 | ||
| “Basketball” | 3/3 | 6/7 | |||
| + CAG | “Cup” | 9/10 | |||
| “Cube” | 8/9 | 1/1 | |||
| “Basketball” | 1/2 | 7/8 | |||
| Move and Pour | |||||
| Model | Instruction | Coke | Sprite | Fanta | Fail |
| “Coke” | 5/9 | 0/1 | |||
| “Sprite” | 1/2 | 5/8 | |||
| “Fanta” | 2/3 | 3/7 | |||
| + CAG | “Coke” | 6/9 | 1/1 | ||
| “Sprite” | 5/10 | ||||
| “Fanta” | 1/1 | 3/9 | |||
| Apple and Banana | |||||
| Model | Instruction | A+B | B+A | A | Fail |
| “Apple+Banana” | 9/10 | ||||
| “Banana+Apple” | 4/4 | 5/6 | |||
| “Apple” | 8/8 | 2/2 | |||
| + CAG | “Apple+Banana” | 8/10 | |||
| “Banana+Apple” | 2/3 | 4/7 | |||
| “Apple” | 3/3 | 7/7 | |||