Overview
This edition covers arXiv v3, dated 23 October 2026, including the complete appendix. All eight original figures, seven numbered tables and eight equations are retained. The central distinction is between a diagnostic correlation and the measured benefit of adding a training objective.
At a glance
Action-only fine-tuning can reduce the task-semantic structure of pretrained robot-policy features. The authors probe this erosion, then add training-only alignment to a frozen encoder through shared/private feature channels. Across tested backbones, success improves without changing the deployed inference graph. On the real bimanual setup, average in-distribution success rises 18.7 percentage points and average out-of-distribution success rises 21.5 points. The feature split has no identifiability guarantee.
1 Introduction
Narrow robot demonstrations can encourage shortcuts while weakening pretrained semantic organization. A mirror-neuron analogy motivates measuring whether action features retain task-level meaning, using an external encoder as reference. The proposed intervention aligns an auxiliary shared feature channel while a private channel and reconstruction objective retain other detail. The neuroscience analogy motivates the design; it does not establish that the learned channels reproduce biological mechanisms.
2 Diagnostic: Action–Instruction Alignment Erosion
The diagnostic follows π₀ fine-tuning and asks whether action–instruction alignment tracks generalization and episode outcome. Lightweight probes measure association rather than directly identifying a causal mechanism.
2.1 Probing Setup
Layer-10 action-token states from the 18-layer π₀ backbone are mean-pooled. Frozen Qwen3-VL-Embedding text features provide an external semantic reference. Two projection heads train with bidirectional InfoNCE using eight tasks per LIBERO suite, then evaluate on two held-out tasks per suite. Bidirectional Recall@1 measures learned retrieval alignment; both source representations stay frozen during probing.
Equation 1
| (1) |
2.2 Alignment Dynamics over Fine-Tuning
Seven independently refitted probes cover pretraining through 30,000 steps. Alignment starts at 62.74%, falls, partially recovers and declines again, while in-distribution success generally rises. Across the six trained checkpoints, alignment and LIBERO-Pro success have Spearman correlation 0.943. The small longitudinal sample supports association; it does not prove that alignment loss causes all observed generalization failure.
2.3 Per-Trajectory Alignment and Rollout Outcome
At the final checkpoint, successful rollouts have higher average action–instruction cosine similarity and retrieval accuracy than failures. This motivates preserving semantics during training, but the outcome-conditioned comparison alone cannot distinguish better representation from easier trajectories or other shared causes.
3 Method
The policy keeps its native action objective and gains an auxiliary training branch at an intermediate action representation. Attention pooling, a shared/private split and contrastive instruction alignment supply additional supervision.
3.1 Problem Formulation
The native objective is token cross-entropy for autoregressive policies or flow-matching regression for continuous policies. The auxiliary target is the frozen text encoder from EgoHOD, which was pretrained with manipulation video. This differs from the Qwen encoder used in diagnostics. Alignment gradients enter through action-token features at the selected layer.
Equation 2
| (2) |
3.2 Anchoring Action Features to the Semantic Manifold
A learned query pools action tokens, then projection heads map action and instruction features into a common space. Task identifiers define all same-task pairs as positives; other tasks in the batch are negatives. A pairwise logistic contrastive loss shapes the shared action channel. Its fixed temperature/bias differ from the learnable temperature of the diagnostic InfoNCE probe.
Equations 3–4
| (3) |
| (4) |
3.3 Shared–Private Decomposition
Two encoders map the pooled feature into shared and private vectors; a decoder reconstructs the feature from their sum. Only the shared vector enters semantic alignment. Reconstruction and squared-cosine decorrelation discourage information loss and channel overlap. The reconstruction loss combines mean squared error and a positive squared mean-residual penalty. These constraints encourage a useful split but do not uniquely identify pure semantics or execution detail.
Equations 5–7
| (5) |
| (6) |
| (7) |
3.4 Training Objective
Training adds alignment, reconstruction and decorrelation to action loss, with reconstruction weight 0.001 and decorrelation weight 0.15. Alignment weight depends on the backbone. No extra robot demonstrations are added, though the frozen target carries external pretraining. All pooling, split, decoder and projection modules are removed for deployment, leaving the baseline inference architecture.
Equation 8
| (8) |
4 Experiments
Experiments test simulation transfer, bimanual physical tasks and architectural/design variations. The real-robot comparison is controlled against a multitask π₀ baseline; per-task ACT and Diffusion Policy comparisons have different conditioning and training organization.
4.1 Simulation Experiments
Simulation includes near-distribution LIBERO, real-to-simulation transfer in SimplerEnv and perturbed LIBERO-Pro evaluation. The intervention is added to each backbone’s standard recipe.
Setup
LIBERO provides 50 demonstrations per task. SimplerEnv trains on BridgeData V2 and evaluates WidowX under Visual Matching. π₀ is fully fine-tuned, SpatialVLA uses low-rank adapters, and GR00T N1.6 tunes its projector/action head while keeping its vision-language backbone frozen. All attach alignment at layer 10 and discard auxiliary modules at inference.
| Method | spoon | carrot | stack | eggplant | Avg |
|---|---|---|---|---|---|
| Octo [10] | 47.2 | 9.7 | 4.2 | 56.9 | 29.5 |
| -FAST [33] | 29.1 | 21.9 | 10.8 | 66.6 | 32.1 |
| [4] | 36.1 | 33.3 | 23.6 | 51.4 | 36.1 |
| + Ours | 48.6 | 37.5 | 31.9 | 56.9 | 43.87.7 |
| SpatialVLA [11] | 23.6 | 29.2 | 23.6 | 95.8 | 43.1 |
| SpatialVLA + Ours | 36.1 | 40.3 | 27.8 | 97.2 | 50.37.2 |
| GR00T N1.6 [9] | 84.7 | 50.0 | 5.6 | 98.6 | 59.7 |
| GR00T N1.6 + Ours | 87.5 | 51.4 | 23.6 | 98.6 | 65.35.6 |
Simulation Results
On LIBERO, π₀ average success rises from 88.5% to 90.8%. SimplerEnv averages rise from 36.1% to 43.8% for π₀, 43.1% to 50.3% for SpatialVLA and 59.7% to 65.3% for GR00T N1.6. Gains are 2.3, 7.7, 7.2 and 5.6 percentage points respectively. Three-seed variation is reported for these comparisons; perturbation results appear in Appendix B.
4.2 Real-Robot Experiments
Physical evaluation uses four task families with two variants each and five generalization axes derived from fruit pick-and-place. The claimed average gains should be distinguished from each family’s individual improvement.
Setup
A Cobot Magic V2 uses two six-joint arms, grippers and three RGB cameras. Eight variants contribute 200 demonstrations each to one multitask π₀ policy. Every evaluated condition has 20 trials. ACT and Diffusion Policy train separately per task because their multitask versions did not converge in this setup; their lack of language conditioning excludes language and task-composition evaluations.
Results
Average in-distribution success increases from 51.3% to 70.0%, while the five-axis generalization average increases from 49.5% to 71.0%. Fruit, stacking, packing and cabinet success all improve. Generalization gains are +10 points for language, +25 position, +25 object, +17.5 visual and +30 task composition. These limited-trial results support the tested setup, with no real-robot training-seed variability reported.
| In-Distribution | Out-of-Distribution | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Fruit Pick&Place | Dish Stacking | Box Packing | Cabinet Storage | Avg | Orig. | Lang. | Pos. | Obj. | Vis. | Task | Avg |
| ACT | 82.5 | 45.0 | 25.0 | 47.5 | 50.0 | 82.5 | – | 7.5 | 40.0 | 45.0 | – | – |
| DP | 80.0 | 47.5 | 32.5 | 42.5 | 50.6 | 80.0 | – | 12.5 | 32.5 | 35.0 | – | – |
| 75.0 | 42.5 | 37.5 | 50.0 | 51.3 | 75.0 | 75.0 | 32.5 | 52.5 | 57.5 | 30.0 | 49.5 | |
| + Ours | 92.5 | 67.5 | 52.5 | 67.5 | 70.0 | 92.5 | 85.0 | 57.5 | 77.5 | 75.0 | 60.0 | 71.0 |
4.3 Ablation and Analysis
Ablations change one component at a time for SpatialVLA on SimplerEnv. They test whether gains depend on decomposition, encoder choice and attachment depth.
Method Components
Contrastive alignment alone raises average success from 43.1% to 48.0%; adding the shared/private structure raises it to 50.3%. This supports both components under this protocol, without proving that their internal channels uniquely encode the intended factors.
Alignment Target
Self-clustering reaches 46.8%; language-only Qwen reaches 45.8%, CLIP-Large 46.9%, Qwen3-VL 49.0% and EgoHOD 50.3%. Manipulation-video grounding is associated with the best result. Encoder training and representation differences remain bundled, so the comparison supports the selected target rather than a universal claim that domain pretraining always dominates scale.
Alignment Layer
Attachment at layer 10 peaks at 50.3%, with layer 15 close at 49.0%; tested early and later layers give smaller gains. The authors use layer 10 for all backbones. This sweep is on SpatialVLA, so it does not separately optimize every architecture’s depth.
5 Related Work
The method combines representation regularization for robot policies with frozen-encoder alignment and shared/private decomposition. Its distinguishing choice is a training-only action-token semantic target.
Vision-Language-Action Models
Existing policy work scales data, changes action architectures or adds auxiliary objectives to reduce shortcuts and feature distortion. Semantic anchoring instead preserves a measured aspect of action-feature organization while retaining the deployed model architecture. Training still gains auxiliary computation and a pretrained target.
Representation Alignment
Representation alignment for diffusion models motivates frozen-encoder supervision. EgoHOD supplies a manipulation-centric target, while Domain Separation Networks motivate shared/private channels. Related robot methods regularize visual, spatial or self-feature pathways; this method aligns action-token representations to an external semantic space.
6 Conclusion, Limitations and Future Directions
The tested intervention improves success while leaving inference unchanged. Limits include dependence on target-encoder quality, lack of identifiable shared/private factors, task-level positives that ignore graded cross-task similarity and token-level meaning, and evaluation only during downstream fine-tuning. Extending alignment to policy pretraining and broader data is future work. Diagnostic correlation and visualization remain evidence about a proxy, not a complete definition of action-representation quality.
Acknowledgments
The authors acknowledge Chinese national and Zhejiang provincial research funding and computing resources from Eastern Institute of Technology, Ningbo, and Ningbo Institute of Digital Twin.
Appendix A Implementation Details
The implementation appendix records backbone-specific optimization, auxiliary architecture, independent diagnostic probes, physical hardware and every task/evaluation definition.
A.1 Backbone Training Hyperparameters
π₀ uses full fine-tuning for 30,000 LIBERO steps, 40,000 BridgeData steps and 30,000 physical-task steps. SpatialVLA uses rank-32 adapters for 10,000 steps. GR00T N1.6 trains the projector and diffusion action head for 15,000 steps with the vision-language backbone frozen. All use AdamW, gradient-norm clipping at 1 and bfloat16. The original table retains batch sizes, learning rates, horizons and hardware counts.
| (LIBERO) | (BridgeV2) | SpatialVLA (BridgeV2) | GR00T N1.6 (BridgeV2) | (Real) | |
| Fine-tuning | Full | Full | LoRA () | Projector + DiT head | Full |
| Backbone LR | |||||
| Training steps | 30k | 40k | 10k | 15k | 30k |
| Batch size | 32 | 128 | 192 | 1024 | 64 |
| 0.9, 0.95 | 0.9, 0.95 | 0.9, 0.999 | 0.9, 0.999 | 0.9, 0.95 | |
| Weight decay | 0 | ||||
| Action horizon | 50 | 5 | 4 | 8 | 50 |
| VLM frozen | – | – | LoRA only | Frozen | – |
| GPUs | 2 | 2 | 2 | 4 | 2 |
| Subspace dim. | 512 | 512 | 1024 | 512 | 512 |
A.2 Alignment Module
Eight-head learned-query pooling feeds equal-dimensional shared/private encoders and an additive reconstruction decoder. Subspace size is 512 for π₀/GR00T and 1,024 for SpatialVLA; projection space is 512 and EgoHOD text features are 768-dimensional. Alignment weight is 0.1 for π₀ and 1.0 for the others. Training contrastive temperature 0.1 and bias 1.5 are fixed; auxiliary learning rates and weights remain in the full implementation details/table.
A.3 Diagnostic Probing Protocol
Diagnostics deliberately use Qwen3-VL rather than the training target EgoHOD. Frozen layer-10 action features are mean-pooled and projected to 512 dimensions. Each of seven checkpoint probes trains anew for 1,000 steps at learning rate 10⁻⁴ on task-disjoint splits, then reports bidirectional Recall@1. Probe fitting is separate from policy training.
A.4 Real-Robot Platform
The robot has 14 controllable dimensions including two grippers and three 256×256 RGB views. Actions are per-dimension-normalized joint positions. Eight task variants yield about 1,600 teleoperated demonstrations, shared by the multitask π₀ policies. Each evaluated condition uses 20 trials.
Baseline Training
ACT trains per task for 100,000 steps with batch 64, a 512-wide transformer, eight heads, four encoder/seven decoder layers and horizon 50. Diffusion Policy trains for 30,000 steps with batch 16 and 14-dimensional actions. Both use learning rate 10⁻⁴ and the same 200 demonstrations for each task; multitask convergence failure explains the baseline protocol difference.
A.5 Real-Robot Task Descriptions
The complete task table specifies banana/grape placement, cup/plate stacking, grape/bear box packing with both flaps closed, and cup/sponge cabinet storage with the door closed. Success requires stable placement or the full placement-and-closure sequence. All eight original demonstration strips are retained.
| Family | Variant | Task Description | Success Criterion |
|---|---|---|---|
| Fruit Pick-and-Place | Banana | Pick up the banana and place it on the plate. | Fruit rests stably on the plate. |
| Grapes | Pick up the grapes and place it on the plate. | Fruit rests stably on the plate. | |
| Dish Stacking | Cup | Stack one cup on top of the other. | Two cups form a stable upright stack on the table. |
| Plate | Stack the plate onto the other plate. | Plate rests stably on the bottom plate. | |
| Box Packing | Grapes | Place the grapes into the box and close both side flaps. | Object inside the box and both side flaps fully closed. |
| Bear | Place the toy bear into the box and close both side flaps. | Object inside the box and both side flaps fully closed. | |
| Cabinet Storage | Cup | Place the cup in the cabinet and close the door. | Object inside the cabinet and door closed. |
| Sponge | Place the sponge in the cabinet and close the door. | Object inside the cabinet and door closed. |
A.6 OOD Evaluation Axes
Generalization tests paraphrased instructions, positions beyond the training range, unseen object attributes/categories, new backgrounds/distractors and unseen combinations of familiar source/target objects. These axes are evaluated on the pick-and-place family, not across every long-horizon task.
| Axis | Description |
|---|---|
| Language | The task instruction is paraphrased into semantically equivalent but lexically different expressions not seen during training. |
| Position | Objects are placed at positions beyond the spatial range seen during training. |
| Object | Both in-category and out-of-category perturbations are applied. In-category: the same object type with different attributes (e.g., plates of different shapes, grapes of different sizes). Out-of-category: entirely novel objects not present in any training demonstration (e.g., mango, strawberry). |
| Visual | The scene background is changed by placing unseen tablecloths, and distractor objects not present during training are added to the workspace. |
| Task | The policy must execute unseen source–target combinations of objects seen during training. |
Appendix B Additional Analyses
Additional analyses report perturbation results, training-seed variability, auxiliary-loss ablations, an alternative regularizer, interpretation of interference and an example of task-composition behavior.
B.1 LIBERO-Pro Results
The same π₀ checkpoint improves LIBERO-Pro average from 36.0% to 40.5% without further training. Language, swap, composition, environment and replacement all improve, but swap remains 2.0% and composition 15.2%. Average gains do not remove severe weaknesses under these shifts.
| Method | lan | swap | task | env | object | Avg |
|---|---|---|---|---|---|---|
| 61.0 | 1.2 | 6.8 | 44.8 | 66.0 | 36.0 | |
| + Ours | 65.0 | 2.0 | 15.2 | 48.2 | 72.2 | 40.5 |
B.2 Multi-Seed Evaluation
Three training seeds give LIBERO π₀ 88.5±1.5% versus 90.8±1.2%. SimplerEnv comparisons are 36.1±3.9 versus 43.8±3.1 for π₀, 43.1±3.7 versus 50.3±2.6 for SpatialVLA, and 59.7±3.2 versus 65.3±2.2 for GR00T. These are standard deviations, not confidence intervals. Rollout counts are 20 per LIBERO task and 24 per SimplerEnv task.
B.3 DSN Loss-Term Ablation
Removing reconstruction reduces SpatialVLA success from 50.3% to 47.9%; removing decorrelation reduces it to 49.0%. Both terms contribute under the tested settings. This ablation measures policy performance rather than directly verifying semantic/execution disentanglement.
B.4 Comparison with VICReg
Replacing only the contrastive objective with VICReg raises the 43.1% baseline to 45.2%, compared with 50.3% for the proposed loss. The authors explicitly acknowledge limited VICReg hyperparameter exploration, so this is not a definitive ranking of the regularizers.
B.5 Relation to Task Interference
The original diagnosis concerns one action objective distorting pretrained features. The proposed method then introduces two potentially competing demands—semantic invariance and action sensitivity—and uses the feature split to reduce their conflict. This differs from earlier work insulating a language backbone from action-expert gradients.
B.6 Qualitative Comparisons
The retained example shows the baseline choosing an unsuitable arm and irrelevant object for an unseen source–target combination, while the aligned policy selects the intended object/arm. It illustrates the task-axis result; a selected rollout cannot establish its frequency or isolate why the intervention helped.