Literature
Display
← All papers

Cross-Embodiment Robot Foundation World Models with Latent Actions

Huang Huang · Sriram Yenamandra · Arjun Majumdar · Elie Aljalbout · Tushar Nagarajan · Tsung-Yen Yang · Akshara Rai · Michael Rabbat · Li Fei-Fei · Jiajun Wu · Tingfan Wu · Franziska Meier

Preprint posted · arXiv (preprint) ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

At a glance

LAC-WM learns action representations from consecutive images, grounds them with motion-label reconstruction, and uses the resulting shared latent space to pretrain a robot world model across human and robot datasets. For a new embodiment, an action projector makes raw controls usable without future images. In simulated dexterous action selection, average success is about 22% versus 15% for the explicitly conditioned comparison, giving the reported 46.7% relative gain rather than 46.7 percentage points. Modified LIBERO success improves from 82.4% to 92.0%, or 9.6 points. The edition retains all nine figures, six tables, three displayed losses and the full appendix; hardware transfer remains untested.

1 Introduction

Robot embodiments differ in motion labels, morphology and camera viewpoints. Separate explicit-action encoders may occupy different regions of latent space, limiting what a shared forward model learns across datasets. The proposed alternative learns a common visual-transition representation, then grounds it in motion labels. It targets pretraining and adaptation of an action-conditioned predictor rather than a direct generalist policy. Main questions are whether image-derived latents align across embodiments and whether that alignment improves downstream planning after fine-tuning. Observed gains support this construction in the tested settings, while UMAP pictures and qualitative transfers alone cannot establish a universally embodiment-invariant action representation.

Refer to caption
Figure 1: LAC-WM Architecture. During pretraining (yellow area), the inverse dynamics model (IDM) encodes cross-embodiment image observations into a unified latent action space, as the condition for the forward dynamics model (FDM). A motion decoder translates the latent actions into motion labels, including delta human hand poses, robot end-effector actions, and camera motion. During finetuning (green area), an action projector encodes raw robot actions of unseen embodiments into the pre-trained unified latent action space.

2 Related Work

Prior robot world models often pretrain without actions or use restricted action spaces; latent-action work can extract controls from unlabeled video for later policies. This study instead uses labeled human/robot data to compare latent conditioning against direct motion encoding. It combines cross-view augmentation with an auxiliary decoder, following motion-supervised latent-action work, and adapts raw controls through a projector as in AdaWorld. Its distinct evaluation focuses on unseen-embodiment prediction and candidate-action ranking, rather than claiming that action labels are unavailable or unnecessary in the reported training.

3 Method

Pretraining jointly learns an inverse dynamics model, forward dynamics model and motion decoder. A pair of consecutive image embeddings produces a latent action; that latent conditions future-embedding prediction and must also reconstruct motion labels. For an unseen embodiment, a new raw-action projector and adaptation of the pretrained forward model permit imagined rollouts from executable controls.

3.1 Latent Action Conditioned Robot World Model

The input is a demonstration video with corresponding motion information. Continuous latent actions mediate between visual transitions, reconstruction of explicit motion and prediction of the next observation embedding.

Latent Action Modeling with an IDM and FDM

A frozen V-JEPA2 encoder turns images into visual tokens. The inverse model contextualizes consecutive tokens with causal attention and extracts action latents through learned-query cross-attention. The forward model alternates observation self-attention, action self-attention and cross-attention from observation to action tokens to predict the next visual embedding. At this stage the inverse model sees the true future frame, so its representation is a training-time inference mechanism rather than an executable test-time controller.

Cross-Augmentation Inputs

Independently augment the same pair of observations twice. The inverse model extracts a latent from one augmented pair; the forward model must use it with the other view to predict that view’s next embedding. The preserved squared reconstruction loss discourages directly copying future appearance through the action latent. It mitigates an identified shortcut without proving that all unwanted visual information has been eliminated.

Equation
ℒrecon=‖x^t+12−xt+12‖22,\mathcal{L}_{\mathrm{recon}}=\|\hat{x}_{t+1}^{2}-x_{t+1}^{2}\|_{2}^{2},

Motion Decoder

A motion decoder grounds continuous latents in known end-effector/hand and camera motion. The latent queries downsampled current-frame visual tokens, then an MLP predicts motion under a squared-error loss. Conditioning on the current image accommodates viewpoint-dependent labels. The full objective combines image reconstruction and motion losses with weights λrecon and λmotion. Unlike quantized or very small bottlenecks, this preserves latent capacity while providing action supervision. The inverse, forward and motion modules train jointly; adaptation retains the forward model and introduces a new action projector.

Equations
ℒmotion=‖𝐚^t−𝐚t‖22.\mathcal{L}_{\mathrm{motion}}=\|\hat{\mathbf{a}}_{t}-\mathbf{a}_{t}\|_{2}^{2}.
ℒ=λrecon​ℒrecon+λmotion​ℒmotion.\mathcal{L}=\lambda_{\mathrm{recon}}\mathcal{L}_{\mathrm{recon}}+\lambda_{\mathrm{motion}}\mathcal{L}_{\mathrm{motion}}.

3.2 Adapting to Unseen Robot Embodiments

A two-layer MLP maps the new robot’s raw controls into latent actions, removing the need for a future observation at inference. Adaptation has three stages: LoRA fine-tuning of inverse/forward models; projector training with the forward model frozen; then joint projector/forward fine-tuning. LoRA rank is 2. At deployment, only the projector and forward model generate action-conditioned predictions. Thus transfer still requires embodiment-specific data and adaptation, rather than zero-shot use of arbitrary controls.

4 Experiments

Experiments compare explicit and latent action conditioning under shared forward-model architecture, cross-embodiment pretraining and adaptation to an unseen simulated dexterous setup. Later tests change the visual tokenizer and evaluate modified LIBERO tasks.

Refer to caption
Figure 2: UMAP visualization of 7,000 action embeddings from three datasets: Droid (red), Agibot (green), and Egodex (blue). 1: Embeddings from separate Explicit Action Encoders (EAE), which are distinctly separated by embodiment. 2: Embeddings from a shared EAE (EAE*), still showing dataset-level separation. 3: Embeddings from IDM without the motion decoder, where Agibot and Egodex cluster together but separate from Droid. 4: Embeddings from the IDM, showing strong cross-dataset alignment.
Refer to caption
Figure 3: Cross-embodiment action transfer results: The left panels illustrate robot-to-human transfer, while the right panels show human-to-robot transfer. The first row presents reference videos. Rows 2–4 display FDM generated videos using embeddings from EAE, IDM without MD, and IDM, respectively. Blue dotted lines trace end-effector trajectories from the references and are aligned across generated videos except for initial positions. IDM embeddings yield trajectories that closely match the references, whereas the other methods exhibit inaccurate motion patterns and visual artifacts (highlighted by red circles).

4.1 Baseline

EAC-WM uses the same forward model but obtains conditioning by directly encoding motion labels. Each source embodiment normally has its own two-layer action encoder, because raw action spaces differ. A new encoder is trained for the target robot and the forward model is fine-tuned with rank 2 LoRA. A shared-encoder variant is also examined in the representation analysis. This is a specific explicit-conditioning baseline; its results do not show that every possible common explicit representation must fail.

4.2 Model Pretraining

Pretraining draws 50,000 trajectories each from EgoDex human manipulation, Agibot bimanual humanoid data and DROID’s single Franka/Robotiq arm. Labels include hand/end-effector motion and camera motion; views differ between egocentric and third-person recordings. Five-step action chunks reduce observation frequency. Both models train 80,000 iterations at batch 512 with 64-dimensional action embeddings and a frozen V-JEPA2 image tokenizer. A separately trained decoder reconstructs RGB for visualization/metrics. The source later varies both embodiment count and total data, so those scaling factors must be distinguished.

4.3 Model Finetuning

The target BFA setup has two Franka arms with Allegro hands, but these tasks use only the right arm. RoboCasa provides 7,265 successful/failed trajectories over 22 kitchen-object pick/place tasks, with egocentric camera images. Actions have 25 coordinates:16 finger joints, three end-effector translations and six rotation coordinates, grouped into five-step chunks. Both models receive 60,000 target training steps at batch 256. LAC-WM allocates 20,000/5,000/35,000 steps to its three adaptation stages; EAC-WM uses the stronger tested single-stage adaptation. This target is simulated and was not a source pretraining embodiment.

5 Latent Action Analysis

UMAP projects 7,000 validation embeddings from the three source datasets. Separate explicit encoders form dataset-specific clusters; sharing one explicit encoder still separates them. Inverse-model latents overlap more, especially with motion decoding. Without motion decoding, the two egocentric datasets align more than third-person DROID, suggesting visual-domain information remains. These projections are qualitative evidence, not a quantitative alignment guarantee.

Cross-embodiment transfer pairs one embodiment’s starting image with another’s eight-frame-derived actions. Latent conditioning reproduces analogous image-space hand/gripper motion; explicit encodings produce little motion, while removing motion supervision can create duplicated hands/floating objects. Figure 3 retains all 24 reference/generated frames and trajectory overlays. Appendix PCA adds representative clusters corresponding to opening/closing and left/right motion. Neither visualization establishes exact physical action equivalence across arbitrary viewpoints or morphologies.

6 Robot Task Evaluation

Target evaluation first measures eight-frame prediction fidelity, then asks whether predicted outcomes rank candidate controls effectively. The second test is essential because improved image metrics alone do not establish better robotic planning.

Eval Split Model PSNR↑\uparrow LPIPS↓\downarrow FID↓\downarrow FVD↓\downarrow
Unseen Instances EAC-WM-S 22.212 0.077 10.648 42.257
EAC-WM 24.252 0.057 9.312 30.349
LAC-WM 27.484 0.037 8.439 23.213
Unseen Categories EAC-WM-S 22.062 0.081 10.642 40.477
EAC-WM 24.091 0.060 9.144 29.702
LAC-WM 27.333 0.040 8.509 24.456
Table 1: Action conditioned imagined roll-out performance. Image and video quality metrics are reported across evaluation splits on unseen instances and unseen categories. LAC-WM performs the best compared to models trained from scratch (EAC-WM-S) or pretrained with explicit actions (EAC-WM).

6.1 Action-conditioned Imagined Roll-out

Held-out BFA splits contain unseen object instances from known categories or entirely unseen donut/lime/banana categories. Evaluate 512 randomly sampled windows per split, predicting eight future frames from the initial observation and raw actions. LAC-WM has the best PSNR/LPIPS/FID/FVD among pretrained explicit conditioning and an explicit model trained only on BFA. PSNR rises from 24.252 to 27.484 on unseen instances and 24.091 to 27.333 on unseen categories. Table 1 retains every metric; these are visual prediction results under supplied action sequences, separate from task completion.

6.2 Robot Planning with Action Selection

Planning assumes one demonstration trajectory per evaluation episode to supply intermediate goal images. A VLA trained for BFA samples 500 candidate 100-step actions; the world model predicts 20 five-step frames for each, ranks terminal embedding distance to the next goal and executes the mean of the best three complete action sequences. Episodes contain about 4–5 subgoals. VLA-random executes one sample and VLA-mean averages all 500. Each split uses 100 episodes with three random seeds; Table 2 reports mean±standard deviation. LAC-WM succeeds on 25% of unseen instances and 18% of unseen categories, averaging about 22%, versus 15% for EAC-WM,18% for target-only EAC-WM-S and 17% for VLA-mean. The reported 46.7%/22.2%/29.0% gains are relative to these rounded averages; absolute dexterous success remains low. Subgoal-distance comparisons with VLA-only methods are only reference measures because those policies do not receive the subgoal images.

Adding source embodiments initially also adds 50,000 trajectories each. Figure 5 separately holds total data fixed while adding an embodiment, then increases data at fixed embodiments. LAC-WM improves in both comparisons, whereas explicit conditioning degrades with heterogeneous sources and improves with more same-source data. This is a small controlled trend, not a general scaling law. Removing both motion decoding and cross-augmentation reduces unseen-category success from 18% to 13%; the combined ablation cannot isolate each component’s causal contribution.

A second experiment replaces V-JEPA with the Cosmos tokenizer and adapts on 2,200 LIBERO trajectories:2,000 standard demonstrations plus 200 perturbed π0.5 rollouts, including 90 failures. Five held-out object/location variants receive 25 trials each. The shared π0.5 backbone proposes 20 fifty-step chunks at noise scale 2; predict 40 steps, select one by final-goal distance and execute 25 before replanning. Each task uses a human demonstration for 40-step goal images. Mean success is 92.0% versus 82.4% EAC-WM,81.6% VLA-mean and 74.4% VLA-random. Thus the abstract’s 11.7% gain is relative; the absolute EAC-WM difference is 9.6 points. Table 4 retains all five task rates.

Val Set Model δ​f\delta f ↓\downarrow δ​fg\delta f_{g} ↓\downarrow S.R.C ↑\uparrow S.R.L ↑\uparrow S.R. ↑\uparrow
Unseen Instances VLA-mean 26.49±\pm0.06 27.07±\pm0.02 0.84±\pm0.04 0.56±\pm0.00 0.20±\pm0.03
VLA-random 26.52±\pm0.06 27.23±\pm0.12 0.82±\pm0.03 0.55±\pm0.03 0.18±\pm0.03
EAC-WM-S 26.74±\pm0.34 27.38±\pm0.35 0.84±\pm0.02 0.58±\pm0.04 0.19±\pm0.04
EAC-WM 26.79±\pm0.26 27.37±\pm0.27 0.84±\pm0.01 0.56±\pm0.04 0.16±\pm0.04
LAC-WM 26.29±\pm0.15 26.87±\pm0.13 0.86±\pm0.03 0.60±\pm0.03 0.25±\pm0.03
Unseen Categories VLA-mean 26.94±\pm0.09 27.86±\pm0.17 0.84±\pm0.04 0.49±\pm0.04 0.14±\pm0.03
VLA-random 26.93±\pm0.06 27.77±\pm0.07 0.80±\pm0.01 0.48±\pm0.01 0.12±\pm0.02
EAC-WM-S 27.16±\pm0.39 28.00±\pm0.41 0.84±\pm0.02 0.49±\pm0.07 0.17±\pm0.03
EAC-WM 27.19±\pm0.28 28.03±\pm0.27 0.82±\pm0.01 0.49±\pm0.04 0.14±\pm0.04
LAC-WM 26.73±\pm0.09 27.58±\pm0.11 0.85±\pm0.02 0.56±\pm0.02 0.18±\pm0.02
Average VLA-mean 26.72±\pm0.08 27.46±\pm0.09 0.84±\pm0.04 0.53±\pm0.02 0.17±\pm0.03
VLA-random 26.72±\pm0.06 27.50±\pm0.09 0.81±\pm0.02 0.52±\pm0.02 0.15±\pm0.02
EAC-WM-S 26.95±\pm0.37 27.69±\pm0.38 0.84±\pm0.02 0.54±\pm0.05 0.18±\pm0.03
EAC-WM 26.99±\pm0.27 27.70±\pm0.27 0.83±\pm0.01 0.52±\pm0.04 0.15±\pm0.04
LAC-WM 26.50±\pm0.12 27.22±\pm0.12 0.85±\pm0.03 0.58±\pm0.02 0.22±\pm0.02
Table 2: Robot planning performance of a VLA with action selection using a world model (EAC-WM-S, EAC-WM, LAC-WM) versus without action selection (VLA-mean, VLA-random). Mean and standard deviation across three seeds are shown.
Refer to caption
Figure 4: We show the image latent embedding distance δ​f\delta f (left) and δ​fg\delta f_{g} (middle) and task success rate S.R. (right), for LAC-WM (blue) and EAC-WM (orange) pre-trained with different number of embodiments. The x axis shows the number of embodiments and dataset size used in pre-training, from left to right of the x axis, the model is trained with Egodex only, Egodex and Agibot, and Egodex, Agibot and Droid dataset.
Refer to caption
Figure 5: Experiments of scaling number of embodiments or data samples. The first two points shows the scaling from two embodiments (Egodex and Droid) to three embodiments (Egodex, Droid and Agibot) with same amount of trajectories, while the last two points shows the scaling from 24k trajectories to 150k trajectories across three embodiments.
Model δ​f↓\delta f\!\downarrow δ​fg↓\delta f_{g}\!\downarrow S.R.C↑S.R.C\!\uparrow S.R.L↑S.R.L\!\uparrow 𝐒.𝐑.↑\mathbf{S.R.\!\uparrow}
LAC-WM-MD-CA 26.98 27.80 0.83 0.46 0.13
LAC-WM 26.73 27.58 0.85 0.56 0.18
Table 3: Ablation study of motion decoding loss and cross-augmentation inputs on the validation set of unseen categories.
Task VLA-random VLA-mean EAC-WM LAC-WM
book 0.76 0.76 0.80 0.92
white bowl 0.76 0.92 0.92 0.96
wine bottle 0.72 0.84 0.84 0.96
alphabet soup 0.80 0.72 0.80 0.88
salad dressing 0.68 0.84 0.76 0.88
Average 0.744 0.816 0.824 0.920
Table 4: Success rate on five unseen LIBERO tasks over 125 total episodes. All methods share the same π0.5\pi_{0.5} VLA backbone, sampling 20 candidate 50-step action chunks.

7 Conclusion and Limitations

Learning latent transitions jointly with motion reconstruction improves the tested cross-embodiment predictor and action selector. The reported setup intentionally uses motion-labeled sources for a fair explicit-conditioning comparison; possible benefit from large unlabeled-video pretraining is future work, not demonstrated here. Dexterous success remains limited, candidate quality and planning could improve, and physical hardware validation remains outstanding. Adaptation requires target data, and evaluation relies on demonstration subgoals and sampled action priors.

Impact Statement

The authors state that the work aims to advance robot world-model learning and do not identify a specific societal consequence requiring separate emphasis.

Appendix A Appendix

The appendix records exact source action conventions, target data generation, RGB decoding, model sizes/training cost, a second candidate-action protocol and the VLA/PCA details. All supplementary figures and tables remain available here.

A.1 Pretraining Dataset Details

Wrist and camera poses use three translations plus six rotation coordinates. Agibot contributes 20 end-effector dimensions,10 per hand including gripper; DROID contributes 10 for one arm. EgoDex uses 69 per hand—nine wrist plus 60 finger-keypoint coordinates—for 138 bimanual dimensions. Camera motion adds nine dimensions for egocentric datasets and is zero for fixed-camera DROID. Explicit conditioning concatenates end-effector and camera labels. The 64-dimensional latent is split evenly for decoding these two targets. This heterogeneous labeling is part of what the shared latent representation must reconcile.

Refer to caption
Figure 6: Example of action conditioned imagined roll-out from the world models. The top row displays the ground truth video, while the subsequent rows show action-conditioned imagined rollouts from EAC-WM-S, EAC-WM, and LAC-WM. While LAC-WM accurately predicts future observations, both EAC-WM-S and EAC-WM exhibit overshooting robot actions, highlighted by red circles, where the robot arm moves excessively. This suggests that LAC-WM better captures the robot dynamics.

A.2 BFA Dataset

MimicGen/RoboCasa expand ten teleoperated trajectories into 7,265 successful and failed trials over 22 object categories. Demonstrators lift an object and place it at least 0.1 m left or right; about 20% of generated trajectories succeed. The platform has two Franka arms, but evaluation uses right-arm manipulation. This world-model dataset differs from the 2,000 successful demonstrations used to train the VLA candidate generator.

A.3 Image decoder

The custom RGB decoder first trains on DROID, then adapts to BFA using image MSE. For 256² images,16×16 V-JEPA2 tokens of dimension 1,408 project to width 1,024, pass through 12 attention layers with 32 heads and map back to pixels. Decoding supports qualitative videos and image metrics; planning itself ranks visual embeddings, so decoder quality and planning distance are distinct parts of the pipeline.

A.4 Model Training and Inference

Table 5 retains component sizes: roughly 46.94M inverse-model parameters,93.84M forward-model parameters,1.55M projector parameters and 153.85M RGB-decoder parameters, in addition to the 1B visual tokenizer. The first 60,000 pretraining steps disable forward temporal attention to encourage action conditioning; the last 20,000 enable an eight-step temporal horizon. Training takes about four days on 64 H200 GPUs. Reported inference is 0.1 s for a batch of ten eight-frame predictions at 256² on an L40S; it is not the latency of ranking 500 twenty-step candidates and executing a robot plan.

Hyperparameter Value
Image Tokenizer V-JEPA 2 (1B)
Action Projector 1,550,528
Image Decoder 153,845,952
IDM:
# Attention Blocks 4
# Attention Heads 16
# Latent Dimension 512
# Parameter 46,939,968
FDM:
# Attention Blocks 8
# Attention Heads 16
# Latent Dimension 512
# Parameter 93,844,480
Table 5: Hyperparameters for LAC-WM model architecture.
Refer to caption
Figure 7: Examples of action selection rollout. The leftmost image shows the initial observation, and the rightmost image depicts the goal state. Numbers in the top-left corner indicate time steps. In this example, three action sequences are sampled and rolled out by the world model, producing eight future predictions. The action sequence whose predicted final image has the smallest embedding distance (δ​fg\delta f_{g}) to the goal image is selected for execution. Here, the action sequence in the top row is chosen.

A.5 Action Selection from Training Dataset

An alternate experiment samples 500 action candidates from the training dataset, predicts six five-step frames, averages the top five and executes all 30 controls. Fifty episodes per split, each repeated with three seeds, use 12–15 subgoals. Table 6 retains single/two/three-stage adaptation variants and all intermediate contact/lift/distance metrics. LAC-WM reaches 12% unseen-instance and 10% unseen-category success, versus 11%/8% for EAC-WM and 11%/3% for target-only training. Lower absolute success than VLA-guided sampling emphasizes dependence on the candidate prior.

The full three-stage procedure is strongest overall, but the source’s LAC-WM-(1,2) prose describes a final joint projector/forward update despite its stage label; that ambiguity is retained rather than resolved by assumption. Some Table 6 aggregate entries are inconsistent with the displayed splits: for example LAC-WM’s mean δf is 24.50 although the split values are 24.60 and 25.39. The table is preserved unchanged and this aggregate is not used as a corrected numerical claim.

Val Set Model δ​f\delta f ↓\downarrow δ​fg\delta f_{g} ↓\downarrow S.R.C↑\uparrow S.R.L ↑\uparrow S.R↑\uparrow
Unseen Instance EAC-WM-S 25.00±\pm0.04 26.53±\pm0.06 0.59±\pm0.03 0.22±\pm0.02 0.11±\pm0.02
EAC-WM-TFT 25.05±\pm0.01 26.49±\pm0.10 0.54±\pm0.03 0.15±\pm0.06 0.09±\pm0.02
EAC-WM 25.03±\pm0.04 26.53±\pm0.12 0.61±\pm0.03 0.21±\pm0.02 0.11±\pm0.01
LAC-WM-DFT 24.82±\pm0.08 26.32±\pm0.20 0.61±\pm0.03 0.19±\pm0.02 0.07±\pm0.01
LAC-WM-(2,3) 24.88±\pm0.04 26.51±\pm0.08 0.54±\pm0.09 0.21±\pm0.01 0.09±\pm0.01
LAC-WM-(1,2) 24.90±\pm0.09 26.37±\pm0.12 0.59±\pm0.07 0.21±\pm0.05 0.09±\pm0.06
LAC-WM 24.60±\pm0.07 26.16±\pm0.19 0.65±\pm0.06 0.27±\pm0.06 0.12±\pm0.02
Unseen Categories EAC-WM-S 25.75±\pm0.04 27.46±\pm0.07 0.56±\pm0.02 0.08±\pm0.03 0.03±\pm0.03
EAC-WM-TFT 25.68±\pm0.06 27.39±\pm0.11 0.54±\pm0.00 0.13±\pm0.05 0.07±\pm0.01
EAC-WM 25.68±\pm0.15 27.50±\pm0.09 0.54±\pm0.07 0.16±\pm0.04 0.08±\pm0.02
LAC-WM-DFT 25.57±\pm0.04 27.35±\pm0.05 0.53±\pm0.08 0.19±\pm0.01 0.09±\pm0.02
LAC-WM-(2,3) 25.78±\pm0.18 27.48±\pm0.28 0.51±\pm0.09 0.21±\pm0.05 0.06±\pm0.04
LAC-WM-(1,2) 25.65±\pm0.09 27.43±\pm0.09 0.48±\pm0.07 0.09±\pm0.01 0.03±\pm0.02
LAC-WM 25.39±\pm0.02 27.21±\pm0.09 0.61±\pm0.01 0.18±\pm0.03 0.10±\pm0.02
Average EAC-WM-S 25.38±\pm0.04 26.99±\pm0.06 0.55±\pm0.03 0.15±\pm0.03 0.07±\pm0.03
EAC-WM-TFT 25.36±\pm0.04 26.94±\pm0.11 0.54±\pm0.02 0.14±\pm0.06 0.08±\pm0.02
EAC-WM 25.36±\pm0.10 27.01±\pm0.10 0.58±\pm0.05 0.18±\pm0.03 0.09±\pm0.02
LAC-WM-DFT 25.19±\pm0.06 26.83±\pm0.12 0.57±\pm0.05 0.19±\pm0.02 0.08±\pm0.02
LAC-WM-(2,3) 25.33±\pm0.11 26.99±\pm0.18 0.52±\pm0.09 0.21±\pm0.03 0.07±\pm0.03
LAC-WM-(1,2) 25.27±\pm0.09 26.90±\pm0.11 0.54±\pm0.07 0.15±\pm0.03 0.06±\pm0.04
LAC-WM 24.50±\pm0.04 26.68±\pm0.14 0.61±\pm0.03 0.22±\pm0.05 0.11±\pm0.02
Table 6: Robot planning performance for different methods using action selection from the training dataset.
Refer to caption
Figure 8: Example roll-out of executing actions from a pre-trained VLA. From top to bottom, the figure shows rollouts for: executing the mean of all sampled actions (VLA-mean), a random sampled action (VLA-random), and actions selected using EAC-WM-S, EAC-WM, and LAC-WM from the VLA samples. The task is to pick up the object and place it back on the counter. While LAC-WM successfully selects actions to complete the task, the other methods fail to grasp and lift the object.

A.6 Vision-Language Action Model for Dexterous Manipulation

The BFA candidate VLA combines frozen DINOv2 images and pretrained language embeddings with learned proprioception/action encoders, a three-layer transformer trunk and a diffusion action head. It predicts 100 future 25-dimensional end-effector/finger commands at 15 Hz, executing the full chunk before replanning, and trains on 2,000 successful demonstrations. The main text also calls candidate generation flow matching, while this appendix calls it a diffusion head/loss; the reading edition does not infer an unreported implementation reconciliation. This task-specific prior is distinct from the LIBERO π0.5 experiment.

A.7 Latent Action PCA Analysis

PCA projects four-step latent-action sequences from all three source datasets onto two components, then K-means forms four clusters. Representative sequences nearest each centroid show start/end images across embodiments, illustrating gripper opening/closing and left/right motion. Figure 9 retains the complete projection, arrows and all representative panels. The analysis gives qualitative semantic examples and does not quantify transfer accuracy or guarantee globally consistent latent coordinates.

Refer to caption
Figure 9: Latent Action PCA Analysis. The representative examples from each of the cluster show that latent action in each cluster corresponds to semantically meaningful actions (e.g. opening and closing gripper, moving left and right) across different embodiments.
Concept demonstration

Open image ↗