Literature
Display
← All papers

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

AgiBot Research Team

Preprint posted · arXiv (preprint) ↗

World models15 min text · CondensedOriginal paper ↗
Contents

At a glance

GE-Act 2.0 predicts visual futures in one generator pass, then converts them into robot actions. Separate video and inverse-dynamics pretraining are joined by KASO, which selects generated futures compatible with the recorded behavior. Expanding co-training from 300 to 30,000 hours raises real-robot success from 17.1% to 44.1% on G1-OP and 13.4% to 31.1% on G2-90D, without evaluation-task fine-tuning. Every scale shares the same much larger pretrained components, and compute is not matched. Grounding is strong for several direct attributes but weak for ordinal instructions.

Introduction

A visual planner can learn from instruction-labeled video without robot actions, while inverse dynamics can learn from action-labeled trajectories without language or success labels. GE-Act 2.0 connects these paths through completed predicted futures. The central training problem is that a plausible future can depict a different valid behavior from the demonstration whose action is used as its label. [31] [54] [64]

GE-Act 2.0 data sources, pretraining paths, and scaling overview
Figure 1: GE-Act 2.0 pretrains its visual planner on video without action labels and its inverse dynamics model on trajectories without language instructions, then co-trains the full system on fully labeled robot demonstrations. Scaling co-training data from 300 to 30,000 hours improves zero-shot OOD success across a 100-task real-robot suite. Each tile in the right panel is one task, coloured by skill group and shaded by success rate; the curve is the mean over all tasks.

Contributions.

The system combines compact control-oriented visual latents, a differentiable one-pass visual predictor, and selective co-training. Its main evaluation asks what the resulting checkpoint can do before task adaptation, across 100 tasks and two robot embodiments.

Related Work

The related work places the design among generalist policies, visual representations for control, fast generative models, and methods that connect predicted futures to actions.

Generalist Policies and World-Action Models

Earlier systems directly predict actions, jointly generate video and actions, or derive actions from a generator’s internal features. Here a completed future is the explicit interface to a separately pretrained inverse dynamics model, differing from GE-Act 1.0’s parallel action branch. [1] [2] [31] [44] [45]

Representations for Robot Video and Control

Prior representations emphasize reconstruction, latent motion, or predictive features. CoAE combines a pixel decoder with alignment to pretrained visual teachers, then measures usefulness through action recovery and instruction matching under a fixed probe architecture. [5] [8] [9] [61] [62] [65] [69]

Fast Video Generation for World Models

The visual predictor builds on MeanFlow and improved MeanFlow, which learn interval-averaged velocity for one-pass generation. The contribution applies this mechanism to conditional multi-view robot futures and makes the completed result differentiable for action training. [12] [15] [16] [32] [47] [50] [67] [68]

Connecting Generated Futures to Actions

Training inverse dynamics on recorded futures leaves an input shift when deployment uses generated ones. Even successful generation does not resolve a different mismatch: a sampled future and recorded action may represent different choices. KASO ranks candidates by the current inverse model’s action response. A concurrent generative-modeling method also uses best-of-K matching. [6] [17] [30] [63] [72]

Evaluating Pretrained Robot Models

Task-specific fine-tuning mixes pretrained ability with adaptation-data coverage. The main real-robot protocol therefore uses final pretrained checkpoints directly; simulator comparisons include their prescribed adaptation and answer a different question. [24] [34] [48] [70]

GE-Act 2.0

The following sections define the visual representation, one-pass planner, inverse dynamics model, and compatibility-based joint training.

System Overview

CoAE compresses each camera frame. A scene-conditioned SVP predicts dense near-term and sparse farther-future latents, and the IDM combines them with the current views and proprioception to produce actions. The controller executes a near-term chunk; sparse actions provide auxiliary training targets. The generator and IDM start randomly, but CoAE transfers compatible DC-AE weights and uses frozen teachers; a pretrained frozen VLM supplies language conditioning. Thus the complete system is not learned without pretrained components.

Refer to caption
Figure 2: GE-Act 2.0 system architecture. The SVP grounds the instruction in the current observation and fully denoises multi-view future visual latents in a single flow-generator pass; the IDM combines current and predicted latents with proprioception to produce dense and sparse action sequences.

A Control-Oriented Autoencoder for World–Action Models

The visual representation must be small enough for frequent prediction while retaining changes that identify the robot’s motion and the instruction’s referents. Reconstruction alone does not directly measure these properties.

Beyond Reconstruction with Multi-Teacher Alignment

CoAE downsamples each spatial axis by 64: a 256×384 frame becomes 24 tokens with 512 channels. Compatible weights come from the 128-channel DC-AE, with new parameters initialized randomly. Pixel, perceptual, and adversarial reconstruction losses are combined with alignment to frozen SigLIP 2, V-JEPA 2.1, and DINOv3 features. [5] [41] [49] [52]

Equations 1–2
ℒrec=‖o^−o‖22+λperc​ℒperc​(o^,o)+λgan​ℒgan​(o^).\mathcal{L}_{\mathrm{rec}}=\bigl\|\hat{o}-o\bigr\|_{2}^{2}+\lambda_{\mathrm{perc}}\,\mathcal{L}_{\mathrm{perc}}\bigl(\hat{o},o\bigr)+\lambda_{\mathrm{gan}}\,\mathcal{L}_{\mathrm{gan}}\bigl(\hat{o}\bigr). (1)
ℒalign=∑k=13λk​(1−cos⁡(gk​(z),Gk​(o))).\mathcal{L}_{\mathrm{align}}=\sum_{k=1}^{3}\lambda_{k}\left(1-\cos\bigl(g_{k}(z),G_{k}(o)\bigr)\right). (2)
Original figure
Figure 3: CoAE architecture and training objectives. At the 256×384256{\times}384 frame resolution used by the SVP, the encoder maps one input frame to a 4×64{\times}6 grid of 512-channel latents, giving 24 tokens per frame. A reconstruction decoder provides pixel-level supervision, while three alignment heads match language-aligned, spatiotemporal, and dense visual features from frozen visual teachers.

Control and Instruction Grounding under Extreme Compression

With frozen encoders and the same two-layer action probe, CoAE reaches MAE 0.01673 on 100 held-out samples from a single shelf task, after training on 400 samples. DINOv3 and V-JEPA 2.1 are better, but use 16 times as many tokens. A separate 5,000-episode caption-matching probe reaches 97.95% accuracy with CoAE; its eight-layer probe retains VLM conditioning. These probe results support compactness and accessibility of information in the tested settings, rather than proving general control superiority or isolating each teacher’s contribution.

Table 1: Controlled probes of five frozen visual representations. Spatial downsampling is reported as the height ×\times width factor relative to the input; for example, 64×6464{\times}64 means that one latent location spans a 64×6464{\times}64-pixel input region. Channels gives the latent feature dimension. Lower is better for both probe metrics.
Representation Spatial downsampling Channels Pixel decoder Action MAE ↓\downarrow Caption-match error (%) ↓\downarrow
CoAE 64×6464{\times}64 512 ✓ 0.01673 2.05
DC-AE 64×6464{\times}64 128 ✓ 0.02573 2.71
DINOv3 16×1616{\times}16 1024 ✗ 0.01273 2.75
V-JEPA 2.1 16×1616{\times}16 1024 ✗ 0.01487 4.15
SigLIP 32×3232{\times}32 2048 ✗ 0.03138 3.28
Original figure
Figure 4: Action-recovery and instruction-grounding probes for five frozen visual representations. Lower is better on both axes. For each 256×384256{\times}384 input frame, labels report tokens per frame ×\times channel dimension; marker area scales with tokens per frame.

A Single-Step Visual Planner as a World–Action Interface

The SVP combines a frozen multimodal understanding module with a randomly initialized conditional flow transformer. It predicts complete future latents in one pass, allowing the inverse model to consume the same kind of input during joint training and deployment.

One-Step Generation Unlocks Underutilized Data

Separate IDM training can exploit failed attempts, play, and deployment rollouts when observations and actions are aligned. Joining it to a multi-step generator makes action backpropagation expensive. A one-pass generator reduces that cost, enabling gradients through the generated future while preserving the separate pretraining paths.

Multi-Scale Visual Prediction

Four dense future frames cover the next 52 control frames, about 1.73 seconds at 30 Hz; two sparse frames extend toward the instruction segment’s end. This spends temporal resolution near execution while retaining a farther goal signal. Sparse action targets are auxiliary and are not the executed chunk. [74]

Equation 3
ℱ={f0}∪{f0+j​HNd}j=1Nd⏟dense: control∪{f0+H+⌊k​T−HNs⌉}k=1Ns⏟sparse: intent.\mathcal{F}\;=\;\{\,f_{0}\,\}\;\cup\;\underbrace{\Bigl\{\,f_{0}+j\,\tfrac{H}{N_{d}}\,\Bigr\}_{j=1}^{N_{d}}}_{\text{dense: control}}\;\cup\;\underbrace{\Bigl\{\,f_{0}+H+\Bigl\lfloor k\,\tfrac{T-H}{N_{s}}\Bigr\rceil\Bigr\}_{k=1}^{N_{s}}}_{\text{sparse: intent}}. (3)

Toward Unified Understanding and Generation

Frozen Qwen3.5 jointly processes the current head-camera image and instruction. A learned gate combines text-span states from its layers, retaining scene context with a text-length conditioning sequence. Every visual-generator block attends to this result. The IDM receives no language directly; instructions influence it through the predicted visual future. [46]

Equations 4–5
(h0,h1,…,hL)=VLM⁡(ohead,c),hℓ∈ℝS×d.\bigl(h_{0},h_{1},\ldots,h_{L}\bigr)=\mathrm{VLM}\bigl(o_{\mathrm{head}},c\bigr),\qquad h_{\ell}\in\mathbb{R}^{S\times d}. (4)
αℓ=softmaxℓ⁡(g⁡(Pool⁡(hℓ​[𝒯]))),cvlm=LN⁡(∑ℓ=0Lαℓ​hℓ​[𝒯]).\alpha_{\ell}=\operatorname{softmax}_{\ell}\!\left(g\!\left(\operatorname{Pool}\bigl(h_{\ell}[\mathcal{T}]\bigr)\right)\right),\qquad c_{\mathrm{vlm}}=\operatorname{LN}\!\left(\sum_{\ell=0}^{L}\alpha_{\ell}h_{\ell}[\mathcal{T}]\right). (5)

Conditional MeanFlow for One-Step Visual Planning

Conditional MeanFlow learns an average velocity that maps noise to future visual latents in one evaluation. An auxiliary instantaneous-velocity head supplies the tangent for a Jacobian–vector product. Masks keep observed-frame coordinates fixed and supervise future coordinates only; the detached derivative correction stabilizes learning. The full equations retain the endpoint conventions and masked losses.

Equations 6–14
u(zt,r,t):⁣=1t−r∫rtv(zτ,τ)dτ,u(z_{t},r,t)\mathrel{:=}\frac{1}{t-r}\int_{r}^{t}v(z_{\tau},\tau)\,\mathrm{d}\tau, (6)
u⁡(zt,r,t)=v⁡(zt,t)−(t−r)​dd​t​u​(zt,r,t),u(z_{t},r,t)=v(z_{t},t)-(t-r)\frac{\mathrm{d}}{\mathrm{d}t}u(z_{t},r,t), (7)
dd​t​uθ=∂zuθ⋅d​ztd​t+∂tuθ.\frac{\mathrm{d}}{\mathrm{d}t}u_{\theta}=\partial_{z}u_{\theta}\cdot\frac{\mathrm{d}z_{t}}{\mathrm{d}t}+\partial_{t}u_{\theta}. (8)
Vθ=uθ+(t−r)​sg⁡[d​uθd​t],V_{\theta}=u_{\theta}+(t-r)\,\operatorname{sg}\!\left[\frac{\mathrm{d}u_{\theta}}{\mathrm{d}t}\right], (9)
mi={0,conditioning frame,1,future frame to predict.m_{i}=\begin{cases}0,&\text{conditioning frame},\\[2.0pt] 1,&\text{future frame to predict}.\end{cases} (10)
zt(i)=(1−mi)​z(i)+mi​((1−t)​z(i)+t​ϵ(i)),vc(i)=mi​(ϵ(i)−z(i)).z_{t}^{(i)}=(1-m_{i})z^{(i)}+m_{i}\bigl((1-t)z^{(i)}+t\epsilon^{(i)}\bigr),\qquad v_{c}^{(i)}=m_{i}\bigl(\epsilon^{(i)}-z^{(i)}\bigr). (11)
d​ztd​t=m⊙vθ​(zt,t),(ri,ti)=(mi​r,mi​t),\frac{\mathrm{d}z_{t}}{\mathrm{d}t}=m\odot v_{\theta}(z_{t},t),\qquad(r_{i},t_{i})=(m_{i}r,m_{i}t), (12)
ℒu=‖m⊙(Vθ−vc)‖2,ℒv=‖m⊙(vθ−vc)‖2.\mathcal{L}_{u}=\bigl\|m\odot(V_{\theta}-v_{c})\bigr\|^{2},\qquad\mathcal{L}_{v}=\bigl\|m\odot(v_{\theta}-v_{c})\bigr\|^{2}. (13)
z^(i)=z1(i)−mi​uθ(i)​(z1,r=0,t=1).\hat{z}^{(i)}=z_{1}^{(i)}-m_{i}\,u_{\theta}^{(i)}\bigl(z_{1},r{=}0,t{=}1\bigr). (14)

World-Action Alignment and KASO

Joint training must adapt the inverse model to generated inputs while avoiding contradictory supervision and loss of separately pretrained capabilities.

Inverse Dynamics Model

The IDM uses flow matching to predict action sequences from current and future visual latents plus proprioception. Recorded futures supervise standalone pretraining; generated futures supervise the connected branch during co-training. Its action sampler remains multi-step even though visual generation is one-pass.

Equation 15
ℓFM​(a,o,s,z)=𝔼t,ε​‖vϕ​(at,t,o,s,z)−(ε−a)‖2.\ell_{\mathrm{FM}}\bigl(a\,;\,o,\,s,\,z\bigr)\;=\;\mathbb{E}_{t,\varepsilon}\Bigl\|v_{\phi}\bigl(a_{t},t;\,o,s,z\bigr)-\bigl(\varepsilon-a\bigr)\Bigr\|^{2}. (15)

Knowledge Transfer with Conflicting Supervision

Plain E2E training propagates action loss through the generated future into both models. E2E+PT additionally retains recorded-video and recorded-action pretraining losses. Those losses help protect prior capabilities but do not ensure that a generated future agrees with the recorded action’s behavior. [10]

Equations 16–17
z^=Gθ​(o,c,ξ),ℒE2E=ℓFM​(a,o,s,z^).\hat{z}=G_{\theta}(o,c,\xi),\qquad\mathcal{L}_{\mathrm{E2E}}=\ell_{\mathrm{FM}}(a;o,s,\hat{z}). (16)
ℒE2E+PT=ℒE2E+ℒSVP+ℒIDM.\mathcal{L}_{\mathrm{E2E+PT}}=\mathcal{L}_{\mathrm{E2E}}+\mathcal{L}_{\mathrm{SVP}}+\mathcal{L}_{\mathrm{IDM}}. (17)
Original figure
Figure 5: Comparison of Decoupled Pretraining, End-to-End Co-training, E2E + Pretraining Losses, and KASO. Decoupled Pretraining is the pretraining stage itself: the SVP and the IDM are trained on their own objectives and never exchange generated futures. The four strategies differ in whether the IDM receives generated future visual states, whether the pretraining losses for the SVP and IDM remain active, and whether generated candidates are filtered for action compatibility.

The Validity Gap

A demonstration supplies a coupled future–action pair. Sampling a new future independently can select another valid grasp, route, or timing while retaining the original action target. The resulting validity gap persists even when the generator matches the true marginal distribution of plausible futures.

Equation 18
(z,a)\displaystyle(z,a) ∼pdata(z,a∣o,s,c),\displaystyle\sim p_{\mathrm{data}}(z,a\mid o,s,c), recorded in the same rollout,\displaystyle\text{recorded in the same rollout},
z^\displaystyle\hat{z} ∼pθ(z∣o,c),a∼pdata(a∣o,s,c),z^⟂a∣(o,s,c),\displaystyle\sim p_{\theta}(z\mid o,c),\qquad a\sim p_{\mathrm{data}}(a\mid o,s,c),\qquad\hat{z}\perp a\mid(o,s,c), paired during co-training.\displaystyle\text{paired during co-training}. (18)

A Controlled Toy Two-Stage System.

The toy study maps four visual modes to two scalar action modes. A deterministic inverse regressor can average contradictory action labels toward zero, while generated states outside the recorded plane expose an intentionally amplified sensitivity. Selection preserves the displayed modes in the seed-0 example. This is a controlled illustration with substantial variation across seeds, not an averaged benchmark or a full replica of the robot model.

Refer to caption
Figure 6: Video samples projected onto the z1z_{1}–z2z_{2} plane (top, with ground-truth contours overlaid) and induced action densities (bottom) learned by each strategy on the toy system. The displayed run uses seed 0 and single-sample evaluation for every learned strategy; seed-to-seed variation is discussed in Appendix B.7. Ground truth contains four video modes and two action modes; in the displayed run, Decoupled Pretraining preserves the four video modes but yields a diffuse action density; End-to-End Co-training collapses both the video and action distributions; with the deterministic IDM, E2E + Pretraining Losses preserves the video modes but collapses the action density to the mean of the two action modes; and KASO recovers the four video modes and the bimodal action structure.

KASO: Knowledge-Aligned Selective Optimization

For each context, KASO generates four futures without gradients and evaluates the IDM at a shared high-noise action input. It keeps the candidate whose dense-action velocity best matches the IDM’s response to the recorded future. Replaying that candidate’s retained noise with gradients gives an exact differentiable sample for the action loss. Recorded video and action losses remain active. Compatibility is judged by the current model, so this selection is an operational proxy rather than a guarantee of physical validity.

Original figure
Figure 7: The KASO pipeline. Candidate visual predictions are sampled from the SVP and scored without gradients; selected noises are then replayed exactly so that the joint action loss updates both the IDM and the SVP.
Equations 19–21
En=1|𝒟|​∑i∈𝒟‖vϕ,i​(a~,tp,o,s,z^n)−vϕ,i​(a~,tp,o,s,z)‖22.E_{n}=\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}\Bigl\|v_{\phi,i}\bigl(\tilde{a},t_{p};\,o,s,\hat{z}_{n}\bigr)-v_{\phi,i}\bigl(\tilde{a},t_{p};\,o,s,z\bigr)\Bigr\|_{2}^{2}. (19)
ℒE2Esel=1k​∑n∈𝒮kℓFM​(a,o,s,z^n).\mathcal{L}_{\mathrm{E2E}}^{\mathrm{sel}}=\frac{1}{k}\sum_{n\in\mathcal{S}_{k}}\ell_{\mathrm{FM}}(a;o,s,\hat{z}_{n}). (20)
ℒKASO=ℒE2Esel+ℒSVP+ℒIDM.\mathcal{L}_{\mathrm{KASO}}=\mathcal{L}_{\mathrm{E2E}}^{\mathrm{sel}}+\mathcal{L}_{\mathrm{SVP}}+\mathcal{L}_{\mathrm{IDM}}. (21)
Algorithm 1 One co-training step with KASO. Setting k=Nk=N recovers E2E+PT; removing in addition ℒSVP\mathcal{L}_{\mathrm{SVP}} and ℒIDM\mathcal{L}_{\mathrm{IDM}} recovers E2E.
1: SVP flow-generator parameters θ\theta, IDM parameters ϕ\phi
2: Current observation latents oo, proprioceptive state ss, instruction cc, recorded future visual latents zz, recorded action aa
3: Group size NN, selected count kk, probe noise level tpt_{p}
4:
5: // Pretraining losses on recorded data
6: Sample video flow times and noise; compute ℒSVP=ℒu+ℒv\mathcal{L}_{\mathrm{SVP}}=\mathcal{L}_{u}+\mathcal{L}_{v} Eq. (13)
7: ℒIDM←ℓFM​(a,o,s,z)\mathcal{L}_{\mathrm{IDM}}\leftarrow\ell_{\mathrm{FM}}(a;\,o,s,z) Eq. (15)
8:
9: // Candidate generation without gradients
10: for n=1,…,Nn=1,\ldots,N do
11:   Draw and retain generation noise ξn\xi_{n}
12:   z^n←\hat{z}_{n}\leftarrow one-step generation from (o,c,ξn)(o,c,\xi_{n}) Eq. (14)
13: end for
14:
15: // Compatibility probe with the active IDM, no gradients
16: Draw shared probe noise ε\varepsilon; a~←tp​ε+(1−tp)​a\tilde{a}\leftarrow t_{p}\varepsilon+(1-t_{p})a
17: vref←vϕ​(a~,tp,o,s,z)v_{\mathrm{ref}}\leftarrow v_{\phi}(\tilde{a},t_{p};\,o,s,z)
18: for n=1,…,Nn=1,\ldots,N do
19:   En←‖vϕ​(a~,tp,o,s,z^n)−vref‖dense2E_{n}\leftarrow\|v_{\phi}(\tilde{a},t_{p};\,o,s,\hat{z}_{n})-v_{\mathrm{ref}}\|_{\mathrm{dense}}^{2} Eq. (19)
20: end for
21:
22: // Top-kk selection and differentiable replay
23: 𝒮k←\mathcal{S}_{k}\leftarrow indices of the kk smallest energies in {En}n=1N\{E_{n}\}_{n=1}^{N}
24: for n∈𝒮kn\in\mathcal{S}_{k} do
25:   z^n←\hat{z}_{n}\leftarrow replay ξn\xi_{n} with gradients
26: end for
27:
28: // Joint update
29: ℒE2Esel←1k​∑n∈𝒮kℓFM​(a,o,s,z^n)\mathcal{L}_{\mathrm{E2E}}^{\mathrm{sel}}\leftarrow\frac{1}{k}\sum_{n\in\mathcal{S}_{k}}\ell_{\mathrm{FM}}(a;\,o,s,\hat{z}_{n}) Eq. (20)
30: ℒKASO←λE2E​ℒE2Esel+λSVP​ℒSVP+λIDM​ℒIDM\mathcal{L}_{\mathrm{KASO}}\leftarrow\lambda_{\mathrm{E2E}}\mathcal{L}_{\mathrm{E2E}}^{\mathrm{sel}}+\lambda_{\mathrm{SVP}}\mathcal{L}_{\mathrm{SVP}}+\lambda_{\mathrm{IDM}}\mathcal{L}_{\mathrm{IDM}} Eq. (21)
31: θ,ϕ←θ,ϕ−α​∇θ,ϕℒKASO\theta,\phi\leftarrow\theta,\phi-\alpha\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{KASO}}

Controlled Real-Robot Ablation

All variants start from the same 39,000-hour SVP and 32,000-hour IDM checkpoints, then undergo only 300 hours of alignment data, including 30 G2-90D hours. On four targets with ten trials each, KASO and E2E+PT both follow the right target in 95% of trials, but successful picks rise from 22.5% to 37.5%. On 25 single-object trials, pickup rises from 12% to 40%. The small matched experiment supports a selection benefit in this setting; it is separate from the final model’s broad evaluation.

Table 2: Controlled real-robot ablation of retaining pretraining losses and action-compatible selection. Follow score requires contacting the instructed target and no distractor; pick success additionally requires completing the pickup. Four-object macro averages assign equal weight to the four targets. All variants initialize from the same full-scale pretrained components (SVP, 39,000 hours; IDM, 32,000 hours) and use the same 300-hour connection/alignment mixture, including 30 hours from G2-90D, without evaluation-specific SFT. Entries are percentages.
(a) Follow score: four-object scene (10 trials per target)
Method Sponge Red apple Mouse Remote Macro avg.
E2E 90 100 90 70 87.5
E2E+PT 100 100 90 90 95
KASO 100 100 90 90 95
(b) Pick success
Single-object scene (25 trials) Four-object scene (10 trials per target)
Method Small block Sponge Red apple Mouse Remote Macro avg.
E2E 12 30 60 20 0 27.5
E2E+PT 12 40 40 10 0 22.5
KASO 40 60 60 20 10 37.5

Training Data and Mixture Composition

The stages use 39,000 hours of instruction–video data, 32,000 hours of action-labeled trajectories, and 30,000 hours of fully labeled co-training data. Their mixtures include robot demonstrations, simulated data, human manipulation video, and failed or deployed rollouts. These are stage-specific totals, not necessarily distinct hours to add together. G1-OP supplies over half of co-training, while G2-90D supplies under 2%.

Original figure
Figure 8: Stage-specific training-data mixtures. The panels show the 39,000-hour SVP pretraining mixture, the 32,000-hour IDM pretraining mixture, and the 30,000-hour co-training mixture. Slice angle denotes the share of data hours; colors and source order are held fixed across panels. Violet and coral highlights emphasize the action-free egocentric/human video used for SVP pretraining and the rollout/failure trajectories used for IDM pretraining, respectively. The keys report source hours and percentages.

Evaluating Zero-Shot OOD Manipulation at Scale

The main study tests direct real-robot deployment on atomic skills and instruction grounding. Separate simulator experiments compare models after in-distribution adaptation. The policy performs low-level manipulation rather than explicit task decomposition or open-ended planning.

Zero-Shot OOD Evaluation without Per-Task SFT

Zero-shot means no evaluation-task demonstrations, fine-tuning, or per-task checkpoint choice. Test scenes, backgrounds, lighting, and object instances are excluded from training. One final checkpoint and fixed deployment settings serve the whole suite at each data scale; the evaluated robot embodiments themselves are represented in training.

Refer to caption
Figure 9: G1-OP zero-shot OOD scaling across 100 atomic tasks and four data volumes. Top: the 100 tasks as one mosaic per training-data scale, with tile intensity denoting per-task SR. Middle: representative zero-shot OOD rollouts of the 30,000-hour model. Bottom: per-task SR at every scale, grouped by skill; colors follow the skill-group legend.

Scaling Manipulation Skills

The suite has 100 tasks in 20 groups, including grasping, placement, tools, insertion, deformable objects, and coordinated arms. Each task receives ten trials per embodiment and data scale, with matched resets and a predefined terminal criterion. Success is reported at task, group, and suite levels.

Scaling with Training Data Volume

Nested co-training pools contain 300, 1,200, 5,000, and 30,000 hours, all starting from the same full pretrained components. G1-OP success is 17.1%, 22.6%, 27.3%, and 44.1%; G2-90D reaches 13.4%, 21.0%, 23.5%, and 31.1%. Gains cover 19/20 and 18/20 skill groups, although some task trajectories decline. Training stops after an epoch or the compute limit, so this is not a matched-compute estimate of data alone. G2-90D’s improvement suggests transfer, but its own data also grows and no fixed-target-data transfer ablation is reported.

Original figure
Figure 10: Skill-group zero-shot OOD scaling trajectories for G1-OP (more than 50% of the co-training mixture) and G2-90D (less than 2%). Each panel reports one group’s mean SR across the 300-, 1,200-, 5,000-, and 30,000-hour models.
Refer to caption
Figure 11: Representative successful zero-shot OOD rollouts on G2-90D across skill groups, each shown as three frames spanning the start, mid-execution, and outcome of the trial. Skill labels are colored by skill group, matching the palette of Figure 9.

Scaling with Skill-Specific Data Volume

Across 19 skill groups after excluding Close, log training hours and logit success correlate at Pearson r=0.80; rank correlation is 0.85. The fitted slope is 1.94 logit units per tenfold coverage increase. Wipe has much more data and higher success than Sweep, while well-covered complex skills can improve. This cross-skill association does not isolate coverage from difficulty, data quality, or other confounding factors.

Original figure
Figure 12: Skill-group training coverage and G1-OP zero-shot OOD success for the 30,000-hour model. Each point is one of the 19 groups retained after omitting the mechanically trivial Close group; the horizontal axis reports the corresponding training hours across all embodiments on a logarithmic scale, and the vertical axis reports SR on a logit scale. Coverage and success are strongly correlated (Pearson r=0.80r=0.80; Spearman ρ=0.85\rho=0.85), with a fitted gain of 1.94 logit units per decade of data.

Instruction Following in Complex Scenes

The final G1-OP model is tested on object identity, color, size, shape, position, and order, plus illustrative changes to an ongoing instruction. Pick and Place constrain motor demands so referent selection can be examined separately from execution.

Fine-Grained Grounding across Instruction Dimensions

The 59 instructions produce 295 trials. Follow Score counts contact with only the intended referent; full success also requires pickup or placement. Overall scores are 83.1% and 72.9%. Follow Score is at least 90% for object identity, color, shape, and direct position in both primitives, but only 13.3%/26.7% for order. Rare descriptors tend to be weaker, with shape an exception; frequency is observational and does not establish a causal training-data effect.

Refer to caption
Figure 13: Controlled scenes for fine-grained instruction following, ordered as introduced in the text: object identity, color, size, shape, and position & order. Position and order share one setup, evaluated in a horizontal layout (left/right and ordinal references) and a vertical layout (near/far). Top row: Pick scenes; bottom row: Place scenes.
Original figure
Figure 14: Fine-grained instruction following and language distribution across six grounding dimensions. Left: occurrence in 95,010 human-written RefCOCOg expressions (Natural) and the GE-Act 2.0 training corpus, shown on a logarithmic scale. Middle and right: Follow Score and full-task SR for Pick and Place, respectively. Dimensions are ordered by their occurrence in the GE training corpus; descriptor categories are counted independently and may overlap.

Instruction Following under Behavioral and Semantic Conflicts

Three illustrative rollouts show target switching near contact, switching the commanded arm, and placing a cup into a shoebox despite a conventional scene association. They show possible responsiveness, including residual motion after target changes. No aggregate conflict-test success rate is reported; these examples are outside the quantitative evaluation.

Refer to caption
Figure 15: Instruction-conflict stress tests. Each panel shows the active instruction, frames from one rollout, and the outcome; red rules mark where the instruction changed. (a) Target switch: the commanded cup changes from green to blue, then to orange, while the gripper is already at the previous target; the robot redirects each time and grasps the final target. (b) Arm switch: the command moves from the right arm on the green cup to the left arm on the orange cup, and the left arm completes it. (c) Semantic conflict: the robot places a cup into a shoebox, following the explicit object–destination relation over the conventional scene association. Illustrative only; not part of the quantitative evaluation.

OOD Performance on Simulation Benchmarks

Simulation uses benchmark-specific in-distribution SFT before testing held-out perturbations. GE-Act leads the compared models on RoboTwin’s combined Hard condition and average GenieSim-Instruction score, while trailing π0.5 overall on LIBERO-Plus. These adapted simulator results should be distinguished from the real-robot no-adaptation protocol. [7] [11] [66]

Conclusion

Compact visual futures provide a practical interface between separately pretrained video and action models. The strongest evidence is broader direct-deployment capability as co-training grows, with selective alignment helping a smaller controlled picking study. Absolute success remains limited across the full task suite.

Limitations and Outlook

The effect of scaling egocentric video alone is untested, and the model lacks explicit deliberation, memory, task decomposition, and long-horizon self-correction. The study also does not isolate data volume from compute or target-embodiment data growth. Small per-task trial counts and illustrative conflict rollouts limit conclusions about reliability beyond the tested settings.

Contributions and Acknowledgments

The tables preserve the full contributor lists and their roles. The authors acknowledge Maniformer’s data support and Yifei Chen’s discussion and editing assistance. The project page provides additional demonstrations.

Core contributors
Renhang Liu Wenzhi Zhao Zhuo Yang Liliang Chen Pengfei Zhou
Shengcong Chen Guanghui Ren
Contributors
Youlun Peng Rongjun Jin Nan Wang Sukai Wang Xindong He
Jinyuan Feng Ziyu Xiong Linqing Zhong Yifei Wei Feng Han
Long Zhang Da Huang Nanshu Zhao Chenghao Yin Mo Wu
Zhaodong Yan Kongtao Hu Yuxiang Yan Aogelijiang Niyazi Yu Fang
Jia Zeng Lizhu Meng Daizhen Lv Haoyu Cao Zhiwen Hou
Lianjin Ye Yuehan Niu Zhikai Cai Xuan Hu Hui Min
Xiongfeng Cai Yue Liao Jing Wu
Academic advisors
Soujanya Poria Ye Li Sanping Zhou
Project leaders
Liliang Chen Guanghui Ren
Supervisors
Guanghui Ren Maoqing Yao

Appendix A OOD Generalization in Simulation

All three simulator comparisons adapt on each benchmark’s designated training condition, then evaluate held-out shifts. They provide standardized cross-model references rather than direct evidence of out-of-box real-world transfer.

RoboTwin Clean-to-Random Generalization

After Easy/Clean adaptation, RoboTwin Hard success is 60.52%, versus 47.90% for π0.5. GE-Act leads five of six conditions; lighting is the exception. Its Easy-to-Hard decrease is 16.19 points. The table retains every model and perturbation.

Table 3: RoboTwin Clean-to-Random success rate (%, ↑\uparrow). Models are adapted only on the Easy/Clean setting; the remaining columns evaluate OOD environment variations. The best result in each column is bolded.
Model Easy Background Light Clutter Height Hard
StarVLA 58.10 27.10 50.90 24.20 48.40 10.60
GR00T-N1.7 43.60 40.40 41.90 27.10 39.00 20.70
π0.5\pi_{0.5} 73.10 67.00 69.20 57.90 67.60 47.90
GE-Act 2.0 76.71 70.28 65.92 66.52 70.39 60.52

GenieSim-Instruction: OOD Instruction Following

GenieSim-Instruction’s mean normalized score is 0.770, versus 0.757 for ACoT-VLA. GE-Act leads four of ten tasks, especially logical composition and common-sense reference; other models remain stronger on several direct attributes. The metric is a normalized task score, not a percentage of successful trials.

Table 4: Results on the OOD GenieSim-Instruction benchmark (normalized score, ↑\uparrow). Evaluation configurations vary background textures, object positions, and other scene factors relative to the training demonstrations. The best result in each row is bolded.
Task π0\pi_{0} GR00T-N1.7 LingBot-VLA 2.0 π0.5\pi_{0.5} ACoT-VLA GE-Act 2.0
pick_billiards_color 0.50 0.73 0.88 0.83 0.91 0.95
pick_block_color 0.49 0.80 0.80 0.87 0.96 0.92
pick_block_number 0.40 0.61 0.72 0.82 0.76 0.71
pick_block_shape 0.20 0.44 0.43 0.55 0.52 0.52
pick_block_size 0.34 0.62 0.68 0.79 0.83 0.76
pick_common_sense 0.19 0.68 0.63 0.59 0.66 0.74
pick_follow_logic_or 0.39 0.72 0.69 0.81 0.79 0.89
pick_object_type 0.42 0.72 0.82 0.80 0.80 0.78
pick_specific_object 0.37 0.69 0.87 0.81 0.83 0.89
straighten_object 0.38 0.45 0.56 0.59 0.51 0.54
Average 0.368 0.646 0.708 0.746 0.757 0.770

LIBERO-Plus Robustness across Perturbation Axes

LIBERO-Plus overall success is 80.4%, below π0.5’s 84.4%. GE-Act leads on camera and sensor-noise shifts but is weaker on robot-state and background changes. The overall score weights axes by their instance counts across 10,030 cases rather than averaging the seven displayed percentages.

Table 5: LIBERO-Plus success rate (%, ↑\uparrow) after adaptation on standard LIBERO. Each column evaluates a held-out perturbation axis. Overall is the success rate over all 10,030 task instances, so axes are weighted by their instance counts, following the benchmark’s protocol; all models are scored the same way. The best result in each column is bolded.
Model Camera Robot Language Light Background Noise Layout Overall
StarVLA 52.5 49.8 88.5 95.7 95.7 73.0 76.9 74.1
π0.5\pi_{0.5} 78.4 73.6 80.8 96.2 94.1 89.0 84.5 84.4
GE-Act 2.0 94.1 50.7 81.4 94.0 60.5 95.5 83.1 80.4

Appendix B Training Recipes and Hyperparameters

The recipes use bf16 mixed precision, ZeRO stage 2, AdamW, gradient clipping, and no classifier-free guidance or caption dropout. Batch sizes below are per GPU. Stage-specific data, warmups, losses, and deployment choices are retained in the following sections.

Architectures

The trainable visual generator has 2.51 billion parameters and the IDM 0.56 billion. Frozen Qwen3.5-2B adds 2.72 billion parameters outside those counts. Actions are 32-dimensional joint-space vectors with zero padding across embodiments; the table retains attention structure and latent sizes.

Table 6: Trained components. Parameter counts are read from the final co-trained checkpoint; the frozen Qwen3.5-2B VLM (2.72 B parameters) is excluded.
Component Configuration
CoAE Framewise deep-compression autoencoder (DC-AE layout) with 64×64\times spatial downsampling and 512 latent channels; a 256×384256\times 384 frame gives a 4×64\times 6 grid of 24 tokens. Latents are standardized per channel with corpus statistics.
SVP flow generator 2.51 B parameters. A 36-block diffusion transformer of width 2048 (32 heads): 20 shared blocks followed by 8 blocks for the mean-velocity head uθu_{\theta} and 8 parallel blocks for the instantaneous-velocity head vθv_{\theta}. Every block cross-attends to the fused VLM states; every third block attends across views. Conditioning enters through adaptive layer normalization as the sum of sinusoidal embeddings of tt and of rr and a learned view-identity embedding. Per view the input is one conditioning frame plus six predicted frames, i.e., 7×24=1687\times 24=168 tokens.
VLM (frozen) Qwen3.5-2B. The prompt holds the current head-view frame (96 vision tokens) and the instruction in a fixed window of 192 tokens; the text-span states of all 25 layers are fused by the layer gate of Eq. (5).
IDM 0.56 B parameters. 28 transformer blocks of width 1152 (16 heads) with self-attention over action tokens and cross-attention to the visual latent tokens of the current and predicted frames. The flow time and a learned embodiment embedding enter through adaptive layer normalization; actions are 32-dimensional joint-space vectors, zero-padded across embodiments, and the proprioceptive state enters through a small MLP.

Temporal Schedule and Sample Construction

Sample construction aligns all sources to a shared dense-control rate while extending sparse supervision toward each instruction segment’s endpoint.

Frames and actions.

Each sample has one observed frame, four dense and two sparse future frames, 52 dense actions at 30 Hz, and two sparse action targets. Absolute joint positions are standardized by source; sparse actions carry weight 0.1. The segment endpoint receives jitter and occasional extension into the next segment.

SVP Pretraining

Three quarters of training pairs use equal flow endpoints; the rest learn a longer interval. Forward-mode differentiation computes the derivative correction, which is detached and clipped to ±5. The observed latent receives small noise augmentation. Five percent of pretraining samples condition on a clean goal image instead of text. Training uses learning rate 10⁻⁴, 20,000 warmup steps, and per-GPU batches of 8 then 12.

IDM Pretraining

IDM pretraining uses recorded futures, a logit-normal action-flow time, and light latent noise and blur. The 32,000-hour run uses batch 16 per GPU, learning rate 10⁻⁴, and 5,000 warmup steps; its checkpoint also initializes the controlled ablation.

KASO Co-training

Co-training retains recorded-data objectives while applying generated-future action supervision only after compatibility selection.

Fused update.

Each batch accumulates three micro-steps with video, recorded-action, and selected-generated-action weights 1, 0.1, and 1, then divides the gradient by three. Recorded and generated action branches each use every example once. CoAE and the VLM remain frozen; the generator and IDM update.

Selection.

Four candidates compete for one selected future. The probe time is 0.95, and compatibility energy averages the 52 dense action tokens. The winner receives unit-weight action loss with fresh action noise and time; candidate selection is recomputed each update.

Runs.

Every scaling run shares the pretrained starting point, learning rate 10⁻⁴, 5,000-step warmup, and per-GPU batch 16. The controlled 300-hour experiment uses batch 8 and a 1,000-step warmup for KASO. Disabling selection gives E2E+PT; dropping the retained objectives gives E2E.

Deployment

At deployment, three views and the instruction yield a future in one visual-generator pass. The IDM uses five Euler steps. The controller executes the first 30 of the 52 dense actions at 30 Hz before predicting again. All main results use each run’s last checkpoint and fixed settings.

Toy Two-Stage System

The toy construction isolates off-manifold inputs and multimodal supervision using tiny networks. Its settings deliberately differ from the robot architecture and sampler.

Data-generating process.

Four balanced Gaussian visual modes lie exactly on a plane and map deterministically to two action modes near ±1. The absent third coordinate creates a direction unconstrained by recorded-pair training.

Networks and initialization.

A 4–24–24–3 tanh generator and 3–24–24–1 inverse regressor start identically across methods. Dividing the third visual coordinate by 0.2 amplifies generated off-plane deviations fivefold while leaving all recorded samples unchanged.

Shared optimization.

All arms train for 7,000 steps with batch 256 and a 48-step Euler generator. Pipeline loss ramps from zero after step 300 to full weight at step 600. Retained pretraining losses have lower coefficients in the connected arms; the original objective comparison remains below.

Equation
Decoupled Pretraining: ℒFM+ℒreal,\displaystyle\mathcal{L}_{\mathrm{FM}}+\mathcal{L}_{\mathrm{real}},
End-to-End Co-training: ρs​ℒpipe,\displaystyle\rho_{s}\,\mathcal{L}_{\mathrm{pipe}},
E2E + Pretraining Losses: 0.3​ℒFM+0.1​ℒreal+ρs​ℒpipe,\displaystyle 0.3\,\mathcal{L}_{\mathrm{FM}}+0.1\,\mathcal{L}_{\mathrm{real}}+\rho_{s}\,\mathcal{L}_{\mathrm{pipe}},
KASO: 0.3​ℒFM+0.1​ℒreal+ρs​ℒsel(8),\displaystyle 0.3\,\mathcal{L}_{\mathrm{FM}}+0.1\,\mathcal{L}_{\mathrm{real}}+\rho_{s}\,\mathcal{L}_{\mathrm{sel}}^{(8)},

Selection and replay.

Toy KASO samples eight candidates, scores their squared action error against the recorded action, and replays the selected noise with gradients. This direct scalar score differs from the robot model’s high-noise velocity probe. Evaluation uses one sample per input for every method.

Evaluation and stress-test controls.

The displayed seed-0 densities use 12,000 fixed noise draws and smoothed histograms. Across five seeds, the decoupled model’s maximum-density ratio to ground truth ranges from 0.240 to 1.035. The chosen run illustrates a failure mechanism despite good recorded-pair prediction; it is not representative evidence of a uniform failure across seeds.

Appendix C Full Results

The complete task taxonomy, per-task scaling scores, instruction prompts, and descriptor counts allow the aggregate results to be inspected.

Atomic-Skill Taxonomy and Evaluation Details

All 100 tasks have observable success criteria and ten trials per embodiment and data scale. Reset configurations are matched across checkpoints. The taxonomy includes requested arm use; otherwise tasks use the right arm.

Table 7: Atomic-task taxonomy for the 100-task real-robot zero-shot OOD suite. Parenthetical counts sum to 100. Unless an arm or both arms are explicitly specified, the task is executed with the right arm.
Skill group Atomic task instances
Pick (13) Red apple; toothpaste; red cube; water bottle (two scene variants); marker pen from water cup; toothbrush; water cup; controller; towel; stapler; whisk; towel from rod.
Place (6) Apple into container; blue cube into plate; orange into small bowl; marker pen into pen holder; red cube into paper cup; marker pen into open drawer.
Wipe (3) Plate with rag; water stain with sponge; table with eraser.
Stack (7) Red cube on yellow block; paper cup on paper cup; bowl on bowl; coaster on coaster; cola can on cola can; cutlery-box lid; trash-can lid.
Pass (5) Remote control, left-to-right; orange ball, right-to-left; sponge eraser, left-to-right; Rubik’s cube, right-to-left; cola can, left-to-right.
Fold (4) Towel, one arm; towel, both arms; rag twice, both arms; unfold towel, both arms.
Straighten (5) Water bottle (two scene variants); water cup (two scene variants); soda drink.
Pour (5) Fruit pieces into plate; sesame into plate; sugar into glass cup; beans into water cup; small cubes into bowl.
Push (8) Red block left; paper cup to centre; black box left; kettle forward; book to table edge; tissue pack into tissue box; close drawer; roll bottle.
Insert (6) Bread into bread machine; flower into vase; straw into cup; plug into power strip; remove book from shelf; place book into shelf.
Close (3) Tissue-box lid; blue-box lid; kettle lid.
Separate (3) Separate paper cups, both arms; uncap signature pen with the opposite arm holding the pen; unscrew bottle cap, both arms.
Press (4) Lamp switch; bread-machine button; stapler; calculator button.
Pull (6) Remove pen cap; pull tissue from bag; pull tray forward; pull cable from box; unplug charger while the opposite arm holds the power strip; open drawer.
Hang (4) Remove towel from rod; hang towel on rod; hang rag on storage box, left arm; hang hat on coat rack.
Tear (2) Open envelope, left arm; tear one sheet from paper-towel roll, both arms.
Flip (6) Close book; open book; turn book face-up; flip plate; flip coaster; flip packaged tissues.
Sweep (3) Sweep fruit cubes into dustpan with broom, both arms; sweep fruit cubes by hand; sweep fruit cubes into dustpan with brush while the opposite arm holds the dustpan.
Zip (3) Zip file bag, both arms; unzip file bag, both arms; unzip pencil case while the opposite arm holds the case.
Shake/Stir (4) Shake glass bottle; shake bottle to check remaining amount; stir coffee with chopstick; stir water with spoon.
Table 7: Atomic-task taxonomy for the 100-task real-robot zero-shot OOD suite (continued).

Task-Level Data-Scaling Results

Every result in the 100-row table is based on ten trials, giving 10-point increments. Both embodiments and all four scales are retained. At 30,000 hours, 24 G1-OP tasks and 28 G2-90D tasks still have no observed successes; no zipper task succeeds at any scale.

Table 8: Task-level zero-shot OOD success rate (SR, %) across data scales. Every G1-OP and G2-90D result is shown separately; each SR is computed over 10 trials, and all 100 tasks are included.
Task G1-OP SR (%) G2-90D SR (%)
0.3k h 1.2k h 5k h 30k h 0.3k h 1.2k h 5k h 30k h
Pick (13 tasks)
pick up red apple 90 100 100 100 60 100 100 100
pick up toothpaste 0 20 40 80 30 40 60 60
pick up red cube 80 80 70 90 70 70 50 50
pick up water bottle 40 50 70 90 60 100 90 100
pick up water bottle (2) 50 70 90 100 100 100 100 100
pick pen from water cup 0 10 30 60 0 30 40 40
pick up toothbrush 30 30 40 50 0 40 30 30
pick up water cup 20 40 40 80 40 60 50 50
pick up controller 10 30 60 50 80 100 80 40
pick up towel 80 90 80 100 70 80 80 70
pick up stapler 70 100 80 90 80 80 70 80
pick up whisk 60 50 70 80 40 50 70 70
pick towel from rod 0 20 50 70 0 60 60 70
Place (6 tasks)
put apple into container 100 100 100 100 100 100 90 90
put blue cube into plate 100 100 100 100 90 90 100 100
put orange into bowl 80 100 100 100 60 60 80 80
put pen into holder 20 20 40 50 0 40 30 70
put cube into paper cup 30 60 80 80 30 0 40 60
put pen into drawer 60 60 90 100 70 50 70 80
Wipe (3 tasks)
wipe plate with rag 20 20 30 70 0 0 10 60
wipe stain with sponge 30 40 40 80 0 0 20 50
wipe table with eraser 40 40 60 80 0 0 30 60
Stack (7 tasks)
stack cube on block 0 10 20 40 0 0 0 10
stack paper cups 0 0 0 30 0 0 0 10
stack bowls 70 80 80 90 90 90 100 90
stack coasters 80 70 80 100 70 70 80 70
stack cola cans 0 0 0 10 0 0 0 10
cover cutlery box lid 0 0 0 0 0 0 0 0
cover trash can lid 0 0 0 20 0 0 0 0
Pass (5 tasks)
pass remote control 40 60 60 80 0 20 30 60
pass orange ball 70 70 80 100 0 0 0 0
pass sponge eraser 60 90 100 100 40 50 60 80
pass rubik’s cube 0 0 0 0 0 0 0 0
pass cola can 0 40 60 70 0 20 20 30
Fold (4 tasks)
fold towel (one arm) 0 0 40 50 30 20 30 10
fold towel (two arms) 0 0 0 10 0 0 0 0
fold rag twice 0 0 0 0 0 0 0 0
unfold folded towel 20 0 0 0 0 0 0 0
Straighten (5 tasks)
straighten water bottle 0 0 0 10 0 0 0 10
straighten water bottle (2) 0 0 0 40 0 0 10 50
straighten water cup 0 0 0 60 0 0 0 40
straighten water cup (2) 0 0 0 30 0 0 0 0
straighten soda drink 0 0 0 10 0 0 0 20
Pour (5 tasks)
pour fruits into plate 20 40 50 70 30 20 20 30
pour sesame into plate 30 40 30 60 0 0 10 40
pour sugar into cup 10 20 30 50 0 0 0 10
pour beans into cup 0 20 40 50 0 0 10 10
pour cubes into bowl 0 0 0 40 0 30 0 10
Push (8 tasks)
push block to left 30 40 0 70 0 0 0 10
push cup to center 0 20 30 60 0 20 30 50
push box to left 40 40 60 70 0 0 10 20
push kettle forward 30 20 20 50 0 60 20 30
push book to edge 0 0 0 30 0 0 0 0
push tissue pack in 0 0 0 0 0 0 0 10
close the drawer 20 50 60 70 0 0 0 0
roll the bottle 0 0 0 10 0 0 10 10
Insert (6 tasks)
insert bread into machine 0 0 0 50 30 30 40 50
insert flower into vase 0 0 0 40 0 0 20 30
insert straw into cup 20 30 40 70 30 40 30 40
insert plug into strip 0 0 0 0 0 0 0 0
take book from shelf 0 0 0 60 0 0 20 50
put book into shelf 0 0 0 10 20 30 30 40
Close (3 tasks)
close tissue box lid 10 30 40 70 0 50 40 40
close blue box lid 30 50 40 90 0 70 60 80
close kettle lid 30 60 90 100 20 30 30 40
Separate (3 tasks)
separate paper cups 0 0 0 20 0 20 20 20
uncap signature pen 0 0 0 80 0 10 10 30
unscrew bottle cap 0 0 0 30 0 10 0 0
Press (4 tasks)
press lamp switch 0 0 0 0 0 0 0 0
press bread machine button 0 0 10 30 0 0 0 0
press stapler to staple 0 0 10 30 0 0 10 20
press calculator button 0 0 0 10 0 0 0 10
Pull (6 tasks)
pull off pen cap 0 0 0 20 0 0 0 20
pull tissue from bag 10 30 30 80 0 0 30 40
pull tray to front 0 0 10 30 0 0 0 10
pull cable from box 0 0 0 0 0 0 0 0
unplug charger from strip 0 0 0 0 0 0 0 10
open the drawer 0 0 0 10 0 0 0 0
Hang (4 tasks)
take towel off rod 0 0 10 60 0 0 30 50
hang towel on rod 0 0 0 0 0 0 0 0
hang rag on box 20 0 0 0 0 30 20 20
hang hat on rack 0 0 0 0 0 0 0 0
Tear (2 tasks)
tear open envelope 0 0 0 10 0 0 0 10
tear paper towel off roll 0 0 0 0 0 0 0 0
Flip (6 tasks)
close the open book 0 50 70 90 0 0 20 30
open the book 0 0 0 0 0 0 0 0
flip book to front 0 0 0 0 0 0 0 0
flip plate over 0 0 0 0 0 0 0 0
flip coaster to front 60 70 60 80 0 60 70 90
flip tissue pack over 0 0 0 0 0 0 0 0
Sweep (3 tasks)
sweep cubes into dustpan 0 0 0 10 0 0 0 0
sweep cubes by hand 0 0 0 0 0 0 0 10
sweep cubes with brush 0 0 0 0 0 0 0 10
Zip (3 tasks)
zip up file bag 0 0 0 0 0 0 0 0
unzip file bag 0 0 0 0 0 0 0 0
unzip pencil case 0 0 0 0 0 0 0 0
Shake/Stir (4 tasks)
shake glass bottle 0 0 10 20 0 40 40 60
shake bottle to check 0 0 10 30 0 30 40 60
stir coffee with chopstick 0 0 0 0 0 0 0 0
stir water with spoon 0 0 0 0 0 0 0 10
Table 8: Task-level zero-shot OOD success rate across data scales (continued).

Instruction-Following Prompts and Results

The appendix records exact issued instructions and distinguishes reaching the intended referent from completing the manipulation.

Protocol.

Thirty-three Pick and 26 Place instructions each receive five trials. Per-instruction scores are multiples of 20%; the overall 83.1% Follow Score and 72.9% success rate average all 295 trials.

Scenes.

Five scene families vary household-object identity, color, size, shape, and spatial arrangement. Horizontal rows test left/right and ordinal references; vertical layouts test near/far. Each scene supports multiple feasible choices but only one instruction-consistent referent.

Dimension mapping.

Size combines extreme and medium references; position combines left/right and near/far; order combines middle and ordinal expressions. The table retains each instruction verbatim, including the source’s minor wording irregularities.

Table 9: Instruction-level results of the fine-grained grounding study (G1-OP, five trials per instruction). Follow and SR are percentages of the five trials.
Dimension Instruction Follow SR
Pick (33 instructions)
Object pick up the white badminton ball with the right arm. 100 80
Object pick up the yellow knife with the left arm. 100 80
Object pick up the blue comb with the right arm. 100 80
Object pick up the black mouse with the left arm. 100 100
Object pick up the black tv controller with the right arm. 100 100
Object pick up the toy bear with the left arm. 100 60
Object pick up the yellow screwdriver with the right arm. 100 80
Object pick up the green tape with the right arm. 100 100
Color pick up the red fruit with the right arm. 100 60
Color pick up the orange fruit with the right arm. 100 100
Color pick up the yellow fruit with the right arm. 100 100
Color pick up the green fruit with the right arm. 100 100
Size (extreme) pick up the largest block with the right arm. 100 100
Size (extreme) pick up the smallest block with the left arm. 100 100
Size (medium) pick up the medium block with the right arm. 0 0
Size (extreme) pick up the larger cup with the left arm. 100 80
Size (medium) pick up the medium block with the left arm. 100 100
Size (extreme) pick up the largest block with the right arm. 100 100
Size (extreme) pick up the smallest block with the right arm. 100 80
Size (medium) pick up the medium block with the right arm. 60 40
Shape pick up the rectangular object with the right arm. 100 20
Shape pick up the cylindrical object with right arm. 60 60
Shape pick up the round object with right arm. 100 100
Shape pick up the cylindrical object with left arm. 100 100
Position (left/right) pick up the cup on the right side. 80 20
Order (ordinal) Pick up the second cup from the right. 0 0
Order (middle) Pick up the middle cup. 40 40
Order (ordinal) Pick up the second cup from the left. 0 0
Position (left/right) pick up the cup on the left side. 100 100
Position (left/right) pick up the cup on the right of the table with right arm. 100 100
Position (left/right) pick up the cup on the left of the table with left arm. 100 100
Position (near) Pick up the cup on the near side of the table with the right arm. 60 40
Position (far) pick up the farthest cup on the table with right arm. 100 100
Place (26 instructions)
Object place the object into the plate with the right arm. 100 100
Object place the object into the box with the right arm. 100 100
Object place the object into the bowl with the right arm. 100 80
Object place the object into the cup with the right arm. 100 60
Color place the object into the blue plate with the right arm. 100 100
Color place the object into the pink plate with the right arm. 100 100
Color place the object into the orange plate with the right arm. 100 100
Color place the object into the yellow plate with the right arm. 100 100
Size (extreme) place the object into the smallest container with left arm. 100 40
Size (extreme) place the object into the largest container with right arm. 100 100
Size (medium) place the object into the medium container with right arm. 0 0
Size (medium) place the object into the medium container with left arm. 80 80
Size (extreme) place the object into the smallest container with right arm 0 0
Size (extreme) place the object into the largest container with right arm. 80 80
Size (medium) place the object into the medium container with right arm. 100 100
Shape place the object into the rectangular container with the right arm. 100 100
Shape place the objectinto the cylindrical container with the right arm 80 40
Shape place the object into the round container with the right arm 80 60
Shape place the object into the round container with the left arm 100 100
Position (left/right) place the object into the container on the right side with the right arm. 100 100
Order (middle) place the object into the middle container with the right arm. 20 20
Order (middle) place the object into the middle container with the left arm. 40 40
Position (left/right) place the object into the container on the left side with the left arm. 100 100
Position (far) Place the object into the container at the far end of the table with the right arm. 100 100
Position (near) Place the object into the container on the near side of the table with the right arm. 100 60
Order (ordinal) place the object into the second container from the left. 20 20
Table 9: Instruction-level results of the fine-grained grounding study (continued).

Descriptor frequencies.

Lexical rules compare 95,010 RefCOCOg expressions with about 3.7 million training instruction segments. Non-object categories may overlap; object identity is a residual category. Simulation is excluded from the training count except for size, whose 0.96% includes simulated superlative references. These differing definitions matter when comparing frequencies. [39]

Table 10: Lexical matching rules and descriptor frequencies used in Figure 14. RefCOCOg counts are out of 95,010 expressions.
Dimension Matching rule RefCOCOg count RefCOCOg (%) GE corpus (%)
Object Residual: no color, size, shape, position, or order descriptor 39,474 41.5 58.3
Color Explicit color words; orange as a fruit name excluded 42,161 44.4 29.6
Size large, small, big, tiny, medium and comparative/superlative forms 5,299 5.6 0.96
Shape round, square, rectangular, cylindrical, triangular 574 0.6 0.67
Position left, right, near, far; body-side phrases such as right hand excluded unless another spatial relation is present 11,927 12.6 13.8
Order middle, center, ordinals, leftmost, rightmost; generic between excluded 2,440 2.6 0.13
Concept demonstration

Open image ↗