At a glance
GE-Act 2.0 predicts visual futures in one generator pass, then converts them into robot actions. Separate video and inverse-dynamics pretraining are joined by KASO, which selects generated futures compatible with the recorded behavior. Expanding co-training from 300 to 30,000 hours raises real-robot success from 17.1% to 44.1% on G1-OP and 13.4% to 31.1% on G2-90D, without evaluation-task fine-tuning. Every scale shares the same much larger pretrained components, and compute is not matched. Grounding is strong for several direct attributes but weak for ordinal instructions.
Introduction
A visual planner can learn from instruction-labeled video without robot actions, while inverse dynamics can learn from action-labeled trajectories without language or success labels. GE-Act 2.0 connects these paths through completed predicted futures. The central training problem is that a plausible future can depict a different valid behavior from the demonstration whose action is used as its label. [31] [54] [64]

Contributions.
The system combines compact control-oriented visual latents, a differentiable one-pass visual predictor, and selective co-training. Its main evaluation asks what the resulting checkpoint can do before task adaptation, across 100 tasks and two robot embodiments.
Related Work
The related work places the design among generalist policies, visual representations for control, fast generative models, and methods that connect predicted futures to actions.
Generalist Policies and World-Action Models
Earlier systems directly predict actions, jointly generate video and actions, or derive actions from a generator’s internal features. Here a completed future is the explicit interface to a separately pretrained inverse dynamics model, differing from GE-Act 1.0’s parallel action branch. [1] [2] [31] [44] [45]
Representations for Robot Video and Control
Prior representations emphasize reconstruction, latent motion, or predictive features. CoAE combines a pixel decoder with alignment to pretrained visual teachers, then measures usefulness through action recovery and instruction matching under a fixed probe architecture. [5] [8] [9] [61] [62] [65] [69]
Fast Video Generation for World Models
The visual predictor builds on MeanFlow and improved MeanFlow, which learn interval-averaged velocity for one-pass generation. The contribution applies this mechanism to conditional multi-view robot futures and makes the completed result differentiable for action training. [12] [15] [16] [32] [47] [50] [67] [68]
Connecting Generated Futures to Actions
Training inverse dynamics on recorded futures leaves an input shift when deployment uses generated ones. Even successful generation does not resolve a different mismatch: a sampled future and recorded action may represent different choices. KASO ranks candidates by the current inverse model’s action response. A concurrent generative-modeling method also uses best-of-K matching. [6] [17] [30] [63] [72]
Evaluating Pretrained Robot Models
Task-specific fine-tuning mixes pretrained ability with adaptation-data coverage. The main real-robot protocol therefore uses final pretrained checkpoints directly; simulator comparisons include their prescribed adaptation and answer a different question. [24] [34] [48] [70]
GE-Act 2.0
The following sections define the visual representation, one-pass planner, inverse dynamics model, and compatibility-based joint training.
System Overview
CoAE compresses each camera frame. A scene-conditioned SVP predicts dense near-term and sparse farther-future latents, and the IDM combines them with the current views and proprioception to produce actions. The controller executes a near-term chunk; sparse actions provide auxiliary training targets. The generator and IDM start randomly, but CoAE transfers compatible DC-AE weights and uses frozen teachers; a pretrained frozen VLM supplies language conditioning. Thus the complete system is not learned without pretrained components.
A Control-Oriented Autoencoder for World–Action Models
The visual representation must be small enough for frequent prediction while retaining changes that identify the robot’s motion and the instruction’s referents. Reconstruction alone does not directly measure these properties.
Beyond Reconstruction with Multi-Teacher Alignment
CoAE downsamples each spatial axis by 64: a 256×384 frame becomes 24 tokens with 512 channels. Compatible weights come from the 128-channel DC-AE, with new parameters initialized randomly. Pixel, perceptual, and adversarial reconstruction losses are combined with alignment to frozen SigLIP 2, V-JEPA 2.1, and DINOv3 features. [5] [41] [49] [52]
Equations 1–2
| (1) |
| (2) |
Control and Instruction Grounding under Extreme Compression
With frozen encoders and the same two-layer action probe, CoAE reaches MAE 0.01673 on 100 held-out samples from a single shelf task, after training on 400 samples. DINOv3 and V-JEPA 2.1 are better, but use 16 times as many tokens. A separate 5,000-episode caption-matching probe reaches 97.95% accuracy with CoAE; its eight-layer probe retains VLM conditioning. These probe results support compactness and accessibility of information in the tested settings, rather than proving general control superiority or isolating each teacher’s contribution.
| Representation | Spatial downsampling | Channels | Pixel decoder | Action MAE | Caption-match error (%) |
|---|---|---|---|---|---|
| CoAE | 512 | ✓ | 0.01673 | 2.05 | |
| DC-AE | 128 | ✓ | 0.02573 | 2.71 | |
| DINOv3 | 1024 | ✗ | 0.01273 | 2.75 | |
| V-JEPA 2.1 | 1024 | ✗ | 0.01487 | 4.15 | |
| SigLIP | 2048 | ✗ | 0.03138 | 3.28 |
A Single-Step Visual Planner as a World–Action Interface
The SVP combines a frozen multimodal understanding module with a randomly initialized conditional flow transformer. It predicts complete future latents in one pass, allowing the inverse model to consume the same kind of input during joint training and deployment.
One-Step Generation Unlocks Underutilized Data
Separate IDM training can exploit failed attempts, play, and deployment rollouts when observations and actions are aligned. Joining it to a multi-step generator makes action backpropagation expensive. A one-pass generator reduces that cost, enabling gradients through the generated future while preserving the separate pretraining paths.
Multi-Scale Visual Prediction
Four dense future frames cover the next 52 control frames, about 1.73 seconds at 30 Hz; two sparse frames extend toward the instruction segment’s end. This spends temporal resolution near execution while retaining a farther goal signal. Sparse action targets are auxiliary and are not the executed chunk. [74]
Equation 3
| (3) |
Toward Unified Understanding and Generation
Frozen Qwen3.5 jointly processes the current head-camera image and instruction. A learned gate combines text-span states from its layers, retaining scene context with a text-length conditioning sequence. Every visual-generator block attends to this result. The IDM receives no language directly; instructions influence it through the predicted visual future. [46]
Equations 4–5
| (4) |
| (5) |
Conditional MeanFlow for One-Step Visual Planning
Conditional MeanFlow learns an average velocity that maps noise to future visual latents in one evaluation. An auxiliary instantaneous-velocity head supplies the tangent for a Jacobian–vector product. Masks keep observed-frame coordinates fixed and supervise future coordinates only; the detached derivative correction stabilizes learning. The full equations retain the endpoint conventions and masked losses.
Equations 6–14
| (6) |
| (7) |
| (8) |
| (9) |
| (10) |
| (11) |
| (12) |
| (13) |
| (14) |
World-Action Alignment and KASO
Joint training must adapt the inverse model to generated inputs while avoiding contradictory supervision and loss of separately pretrained capabilities.
Inverse Dynamics Model
The IDM uses flow matching to predict action sequences from current and future visual latents plus proprioception. Recorded futures supervise standalone pretraining; generated futures supervise the connected branch during co-training. Its action sampler remains multi-step even though visual generation is one-pass.
Equation 15
| (15) |
Knowledge Transfer with Conflicting Supervision
Plain E2E training propagates action loss through the generated future into both models. E2E+PT additionally retains recorded-video and recorded-action pretraining losses. Those losses help protect prior capabilities but do not ensure that a generated future agrees with the recorded action’s behavior. [10]
Equations 16–17
| (16) |
| (17) |
The Validity Gap
A demonstration supplies a coupled future–action pair. Sampling a new future independently can select another valid grasp, route, or timing while retaining the original action target. The resulting validity gap persists even when the generator matches the true marginal distribution of plausible futures.
Equation 18
| (18) |
A Controlled Toy Two-Stage System.
The toy study maps four visual modes to two scalar action modes. A deterministic inverse regressor can average contradictory action labels toward zero, while generated states outside the recorded plane expose an intentionally amplified sensitivity. Selection preserves the displayed modes in the seed-0 example. This is a controlled illustration with substantial variation across seeds, not an averaged benchmark or a full replica of the robot model.
KASO: Knowledge-Aligned Selective Optimization
For each context, KASO generates four futures without gradients and evaluates the IDM at a shared high-noise action input. It keeps the candidate whose dense-action velocity best matches the IDM’s response to the recorded future. Replaying that candidate’s retained noise with gradients gives an exact differentiable sample for the action loss. Recorded video and action losses remain active. Compatibility is judged by the current model, so this selection is an operational proxy rather than a guarantee of physical validity.
Equations 19–21
| (19) |
| (20) |
| (21) |
Controlled Real-Robot Ablation
All variants start from the same 39,000-hour SVP and 32,000-hour IDM checkpoints, then undergo only 300 hours of alignment data, including 30 G2-90D hours. On four targets with ten trials each, KASO and E2E+PT both follow the right target in 95% of trials, but successful picks rise from 22.5% to 37.5%. On 25 single-object trials, pickup rises from 12% to 40%. The small matched experiment supports a selection benefit in this setting; it is separate from the final model’s broad evaluation.
| (a) Follow score: four-object scene (10 trials per target) | ||||||
| Method | Sponge | Red apple | Mouse | Remote | Macro avg. | |
| E2E | 90 | 100 | 90 | 70 | 87.5 | |
| E2E+PT | 100 | 100 | 90 | 90 | 95 | |
| KASO | 100 | 100 | 90 | 90 | 95 | |
| (b) Pick success | ||||||
| Single-object scene (25 trials) | Four-object scene (10 trials per target) | |||||
| Method | Small block | Sponge | Red apple | Mouse | Remote | Macro avg. |
| E2E | 12 | 30 | 60 | 20 | 0 | 27.5 |
| E2E+PT | 12 | 40 | 40 | 10 | 0 | 22.5 |
| KASO | 40 | 60 | 60 | 20 | 10 | 37.5 |
Training Data and Mixture Composition
The stages use 39,000 hours of instruction–video data, 32,000 hours of action-labeled trajectories, and 30,000 hours of fully labeled co-training data. Their mixtures include robot demonstrations, simulated data, human manipulation video, and failed or deployed rollouts. These are stage-specific totals, not necessarily distinct hours to add together. G1-OP supplies over half of co-training, while G2-90D supplies under 2%.
Evaluating Zero-Shot OOD Manipulation at Scale
The main study tests direct real-robot deployment on atomic skills and instruction grounding. Separate simulator experiments compare models after in-distribution adaptation. The policy performs low-level manipulation rather than explicit task decomposition or open-ended planning.
Zero-Shot OOD Evaluation without Per-Task SFT
Zero-shot means no evaluation-task demonstrations, fine-tuning, or per-task checkpoint choice. Test scenes, backgrounds, lighting, and object instances are excluded from training. One final checkpoint and fixed deployment settings serve the whole suite at each data scale; the evaluated robot embodiments themselves are represented in training.
Scaling Manipulation Skills
The suite has 100 tasks in 20 groups, including grasping, placement, tools, insertion, deformable objects, and coordinated arms. Each task receives ten trials per embodiment and data scale, with matched resets and a predefined terminal criterion. Success is reported at task, group, and suite levels.
Scaling with Training Data Volume
Nested co-training pools contain 300, 1,200, 5,000, and 30,000 hours, all starting from the same full pretrained components. G1-OP success is 17.1%, 22.6%, 27.3%, and 44.1%; G2-90D reaches 13.4%, 21.0%, 23.5%, and 31.1%. Gains cover 19/20 and 18/20 skill groups, although some task trajectories decline. Training stops after an epoch or the compute limit, so this is not a matched-compute estimate of data alone. G2-90D’s improvement suggests transfer, but its own data also grows and no fixed-target-data transfer ablation is reported.
Scaling with Skill-Specific Data Volume
Across 19 skill groups after excluding Close, log training hours and logit success correlate at Pearson r=0.80; rank correlation is 0.85. The fitted slope is 1.94 logit units per tenfold coverage increase. Wipe has much more data and higher success than Sweep, while well-covered complex skills can improve. This cross-skill association does not isolate coverage from difficulty, data quality, or other confounding factors.
Instruction Following in Complex Scenes
The final G1-OP model is tested on object identity, color, size, shape, position, and order, plus illustrative changes to an ongoing instruction. Pick and Place constrain motor demands so referent selection can be examined separately from execution.
Fine-Grained Grounding across Instruction Dimensions
The 59 instructions produce 295 trials. Follow Score counts contact with only the intended referent; full success also requires pickup or placement. Overall scores are 83.1% and 72.9%. Follow Score is at least 90% for object identity, color, shape, and direct position in both primitives, but only 13.3%/26.7% for order. Rare descriptors tend to be weaker, with shape an exception; frequency is observational and does not establish a causal training-data effect.
Instruction Following under Behavioral and Semantic Conflicts
Three illustrative rollouts show target switching near contact, switching the commanded arm, and placing a cup into a shoebox despite a conventional scene association. They show possible responsiveness, including residual motion after target changes. No aggregate conflict-test success rate is reported; these examples are outside the quantitative evaluation.
OOD Performance on Simulation Benchmarks
Simulation uses benchmark-specific in-distribution SFT before testing held-out perturbations. GE-Act leads the compared models on RoboTwin’s combined Hard condition and average GenieSim-Instruction score, while trailing π0.5 overall on LIBERO-Plus. These adapted simulator results should be distinguished from the real-robot no-adaptation protocol. [7] [11] [66]
Conclusion
Compact visual futures provide a practical interface between separately pretrained video and action models. The strongest evidence is broader direct-deployment capability as co-training grows, with selective alignment helping a smaller controlled picking study. Absolute success remains limited across the full task suite.
Limitations and Outlook
The effect of scaling egocentric video alone is untested, and the model lacks explicit deliberation, memory, task decomposition, and long-horizon self-correction. The study also does not isolate data volume from compute or target-embodiment data growth. Small per-task trial counts and illustrative conflict rollouts limit conclusions about reliability beyond the tested settings.
Contributions and Acknowledgments
The tables preserve the full contributor lists and their roles. The authors acknowledge Maniformer’s data support and Yifei Chen’s discussion and editing assistance. The project page provides additional demonstrations.
| Renhang Liu | Wenzhi Zhao | Zhuo Yang | Liliang Chen | Pengfei Zhou |
| Shengcong Chen | Guanghui Ren |
| Youlun Peng | Rongjun Jin | Nan Wang | Sukai Wang | Xindong He |
| Jinyuan Feng | Ziyu Xiong | Linqing Zhong | Yifei Wei | Feng Han |
| Long Zhang | Da Huang | Nanshu Zhao | Chenghao Yin | Mo Wu |
| Zhaodong Yan | Kongtao Hu | Yuxiang Yan | Aogelijiang Niyazi | Yu Fang |
| Jia Zeng | Lizhu Meng | Daizhen Lv | Haoyu Cao | Zhiwen Hou |
| Lianjin Ye | Yuehan Niu | Zhikai Cai | Xuan Hu | Hui Min |
| Xiongfeng Cai | Yue Liao | Jing Wu |
| Soujanya Poria | Ye Li | Sanping Zhou |
| Liliang Chen | Guanghui Ren |
| Guanghui Ren | Maoqing Yao |
Appendix A OOD Generalization in Simulation
All three simulator comparisons adapt on each benchmark’s designated training condition, then evaluate held-out shifts. They provide standardized cross-model references rather than direct evidence of out-of-box real-world transfer.
RoboTwin Clean-to-Random Generalization
After Easy/Clean adaptation, RoboTwin Hard success is 60.52%, versus 47.90% for π0.5. GE-Act leads five of six conditions; lighting is the exception. Its Easy-to-Hard decrease is 16.19 points. The table retains every model and perturbation.
| Model | Easy | Background | Light | Clutter | Height | Hard |
|---|---|---|---|---|---|---|
| StarVLA | 58.10 | 27.10 | 50.90 | 24.20 | 48.40 | 10.60 |
| GR00T-N1.7 | 43.60 | 40.40 | 41.90 | 27.10 | 39.00 | 20.70 |
| 73.10 | 67.00 | 69.20 | 57.90 | 67.60 | 47.90 | |
| GE-Act 2.0 | 76.71 | 70.28 | 65.92 | 66.52 | 70.39 | 60.52 |
GenieSim-Instruction: OOD Instruction Following
GenieSim-Instruction’s mean normalized score is 0.770, versus 0.757 for ACoT-VLA. GE-Act leads four of ten tasks, especially logical composition and common-sense reference; other models remain stronger on several direct attributes. The metric is a normalized task score, not a percentage of successful trials.
| Task | GR00T-N1.7 | LingBot-VLA 2.0 | ACoT-VLA | GE-Act 2.0 | ||
|---|---|---|---|---|---|---|
| pick_billiards_color | 0.50 | 0.73 | 0.88 | 0.83 | 0.91 | 0.95 |
| pick_block_color | 0.49 | 0.80 | 0.80 | 0.87 | 0.96 | 0.92 |
| pick_block_number | 0.40 | 0.61 | 0.72 | 0.82 | 0.76 | 0.71 |
| pick_block_shape | 0.20 | 0.44 | 0.43 | 0.55 | 0.52 | 0.52 |
| pick_block_size | 0.34 | 0.62 | 0.68 | 0.79 | 0.83 | 0.76 |
| pick_common_sense | 0.19 | 0.68 | 0.63 | 0.59 | 0.66 | 0.74 |
| pick_follow_logic_or | 0.39 | 0.72 | 0.69 | 0.81 | 0.79 | 0.89 |
| pick_object_type | 0.42 | 0.72 | 0.82 | 0.80 | 0.80 | 0.78 |
| pick_specific_object | 0.37 | 0.69 | 0.87 | 0.81 | 0.83 | 0.89 |
| straighten_object | 0.38 | 0.45 | 0.56 | 0.59 | 0.51 | 0.54 |
| Average | 0.368 | 0.646 | 0.708 | 0.746 | 0.757 | 0.770 |
LIBERO-Plus Robustness across Perturbation Axes
LIBERO-Plus overall success is 80.4%, below π0.5’s 84.4%. GE-Act leads on camera and sensor-noise shifts but is weaker on robot-state and background changes. The overall score weights axes by their instance counts across 10,030 cases rather than averaging the seven displayed percentages.
| Model | Camera | Robot | Language | Light | Background | Noise | Layout | Overall |
|---|---|---|---|---|---|---|---|---|
| StarVLA | 52.5 | 49.8 | 88.5 | 95.7 | 95.7 | 73.0 | 76.9 | 74.1 |
| 78.4 | 73.6 | 80.8 | 96.2 | 94.1 | 89.0 | 84.5 | 84.4 | |
| GE-Act 2.0 | 94.1 | 50.7 | 81.4 | 94.0 | 60.5 | 95.5 | 83.1 | 80.4 |
Appendix B Training Recipes and Hyperparameters
The recipes use bf16 mixed precision, ZeRO stage 2, AdamW, gradient clipping, and no classifier-free guidance or caption dropout. Batch sizes below are per GPU. Stage-specific data, warmups, losses, and deployment choices are retained in the following sections.
Architectures
The trainable visual generator has 2.51 billion parameters and the IDM 0.56 billion. Frozen Qwen3.5-2B adds 2.72 billion parameters outside those counts. Actions are 32-dimensional joint-space vectors with zero padding across embodiments; the table retains attention structure and latent sizes.
| Component | Configuration |
|---|---|
| CoAE | Framewise deep-compression autoencoder (DC-AE layout) with spatial downsampling and 512 latent channels; a frame gives a grid of 24 tokens. Latents are standardized per channel with corpus statistics. |
| SVP flow generator | 2.51 B parameters. A 36-block diffusion transformer of width 2048 (32 heads): 20 shared blocks followed by 8 blocks for the mean-velocity head and 8 parallel blocks for the instantaneous-velocity head . Every block cross-attends to the fused VLM states; every third block attends across views. Conditioning enters through adaptive layer normalization as the sum of sinusoidal embeddings of and of and a learned view-identity embedding. Per view the input is one conditioning frame plus six predicted frames, i.e., tokens. |
| VLM (frozen) | Qwen3.5-2B. The prompt holds the current head-view frame (96 vision tokens) and the instruction in a fixed window of 192 tokens; the text-span states of all 25 layers are fused by the layer gate of Eq. (5). |
| IDM | 0.56 B parameters. 28 transformer blocks of width 1152 (16 heads) with self-attention over action tokens and cross-attention to the visual latent tokens of the current and predicted frames. The flow time and a learned embodiment embedding enter through adaptive layer normalization; actions are 32-dimensional joint-space vectors, zero-padded across embodiments, and the proprioceptive state enters through a small MLP. |
Temporal Schedule and Sample Construction
Sample construction aligns all sources to a shared dense-control rate while extending sparse supervision toward each instruction segment’s endpoint.
Frames and actions.
Each sample has one observed frame, four dense and two sparse future frames, 52 dense actions at 30 Hz, and two sparse action targets. Absolute joint positions are standardized by source; sparse actions carry weight 0.1. The segment endpoint receives jitter and occasional extension into the next segment.
SVP Pretraining
Three quarters of training pairs use equal flow endpoints; the rest learn a longer interval. Forward-mode differentiation computes the derivative correction, which is detached and clipped to ±5. The observed latent receives small noise augmentation. Five percent of pretraining samples condition on a clean goal image instead of text. Training uses learning rate 10⁻⁴, 20,000 warmup steps, and per-GPU batches of 8 then 12.
IDM Pretraining
IDM pretraining uses recorded futures, a logit-normal action-flow time, and light latent noise and blur. The 32,000-hour run uses batch 16 per GPU, learning rate 10⁻⁴, and 5,000 warmup steps; its checkpoint also initializes the controlled ablation.
KASO Co-training
Co-training retains recorded-data objectives while applying generated-future action supervision only after compatibility selection.
Fused update.
Each batch accumulates three micro-steps with video, recorded-action, and selected-generated-action weights 1, 0.1, and 1, then divides the gradient by three. Recorded and generated action branches each use every example once. CoAE and the VLM remain frozen; the generator and IDM update.
Selection.
Four candidates compete for one selected future. The probe time is 0.95, and compatibility energy averages the 52 dense action tokens. The winner receives unit-weight action loss with fresh action noise and time; candidate selection is recomputed each update.
Runs.
Every scaling run shares the pretrained starting point, learning rate 10⁻⁴, 5,000-step warmup, and per-GPU batch 16. The controlled 300-hour experiment uses batch 8 and a 1,000-step warmup for KASO. Disabling selection gives E2E+PT; dropping the retained objectives gives E2E.
Deployment
At deployment, three views and the instruction yield a future in one visual-generator pass. The IDM uses five Euler steps. The controller executes the first 30 of the 52 dense actions at 30 Hz before predicting again. All main results use each run’s last checkpoint and fixed settings.
Toy Two-Stage System
The toy construction isolates off-manifold inputs and multimodal supervision using tiny networks. Its settings deliberately differ from the robot architecture and sampler.
Data-generating process.
Four balanced Gaussian visual modes lie exactly on a plane and map deterministically to two action modes near ±1. The absent third coordinate creates a direction unconstrained by recorded-pair training.
Networks and initialization.
A 4–24–24–3 tanh generator and 3–24–24–1 inverse regressor start identically across methods. Dividing the third visual coordinate by 0.2 amplifies generated off-plane deviations fivefold while leaving all recorded samples unchanged.
Shared optimization.
All arms train for 7,000 steps with batch 256 and a 48-step Euler generator. Pipeline loss ramps from zero after step 300 to full weight at step 600. Retained pretraining losses have lower coefficients in the connected arms; the original objective comparison remains below.
Selection and replay.
Toy KASO samples eight candidates, scores their squared action error against the recorded action, and replays the selected noise with gradients. This direct scalar score differs from the robot model’s high-noise velocity probe. Evaluation uses one sample per input for every method.
Evaluation and stress-test controls.
The displayed seed-0 densities use 12,000 fixed noise draws and smoothed histograms. Across five seeds, the decoupled model’s maximum-density ratio to ground truth ranges from 0.240 to 1.035. The chosen run illustrates a failure mechanism despite good recorded-pair prediction; it is not representative evidence of a uniform failure across seeds.
Appendix C Full Results
The complete task taxonomy, per-task scaling scores, instruction prompts, and descriptor counts allow the aggregate results to be inspected.
Atomic-Skill Taxonomy and Evaluation Details
All 100 tasks have observable success criteria and ten trials per embodiment and data scale. Reset configurations are matched across checkpoints. The taxonomy includes requested arm use; otherwise tasks use the right arm.
| Skill group | Atomic task instances |
|---|---|
| Pick (13) | Red apple; toothpaste; red cube; water bottle (two scene variants); marker pen from water cup; toothbrush; water cup; controller; towel; stapler; whisk; towel from rod. |
| Place (6) | Apple into container; blue cube into plate; orange into small bowl; marker pen into pen holder; red cube into paper cup; marker pen into open drawer. |
| Wipe (3) | Plate with rag; water stain with sponge; table with eraser. |
| Stack (7) | Red cube on yellow block; paper cup on paper cup; bowl on bowl; coaster on coaster; cola can on cola can; cutlery-box lid; trash-can lid. |
| Pass (5) | Remote control, left-to-right; orange ball, right-to-left; sponge eraser, left-to-right; Rubik’s cube, right-to-left; cola can, left-to-right. |
| Fold (4) | Towel, one arm; towel, both arms; rag twice, both arms; unfold towel, both arms. |
| Straighten (5) | Water bottle (two scene variants); water cup (two scene variants); soda drink. |
| Pour (5) | Fruit pieces into plate; sesame into plate; sugar into glass cup; beans into water cup; small cubes into bowl. |
| Push (8) | Red block left; paper cup to centre; black box left; kettle forward; book to table edge; tissue pack into tissue box; close drawer; roll bottle. |
| Insert (6) | Bread into bread machine; flower into vase; straw into cup; plug into power strip; remove book from shelf; place book into shelf. |
| Close (3) | Tissue-box lid; blue-box lid; kettle lid. |
| Separate (3) | Separate paper cups, both arms; uncap signature pen with the opposite arm holding the pen; unscrew bottle cap, both arms. |
| Press (4) | Lamp switch; bread-machine button; stapler; calculator button. |
| Pull (6) | Remove pen cap; pull tissue from bag; pull tray forward; pull cable from box; unplug charger while the opposite arm holds the power strip; open drawer. |
| Hang (4) | Remove towel from rod; hang towel on rod; hang rag on storage box, left arm; hang hat on coat rack. |
| Tear (2) | Open envelope, left arm; tear one sheet from paper-towel roll, both arms. |
| Flip (6) | Close book; open book; turn book face-up; flip plate; flip coaster; flip packaged tissues. |
| Sweep (3) | Sweep fruit cubes into dustpan with broom, both arms; sweep fruit cubes by hand; sweep fruit cubes into dustpan with brush while the opposite arm holds the dustpan. |
| Zip (3) | Zip file bag, both arms; unzip file bag, both arms; unzip pencil case while the opposite arm holds the case. |
| Shake/Stir (4) | Shake glass bottle; shake bottle to check remaining amount; stir coffee with chopstick; stir water with spoon. |
Task-Level Data-Scaling Results
Every result in the 100-row table is based on ten trials, giving 10-point increments. Both embodiments and all four scales are retained. At 30,000 hours, 24 G1-OP tasks and 28 G2-90D tasks still have no observed successes; no zipper task succeeds at any scale.
| Task | G1-OP SR (%) | G2-90D SR (%) | ||||||
|---|---|---|---|---|---|---|---|---|
| 0.3k h | 1.2k h | 5k h | 30k h | 0.3k h | 1.2k h | 5k h | 30k h | |
| Pick (13 tasks) | ||||||||
| pick up red apple | 90 | 100 | 100 | 100 | 60 | 100 | 100 | 100 |
| pick up toothpaste | 0 | 20 | 40 | 80 | 30 | 40 | 60 | 60 |
| pick up red cube | 80 | 80 | 70 | 90 | 70 | 70 | 50 | 50 |
| pick up water bottle | 40 | 50 | 70 | 90 | 60 | 100 | 90 | 100 |
| pick up water bottle (2) | 50 | 70 | 90 | 100 | 100 | 100 | 100 | 100 |
| pick pen from water cup | 0 | 10 | 30 | 60 | 0 | 30 | 40 | 40 |
| pick up toothbrush | 30 | 30 | 40 | 50 | 0 | 40 | 30 | 30 |
| pick up water cup | 20 | 40 | 40 | 80 | 40 | 60 | 50 | 50 |
| pick up controller | 10 | 30 | 60 | 50 | 80 | 100 | 80 | 40 |
| pick up towel | 80 | 90 | 80 | 100 | 70 | 80 | 80 | 70 |
| pick up stapler | 70 | 100 | 80 | 90 | 80 | 80 | 70 | 80 |
| pick up whisk | 60 | 50 | 70 | 80 | 40 | 50 | 70 | 70 |
| pick towel from rod | 0 | 20 | 50 | 70 | 0 | 60 | 60 | 70 |
| Place (6 tasks) | ||||||||
| put apple into container | 100 | 100 | 100 | 100 | 100 | 100 | 90 | 90 |
| put blue cube into plate | 100 | 100 | 100 | 100 | 90 | 90 | 100 | 100 |
| put orange into bowl | 80 | 100 | 100 | 100 | 60 | 60 | 80 | 80 |
| put pen into holder | 20 | 20 | 40 | 50 | 0 | 40 | 30 | 70 |
| put cube into paper cup | 30 | 60 | 80 | 80 | 30 | 0 | 40 | 60 |
| put pen into drawer | 60 | 60 | 90 | 100 | 70 | 50 | 70 | 80 |
| Wipe (3 tasks) | ||||||||
| wipe plate with rag | 20 | 20 | 30 | 70 | 0 | 0 | 10 | 60 |
| wipe stain with sponge | 30 | 40 | 40 | 80 | 0 | 0 | 20 | 50 |
| wipe table with eraser | 40 | 40 | 60 | 80 | 0 | 0 | 30 | 60 |
| Stack (7 tasks) | ||||||||
| stack cube on block | 0 | 10 | 20 | 40 | 0 | 0 | 0 | 10 |
| stack paper cups | 0 | 0 | 0 | 30 | 0 | 0 | 0 | 10 |
| stack bowls | 70 | 80 | 80 | 90 | 90 | 90 | 100 | 90 |
| stack coasters | 80 | 70 | 80 | 100 | 70 | 70 | 80 | 70 |
| stack cola cans | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 10 |
| cover cutlery box lid | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| cover trash can lid | 0 | 0 | 0 | 20 | 0 | 0 | 0 | 0 |
| Pass (5 tasks) | ||||||||
| pass remote control | 40 | 60 | 60 | 80 | 0 | 20 | 30 | 60 |
| pass orange ball | 70 | 70 | 80 | 100 | 0 | 0 | 0 | 0 |
| pass sponge eraser | 60 | 90 | 100 | 100 | 40 | 50 | 60 | 80 |
| pass rubik’s cube | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| pass cola can | 0 | 40 | 60 | 70 | 0 | 20 | 20 | 30 |
| Fold (4 tasks) | ||||||||
| fold towel (one arm) | 0 | 0 | 40 | 50 | 30 | 20 | 30 | 10 |
| fold towel (two arms) | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 0 |
| fold rag twice | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| unfold folded towel | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Straighten (5 tasks) | ||||||||
| straighten water bottle | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 10 |
| straighten water bottle (2) | 0 | 0 | 0 | 40 | 0 | 0 | 10 | 50 |
| straighten water cup | 0 | 0 | 0 | 60 | 0 | 0 | 0 | 40 |
| straighten water cup (2) | 0 | 0 | 0 | 30 | 0 | 0 | 0 | 0 |
| straighten soda drink | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 20 |
| Pour (5 tasks) | ||||||||
| pour fruits into plate | 20 | 40 | 50 | 70 | 30 | 20 | 20 | 30 |
| pour sesame into plate | 30 | 40 | 30 | 60 | 0 | 0 | 10 | 40 |
| pour sugar into cup | 10 | 20 | 30 | 50 | 0 | 0 | 0 | 10 |
| pour beans into cup | 0 | 20 | 40 | 50 | 0 | 0 | 10 | 10 |
| pour cubes into bowl | 0 | 0 | 0 | 40 | 0 | 30 | 0 | 10 |
| Push (8 tasks) | ||||||||
| push block to left | 30 | 40 | 0 | 70 | 0 | 0 | 0 | 10 |
| push cup to center | 0 | 20 | 30 | 60 | 0 | 20 | 30 | 50 |
| push box to left | 40 | 40 | 60 | 70 | 0 | 0 | 10 | 20 |
| push kettle forward | 30 | 20 | 20 | 50 | 0 | 60 | 20 | 30 |
| push book to edge | 0 | 0 | 0 | 30 | 0 | 0 | 0 | 0 |
| push tissue pack in | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| close the drawer | 20 | 50 | 60 | 70 | 0 | 0 | 0 | 0 |
| roll the bottle | 0 | 0 | 0 | 10 | 0 | 0 | 10 | 10 |
| Insert (6 tasks) | ||||||||
| insert bread into machine | 0 | 0 | 0 | 50 | 30 | 30 | 40 | 50 |
| insert flower into vase | 0 | 0 | 0 | 40 | 0 | 0 | 20 | 30 |
| insert straw into cup | 20 | 30 | 40 | 70 | 30 | 40 | 30 | 40 |
| insert plug into strip | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| take book from shelf | 0 | 0 | 0 | 60 | 0 | 0 | 20 | 50 |
| put book into shelf | 0 | 0 | 0 | 10 | 20 | 30 | 30 | 40 |
| Close (3 tasks) | ||||||||
| close tissue box lid | 10 | 30 | 40 | 70 | 0 | 50 | 40 | 40 |
| close blue box lid | 30 | 50 | 40 | 90 | 0 | 70 | 60 | 80 |
| close kettle lid | 30 | 60 | 90 | 100 | 20 | 30 | 30 | 40 |
| Separate (3 tasks) | ||||||||
| separate paper cups | 0 | 0 | 0 | 20 | 0 | 20 | 20 | 20 |
| uncap signature pen | 0 | 0 | 0 | 80 | 0 | 10 | 10 | 30 |
| unscrew bottle cap | 0 | 0 | 0 | 30 | 0 | 10 | 0 | 0 |
| Press (4 tasks) | ||||||||
| press lamp switch | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| press bread machine button | 0 | 0 | 10 | 30 | 0 | 0 | 0 | 0 |
| press stapler to staple | 0 | 0 | 10 | 30 | 0 | 0 | 10 | 20 |
| press calculator button | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 10 |
| Pull (6 tasks) | ||||||||
| pull off pen cap | 0 | 0 | 0 | 20 | 0 | 0 | 0 | 20 |
| pull tissue from bag | 10 | 30 | 30 | 80 | 0 | 0 | 30 | 40 |
| pull tray to front | 0 | 0 | 10 | 30 | 0 | 0 | 0 | 10 |
| pull cable from box | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| unplug charger from strip | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| open the drawer | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 0 |
| Hang (4 tasks) | ||||||||
| take towel off rod | 0 | 0 | 10 | 60 | 0 | 0 | 30 | 50 |
| hang towel on rod | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hang rag on box | 20 | 0 | 0 | 0 | 0 | 30 | 20 | 20 |
| hang hat on rack | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Tear (2 tasks) | ||||||||
| tear open envelope | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 10 |
| tear paper towel off roll | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Flip (6 tasks) | ||||||||
| close the open book | 0 | 50 | 70 | 90 | 0 | 0 | 20 | 30 |
| open the book | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| flip book to front | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| flip plate over | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| flip coaster to front | 60 | 70 | 60 | 80 | 0 | 60 | 70 | 90 |
| flip tissue pack over | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Sweep (3 tasks) | ||||||||
| sweep cubes into dustpan | 0 | 0 | 0 | 10 | 0 | 0 | 0 | 0 |
| sweep cubes by hand | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| sweep cubes with brush | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| Zip (3 tasks) | ||||||||
| zip up file bag | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| unzip file bag | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| unzip pencil case | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Shake/Stir (4 tasks) | ||||||||
| shake glass bottle | 0 | 0 | 10 | 20 | 0 | 40 | 40 | 60 |
| shake bottle to check | 0 | 0 | 10 | 30 | 0 | 30 | 40 | 60 |
| stir coffee with chopstick | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| stir water with spoon | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
Instruction-Following Prompts and Results
The appendix records exact issued instructions and distinguishes reaching the intended referent from completing the manipulation.
Protocol.
Thirty-three Pick and 26 Place instructions each receive five trials. Per-instruction scores are multiples of 20%; the overall 83.1% Follow Score and 72.9% success rate average all 295 trials.
Scenes.
Five scene families vary household-object identity, color, size, shape, and spatial arrangement. Horizontal rows test left/right and ordinal references; vertical layouts test near/far. Each scene supports multiple feasible choices but only one instruction-consistent referent.
Dimension mapping.
Size combines extreme and medium references; position combines left/right and near/far; order combines middle and ordinal expressions. The table retains each instruction verbatim, including the source’s minor wording irregularities.
| Dimension | Instruction | Follow | SR |
|---|---|---|---|
| Pick (33 instructions) | |||
| Object | pick up the white badminton ball with the right arm. | 100 | 80 |
| Object | pick up the yellow knife with the left arm. | 100 | 80 |
| Object | pick up the blue comb with the right arm. | 100 | 80 |
| Object | pick up the black mouse with the left arm. | 100 | 100 |
| Object | pick up the black tv controller with the right arm. | 100 | 100 |
| Object | pick up the toy bear with the left arm. | 100 | 60 |
| Object | pick up the yellow screwdriver with the right arm. | 100 | 80 |
| Object | pick up the green tape with the right arm. | 100 | 100 |
| Color | pick up the red fruit with the right arm. | 100 | 60 |
| Color | pick up the orange fruit with the right arm. | 100 | 100 |
| Color | pick up the yellow fruit with the right arm. | 100 | 100 |
| Color | pick up the green fruit with the right arm. | 100 | 100 |
| Size (extreme) | pick up the largest block with the right arm. | 100 | 100 |
| Size (extreme) | pick up the smallest block with the left arm. | 100 | 100 |
| Size (medium) | pick up the medium block with the right arm. | 0 | 0 |
| Size (extreme) | pick up the larger cup with the left arm. | 100 | 80 |
| Size (medium) | pick up the medium block with the left arm. | 100 | 100 |
| Size (extreme) | pick up the largest block with the right arm. | 100 | 100 |
| Size (extreme) | pick up the smallest block with the right arm. | 100 | 80 |
| Size (medium) | pick up the medium block with the right arm. | 60 | 40 |
| Shape | pick up the rectangular object with the right arm. | 100 | 20 |
| Shape | pick up the cylindrical object with right arm. | 60 | 60 |
| Shape | pick up the round object with right arm. | 100 | 100 |
| Shape | pick up the cylindrical object with left arm. | 100 | 100 |
| Position (left/right) | pick up the cup on the right side. | 80 | 20 |
| Order (ordinal) | Pick up the second cup from the right. | 0 | 0 |
| Order (middle) | Pick up the middle cup. | 40 | 40 |
| Order (ordinal) | Pick up the second cup from the left. | 0 | 0 |
| Position (left/right) | pick up the cup on the left side. | 100 | 100 |
| Position (left/right) | pick up the cup on the right of the table with right arm. | 100 | 100 |
| Position (left/right) | pick up the cup on the left of the table with left arm. | 100 | 100 |
| Position (near) | Pick up the cup on the near side of the table with the right arm. | 60 | 40 |
| Position (far) | pick up the farthest cup on the table with right arm. | 100 | 100 |
| Place (26 instructions) | |||
| Object | place the object into the plate with the right arm. | 100 | 100 |
| Object | place the object into the box with the right arm. | 100 | 100 |
| Object | place the object into the bowl with the right arm. | 100 | 80 |
| Object | place the object into the cup with the right arm. | 100 | 60 |
| Color | place the object into the blue plate with the right arm. | 100 | 100 |
| Color | place the object into the pink plate with the right arm. | 100 | 100 |
| Color | place the object into the orange plate with the right arm. | 100 | 100 |
| Color | place the object into the yellow plate with the right arm. | 100 | 100 |
| Size (extreme) | place the object into the smallest container with left arm. | 100 | 40 |
| Size (extreme) | place the object into the largest container with right arm. | 100 | 100 |
| Size (medium) | place the object into the medium container with right arm. | 0 | 0 |
| Size (medium) | place the object into the medium container with left arm. | 80 | 80 |
| Size (extreme) | place the object into the smallest container with right arm | 0 | 0 |
| Size (extreme) | place the object into the largest container with right arm. | 80 | 80 |
| Size (medium) | place the object into the medium container with right arm. | 100 | 100 |
| Shape | place the object into the rectangular container with the right arm. | 100 | 100 |
| Shape | place the objectinto the cylindrical container with the right arm | 80 | 40 |
| Shape | place the object into the round container with the right arm | 80 | 60 |
| Shape | place the object into the round container with the left arm | 100 | 100 |
| Position (left/right) | place the object into the container on the right side with the right arm. | 100 | 100 |
| Order (middle) | place the object into the middle container with the right arm. | 20 | 20 |
| Order (middle) | place the object into the middle container with the left arm. | 40 | 40 |
| Position (left/right) | place the object into the container on the left side with the left arm. | 100 | 100 |
| Position (far) | Place the object into the container at the far end of the table with the right arm. | 100 | 100 |
| Position (near) | Place the object into the container on the near side of the table with the right arm. | 100 | 60 |
| Order (ordinal) | place the object into the second container from the left. | 20 | 20 |
Descriptor frequencies.
Lexical rules compare 95,010 RefCOCOg expressions with about 3.7 million training instruction segments. Non-object categories may overlap; object identity is a residual category. Simulation is excluded from the training count except for size, whose 0.96% includes simulated superlative references. These differing definitions matter when comparing frequencies. [39]
| Dimension | Matching rule | RefCOCOg count | RefCOCOg (%) | GE corpus (%) |
|---|---|---|---|---|
| Object | Residual: no color, size, shape, position, or order descriptor | 39,474 | 41.5 | 58.3 |
| Color | Explicit color words; orange as a fruit name excluded | 42,161 | 44.4 | 29.6 |
| Size | large, small, big, tiny, medium and comparative/superlative forms | 5,299 | 5.6 | 0.96 |
| Shape | round, square, rectangular, cylindrical, triangular | 574 | 0.6 | 0.67 |
| Position | left, right, near, far; body-side phrases such as right hand excluded unless another spatial relation is present | 11,927 | 12.6 | 13.8 |
| Order | middle, center, ordinals, leftmost, rightmost; generic between excluded | 2,440 | 2.6 | 0.13 |