Overview
At a glance
RoboDawn uses a frozen VLM as a robot’s decision maker: observe the scene, reason, issue simple motion commands, and use execution feedback to decide again. One task demonstration raises reported success from 53.2% to 73.6% on RoboTwin 2.0 and from 35.67% to 47.17% on RoboDojo. These results require no task-specific parameter updates, but rely on a motion-planning interface and substantial inference time. Real-world tests are small: 9/10 block-in-basket, 5/10 stacking, and 0/10 cloth-folding successes.
1 Introduction
The paper asks whether pretrained visual reasoning can drive manipulation when robot actions are expressed in familiar spatial language. Instead of training an action head, RoboDawn keeps model weights fixed and supplies an interface, task instructions, observations, and examples. This complements VLA and WAM training: it tests high-level decision making, while engineered controllers still execute the physical motion. [25] [55] [48]
2 Related Work
2.1 From Task-Specific Intelligence to Transferable Intelligence
Large pretrained language and vision-language models can adapt through instructions and examples across tasks that previously required separate models. RoboDawn investigates whether that reuse extends to physical control without robot-specific weight updates. [5] [30]
2.2 End-to-end Robotic Policy
End-to-end robot policies learn actions from observations; VLA models adapt visual-language representations, while WAMs combine future prediction with action generation. RoboDawn instead connects an unchanged VLM to executable motion primitives. [3] [40] [41]
2.3 Agentic Methods for Robotic Control
Earlier language-driven robots use planners, skill libraries, or executable programs; newer agent systems add memory and recovery around learned policies. Concurrent Show-Harness also controls robots through semantic commands. RoboDawn’s additional focus is matching demonstrations to the same command interface used online. [22] [29] [53] [11]
3 Robo Dawn
3.1 Overall Framework
Each decision uses annotated camera views, measured robot state, previous execution feedback, and a running memory, alongside a fixed environment profile and demonstrations. The VLM produces commands plus a progress estimate, plan, and scratchpad. Execution updates the physical scene and feedback; adaptation happens through this changing context. Online observations exclude privileged object poses, although a lower-level execution system still plans and performs motions.
Equation 1
| (1) | ||||
3.2 Human-Intuitive Interface
The interface defines a GIP midway between the fingertips. Visual annotations, state reports, and commands all refer to this interaction point, avoiding confusion with the wrist position used by the controller.
Semantic action interface.
Commands translate or rotate either arm, choose common orientations, adjust the gripper, return home, wait, or request a completion check. Translation and rotation use world-frame axes and change the GIP incrementally. Per-command magnitudes are capped at 20 cm and 90°; rotating preserves the interaction point’s position. The original command grammar is retained below.
Equation 2
| (2) | ||||
From semantic commands to physical motion.
The execution layer converts each command into a target interaction-point pose, plans a complete motion, and runs it until the robot is stationary. The VLM therefore chooses movements without generating joint trajectories or continuous low-level control.
3.3 In-Context Learning Design
Context contains a shared command primer and optional complete task demonstrations. The primer illustrates primitive effects, especially rotations; task examples show how to combine those primitives. “Zero-shot” means no task demonstration, so it still includes the shared primer.
Equations 3–4
| (3) |
| (4) |
Interface-compatible demonstrations.
Expert trajectories are reduced to end-effector waypoints and gripper states, then translated into the same discrete commands available during execution. This makes examples directly actionable rather than presenting continuous controls the model cannot issue.
Complete in-context demonstrations.
Demonstrations come from separate scenes: scripted experts in simulation and teleoperation on physical robots. Each records images, robot state, commands, measured effects, and rationales added afterward by a VLM. Long RoboDojo demonstrations retain the full textual trajectory but sparsify images around informative events to fit the stated 16-observation image budget. Simulation examples are drawn from the trajectories used to train comparison policies.
Equation 5
| (5) |
4 Experiments
The evaluation covers two simulation benchmarks and three physical tasks. Comparisons contrast frozen-model prompting with policies post-trained on benchmark demonstrations; they establish performance under the reported protocols, without matching pretrained model resources or online computation. [10] [9]
| Method | Number of Shots | Success rate (%) |
|---|---|---|
| FastWAM Yuan et al. (2026) | Full Set | 1.9 |
| StarVLA Community (2026) | Full Set | 10.6 |
| GalaxeaVLA Jiang et al. (2025) | Full Set | 12.7 |
| Xiaomi-Robotics-0 Cai et al. (2026) | Full Set | 18.2 |
| GR00T-1.7Bjorck et al. (2025) | Full Set | 20.7 |
| X-VLA Zheng et al. (2025) | Full Set | 20.9 |
| X-WAM Guo et al. (2026) | Full Set | 25.8 |
| Intelligence et al. (2025) | Full Set | 46.0 |
| LingBot-VLA Wu et al. (2026) | Full Set | 50.4 |
| HarnessVLA (Codex) Zhang et al. (2026c) | Full Set | 58.0 |
| HarnessVLA (CC) Zhang et al. (2026c) | Full Set | 58.4 |
| RoboDawn(Ours, Gemini-3.8-Flash) | 0 | 47.0 |
| RoboDawn(Ours, Gemini-3.8-Flash) | 1 | 62.2 |
| RoboDawn(Ours, GPT-6 Astra) | 0 | 53.2 |
| RoboDawn(Ours, GPT-6 Astra) | 1 | 73.6 |
4.1 Simulation Benchmark Test
4.1.1 RoboTwin 2.0
RoboTwin’s C2R protocol has 50 bimanual tasks, with clean demonstrations and randomized evaluation scenes. The paper runs 10 evaluations per task. With GPT-6 Astra, RoboDawn scores 53.2% without task examples and 73.6% with one, compared with 46.0% for π0.5, 50.4% for LingBot-VLA, and 58.4% for HarnessVLA with Claude Code. The one-shot advantage over the latter is 15.2 percentage points. [23] [43] [53]
Ablation Study.
With Gemini 3.8 Flash, success rises from 47.0% without task examples to 62.2% with one and 65.4% with four, then falls to 62.7% with eight. This is consistent with diminishing returns from longer context. Removing reasoning lowers zero-shot success to 34.8%; removing localization grids lowers it to 32.4%; removing the primer gives 44.0%. One-shot performance also varies substantially across backbone models.
| Model | Infer. | Actions | Motion | I/M |
|---|---|---|---|---|
| 101 ms | 45.0 steps | 2.70 s | 0.037 | |
| StarVLA | 70 ms | 16.0 steps | 0.96 s | 0.073 |
| X-VLA | 143 ms | 28.6 steps | 1.71 s | 0.084 |
| FastWAM | 523 ms | 27.3 steps | 1.64 s | 0.32 |
| Motus | 1.93 s | 16.0 steps | 0.96 s | 2.01 |
| LingBot-VA | 8.89 s | 22.2 steps | 1.33 s | 6.67 |
| RoboDawn | 9.74 s | 3.4 cmds | 2.09 s | 4.65 |
Execution Efficiency.
Latency is measured with Seed-2.1-Pro, a different backbone from the best-success evaluation. RoboDawn averages 9.74 seconds of inference and 2.09 seconds of motion for 3.4 commands, an inference-to-motion ratio of 4.65. This is much slower than the fastest learned policies and does not demonstrate high-frequency control or streaming overlap.
4.1.2 RoboDojo
RoboDojo evaluates 42 tasks spanning generalization, memory, long sequences, precision, and open tasks, reporting averages over five runs. GPT-6 Astra reaches 35.67% success without task examples and 47.17% with one; task-progress scores are 39.92 and 54.63. Increasing the command budget from 60 to 240 improves one-shot success from 31.2% to 47.2%. The 34-task comparison excludes eight open tasks that may lack training examples for other policies. These gains use more interaction and inference, rather than a fixed execution budget. [9]
4.2 Real-World Robotics Deployment
Zero-shot physical tests use Gemini 3.8 Flash. A Franka robot succeeds in 9 of 10 block-in-basket trials and 5 of 10 stacking trials; a Piper robot succeeds in none of 10 cloth-folding trials. Near-complete but untidy folds count as failures. These limited tasks demonstrate transfer of the interface, with substantially weaker evidence for precise placement and deformable-object manipulation than for simple pick-and-place.
| # Shots | SR (%) |
|---|---|
| 0 | 47.0 |
| 1 | 62.2 |
| 2 | 63.6 |
| 4 | 65.4 |
| 8 | 62.7 |
| Model | SR (%) |
|---|---|
| GPT-5.6-Luna | 14.4 |
| GPT-5.6-Sol | 43.2 |
| Seed-2.1-Pro | 45.0 |
| Gemini-3.8-Flash | 62.2 |
| GPT-6 Astra | 73.6 |
| Method | # Shots | Score | Success rate (%) |
|---|---|---|---|
| LingBot-VLA Wu et al. (2026) | Full Set | 5.50 | 2.96 |
| X-VLA Zheng et al. (2025) | Full Set | 10.13 | 6.52 |
| Intelligence et al. (2025) | Full Set | 11.41 | 6.91 |
| Hy-Embodied-0.5-VLA Zhang et al. (2026a) | Full Set | 13.07 | 8.80 |
| Xiaomi-Robotics-1 Team et al. (2026b) | Full Set | 20.07 | 13.93 |
| GalaxeaVLA (G0.5) Liu et al. (2026a) | Full Set | 20.23 | 14.88 |
| DM0.5 Dexmal (2026) | Full Set | 24.90 | 19.34 |
| GPT-6 Astra Zhang et al. (2026b) | 0 | 28.97 | 22.58 |
| RoboDawn(Ours, GPT-6 Astra, all tasks) | 0 | 39.92 | 35.67 |
| RoboDawn(Ours, GPT-6 Astra, all tasks) | 1 | 54.63 | 47.17 |
| RoboDawn(Ours, GPT-6 Astra, 34 tasks) | 0 | 39.02 | 33.96 |
| RoboDawn(Ours, GPT-6 Astra, 34 tasks) | 1 | 51.29 | 43.33 |
| Block in Basket (Franka) | Block Stacking (Franka) | Cloth Folding (Piper) | |
| Success Rate | 9/10 | 5/10 | 0/10 |
4.3 Failure Analysis.
Observed failures include correct high-level plans with inaccurate final positioning, IK-related motions that collide with nearby objects, and premature success declarations. The last case separates the model’s interpretation of “done” from the benchmark’s actual completion criteria.
5 Limitations
Main limitations are long inference delays, unreliable orientation reasoning, coarse corrections near contact, and collisions or other unsafe execution. Explanations based on pretraining’s lack of rotation examples are hypotheses. Strong simulation success and a few physical tasks do not establish robust control for unfamiliar robots, fine manipulation, or settings requiring fast reactions.
6 Conclusion
A suitable action interface and demonstrations can expose useful manipulation capabilities in frozen VLMs. The practical contribution is a reusable decision-and-feedback loop; faster inference, more precise motion, and reliable completion and collision handling remain necessary for broader deployment. The project site provides further demonstrations.
Author affiliations
The authors are affiliated with Tsinghua University and Tencent Hunyuan.