Literature
Display
← All papers

Transferring the Intelligence of VLMs to Robotic Control

Meng-Hao Guo · Zhe-Han Mo · Jia-Jun Wang · Yi Zhang · Kejin Wang · Yi-Xuan Deng · Jia-Peng Zhang · Yongming Rao · Shi-Min Hu

· arXiv (preprint) ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

RoboDawn uses a frozen VLM as a robot’s decision maker: observe the scene, reason, issue simple motion commands, and use execution feedback to decide again. One task demonstration raises reported success from 53.2% to 73.6% on RoboTwin 2.0 and from 35.67% to 47.17% on RoboDojo. These results require no task-specific parameter updates, but rely on a motion-planning interface and substantial inference time. Real-world tests are small: 9/10 block-in-basket, 5/10 stacking, and 0/10 cloth-folding successes.

1 Introduction

The paper asks whether pretrained visual reasoning can drive manipulation when robot actions are expressed in familiar spatial language. Instead of training an action head, RoboDawn keeps model weights fixed and supplies an interface, task instructions, observations, and examples. This complements VLA and WAM training: it tests high-level decision making, while engineered controllers still execute the physical motion. [25] [55] [48]

Refer to caption
Figure 1: RoboDawn transfers general-purpose VLM intelligence from the digital world to physical robot control through a human-intuitive action interface and in-context demonstrations.

2 Related Work

2.1 From Task-Specific Intelligence to Transferable Intelligence

Large pretrained language and vision-language models can adapt through instructions and examples across tasks that previously required separate models. RoboDawn investigates whether that reuse extends to physical control without robot-specific weight updates. [5] [30]

2.2 End-to-end Robotic Policy

End-to-end robot policies learn actions from observations; VLA models adapt visual-language representations, while WAMs combine future prediction with action generation. RoboDawn instead connects an unchanged VLM to executable motion primitives. [3] [40] [41]

2.3 Agentic Methods for Robotic Control

Earlier language-driven robots use planners, skill libraries, or executable programs; newer agent systems add memory and recovery around learned policies. Concurrent Show-Harness also controls robots through semantic commands. RoboDawn’s additional focus is matching demonstrations to the same command interface used online. [22] [29] [53] [11]

3 Robo Dawn

3.1 Overall Framework

Each decision uses annotated camera views, measured robot state, previous execution feedback, and a running memory, alongside a fixed environment profile and demonstrations. The VLM produces commands plus a progress estimate, plan, and scratchpad. Execution updates the physical scene and feedback; adaptation happens through this changing context. Online observations exclude privileged object poses, although a lower-level execution system still plans and performs motions.

Equation 1
(yt,𝐚t)\displaystyle(y_{t},\mathbf{a}_{t}) =πθ​(L,E,D,It,xt,Ft−1,Mt),\displaystyle=\pi_{\theta}\left(L,E,D;I_{t},x_{t},F_{t-1},M_{t}\right), (1)
(st+1,Ft)\displaystyle(s_{t+1},F_{t}) =ℰP​(st,𝐚t),\displaystyle=\mathcal{E}_{P}\left(s_{t},\mathbf{a}_{t}\right),
(It+1,xt+1)\displaystyle(I_{t+1},x_{t+1}) =𝒪P​(st+1),\displaystyle=\mathcal{O}_{P}(s_{t+1}),
Mt+1\displaystyle M_{t+1} =𝒰⁡(Mt,𝐚t,yt,Ft,xt+1).\displaystyle=\mathcal{U}\left(M_{t},\mathbf{a}_{t},y_{t},F_{t},x_{t+1}\right).
Refer to caption
Figure 2: Overview of RoboDawn. RoboDawn conditions a pretrained VLM on (a) a human-intuitive semantic action interface and (b) in-context demonstrations. At each interaction round, the VLM receives (c) multi-view visual observations and robot state together with task context, execution feedback, and memory, performs (d) reasoning, and generates (e) semantic action commands. The commands are grounded into (f) planned robot motions and executed in the environment; the resulting physical outcomes are then converted into (g) grounded feedback and memory updates for the next round. This loop enables continual observation, reasoning, execution, and adaptation without task-specific parameter updates.

3.2 Human-Intuitive Interface

The interface defines a GIP midway between the fingertips. Visual annotations, state reports, and commands all refer to this interaction point, avoiding confusion with the wrist position used by the controller.

Semantic action interface.

Commands translate or rotate either arm, choose common orientations, adjust the gripper, return home, wait, or request a completion check. Translation and rotation use world-frame axes and change the GIP incrementally. Per-command magnitudes are capped at 20 cm and 90°; rotating preserves the interaction point’s position. The original command grammar is retained below.

Equation 2
𝒜={\displaystyle\mathcal{A}=\{ <arm> move <axis> <d>,\displaystyle\texttt{<arm> move <axis> <d>}, (2)
<arm> rotate <rot> <θ>,\displaystyle\texttt{<arm> rotate <rot> <$\theta$>},
<arm> point <pose>,\displaystyle\texttt{<arm> point <pose>},
<arm> gripper <g>,\displaystyle\texttt{<arm> gripper <g>},
<arm> home,wait,done}.\displaystyle\texttt{<arm> home},~\texttt{wait},~\texttt{done}\}.

From semantic commands to physical motion.

The execution layer converts each command into a target interaction-point pose, plans a complete motion, and runs it until the robot is stationary. The VLM therefore chooses movements without generating joint trajectories or continuous low-level control.

3.3 In-Context Learning Design

Context contains a shared command primer and optional complete task demonstrations. The primer illustrates primitive effects, especially rotations; task examples show how to combine those primitives. “Zero-shot” means no task demonstration, so it still includes the shared primer.

Equations 3–4
D=Dprim⊕Dtask,D=D_{\mathrm{prim}}\oplus D_{\mathrm{task}}, (3)
Dtask={𝒟(m)}m=1NDD_{\mathrm{task}}=\{\mathcal{D}^{(m)}\}_{m=1}^{N_{D}} (4)

Interface-compatible demonstrations.

Expert trajectories are reduced to end-effector waypoints and gripper states, then translated into the same discrete commands available during execution. This makes examples directly actionable rather than presenting continuous controls the model cannot issue.

Complete in-context demonstrations.

Demonstrations come from separate scenes: scripted experts in simulation and teleoperation on physical robots. Each records images, robot state, commands, measured effects, and rationales added afterward by a VLM. Long RoboDojo demonstrations retain the full textual trajectory but sparsify images around informative events to fit the stated 16-observation image budget. Simulation examples are drawn from the trajectories used to train comparison policies.

Equation 5
𝒟(m)={(Ij(m),xj(m),rj(m),𝐚j(m),fj(m))}j=1Nm,\mathcal{D}^{(m)}=\left\{\left(I_{j}^{(m)},x_{j}^{(m)},r_{j}^{(m)},\mathbf{a}_{j}^{(m)},f_{j}^{(m)}\right)\right\}_{j=1}^{N_{m}}, (5)

4 Experiments

The evaluation covers two simulation benchmarks and three physical tasks. Comparisons contrast frozen-model prompting with policies post-trained on benchmark demonstrations; they establish performance under the reported protocols, without matching pretrained model resources or online computation. [10] [9]

Table 1: Evaluation on RoboTwin 2.0 C2R Chen et al. (2025). Full Set follows the official training protocol, where a single policy is jointly post-trained on all 50 benchmark tasks using 50 clean demonstrations per task. 1-Shot denotes using one randomly sampled demonstration from the clean set. Success rates (%) are reported under the domain-randomized evaluation setting. CC denotes Claude Code. HarnessVLA uses the pretrained LingBot-VLA as the default VLA.
  Method   Number of Shots   Success rate (%)
  FastWAM Yuan et al. (2026)   Full Set   1.9
  StarVLA  Community (2026)   Full Set   10.6
  GalaxeaVLA Jiang et al. (2025)   Full Set   12.7
  Xiaomi-Robotics-0 Cai et al. (2026)   Full Set   18.2
  GR00T-1.7Bjorck et al. (2025)   Full Set   20.7
  X-VLA Zheng et al. (2025)   Full Set   20.9
  X-WAM Guo et al. (2026)   Full Set   25.8
  π0.5\pi_{0.5} Intelligence et al. (2025)   Full Set   46.0
  LingBot-VLA Wu et al. (2026)   Full Set   50.4
  HarnessVLA (Codex) Zhang et al. (2026c)   Full Set   58.0
  HarnessVLA (CC) Zhang et al. (2026c)   Full Set   58.4
  RoboDawn(Ours, Gemini-3.8-Flash)   0   47.0
  RoboDawn(Ours, Gemini-3.8-Flash)   1   62.2
  RoboDawn(Ours, GPT-6 Astra)   0   53.2
  RoboDawn(Ours, GPT-6 Astra)   1   73.6

4.1 Simulation Benchmark Test

4.1.1 RoboTwin 2.0

RoboTwin’s C2R protocol has 50 bimanual tasks, with clean demonstrations and randomized evaluation scenes. The paper runs 10 evaluations per task. With GPT-6 Astra, RoboDawn scores 53.2% without task examples and 73.6% with one, compared with 46.0% for π0.5, 50.4% for LingBot-VLA, and 58.4% for HarnessVLA with Claude Code. The one-shot advantage over the latter is 15.2 percentage points. [23] [43] [53]

Ablation Study.

With Gemini 3.8 Flash, success rises from 47.0% without task examples to 62.2% with one and 65.4% with four, then falls to 62.7% with eight. This is consistent with diminishing returns from longer context. Removing reasoning lowers zero-shot success to 34.8%; removing localization grids lowers it to 32.4%; removing the primer gives 44.0%. One-shot performance also varies substantially across backbone models.


Model Infer. Actions Motion I/M
π0.5\pi_{0.5} 101 ms 45.0 steps 2.70 s 0.037
StarVLA 70 ms 16.0 steps 0.96 s 0.073
X-VLA 143 ms 28.6 steps 1.71 s 0.084
FastWAM 523 ms 27.3 steps 1.64 s 0.32
Motus 1.93 s 16.0 steps 0.96 s 2.01
LingBot-VA 8.89 s 22.2 steps 1.33 s 6.67
RoboDawn 9.74 s 3.4 cmds 2.09 s 4.65
Table 2: Inference and execution efficiency of representative VLA and WAM policies. Infer.: inference latency; Motion: motion execution time; I/M: inference-to-motion time ratio. Seed-2.1-Pro on Volcano Engine is used as the VLM of RoboDawn, considering network latency. cmds denotes commands.
Original figure
Figure 3: RoboDojo success rate under varying per-episode command budgets. RoboDawn exhibits clear test-time scaling.

Execution Efficiency.

Latency is measured with Seed-2.1-Pro, a different backbone from the best-success evaluation. RoboDawn averages 9.74 seconds of inference and 2.09 seconds of motion for 3.4 commands, an inference-to-motion ratio of 4.65. This is much slower than the fastest learned policies and does not demonstrate high-frequency control or streaming overlap.

4.1.2 RoboDojo

RoboDojo evaluates 42 tasks spanning generalization, memory, long sequences, precision, and open tasks, reporting averages over five runs. GPT-6 Astra reaches 35.67% success without task examples and 47.17% with one; task-progress scores are 39.92 and 54.63. Increasing the command budget from 60 to 240 improves one-shot success from 31.2% to 47.2%. The 34-task comparison excludes eight open tasks that may lack training examples for other policies. These gains use more interaction and inference, rather than a fixed execution budget. [9]

4.2 Real-World Robotics Deployment

Zero-shot physical tests use Gemini 3.8 Flash. A Franka robot succeeds in 9 of 10 block-in-basket trials and 5 of 10 stacking trials; a Piper robot succeeds in none of 10 cloth-folding trials. Near-complete but untidy folds count as failures. These limited tasks demonstrate transfer of the interface, with substantially weaker evidence for precise placement and deformable-object manipulation than for simple pick-and-place.

Table 3: Ablation studies of RoboDawn on RoboTwin 2.0.
(a) Number of shots (Gemini-3.8-Flash).
# Shots SR (%)
0 47.0
1 62.2
2 63.6
4 65.4
8 62.7
(b) Models. Results use 1-shot by default
Model SR (%)
GPT-5.6-Luna 14.4
GPT-5.6-Sol 43.2
Seed-2.1-Pro 45.0
Gemini-3.8-Flash 62.2
GPT-6 Astra 73.6
(c) Harness (Gemini 3.8 Flash, zero-shot).)
Harness SR (%)
Full 47.0
w/o reasoning 34.8
w/o grids 32.4
w/o primer 44.0
Table 4: Evaluation on RoboDojo Chen et al. (2026a). Full Set follows the official RoboDojo training protocol, in which a single policy is post-trained on the full RoboDojo-specific training set. Score denotes the task progress score. Compared with the full set of 42 tasks, the 34 tasks setting excludes 8 open tasks. For these 8 tasks, other methods may not have corresponding training data.
Method # Shots Score Success rate (%)
LingBot-VLA Wu et al. (2026) Full Set 5.50 2.96
X-VLA Zheng et al. (2025) Full Set 10.13 6.52
π0.5\pi_{0.5} Intelligence et al. (2025) Full Set 11.41 6.91
Hy-Embodied-0.5-VLA Zhang et al. (2026a) Full Set 13.07 8.80
Xiaomi-Robotics-1 Team et al. (2026b) Full Set 20.07 13.93
GalaxeaVLA (G0.5) Liu et al. (2026a) Full Set 20.23 14.88
DM0.5 Dexmal (2026) Full Set 24.90 19.34
GPT-6 Astra Zhang et al. (2026b) 0 28.97 22.58
RoboDawn(Ours, GPT-6 Astra, all tasks) 0 39.92 35.67
RoboDawn(Ours, GPT-6 Astra, all tasks) 1 54.63 47.17
RoboDawn(Ours, GPT-6 Astra, 34 tasks) 0 39.02 33.96
RoboDawn(Ours, GPT-6 Astra, 34 tasks) 1 51.29 43.33
Refer to caption
Figure 4: Representative failure cases of RoboDawn on RoboDojo.
Table 5: Real-world evaluation. Success rates on three real-world manipulation tasks.
Block in Basket (Franka) Block Stacking (Franka) Cloth Folding (Piper)
Success Rate 9/10 5/10 0/10

4.3 Failure Analysis.

Observed failures include correct high-level plans with inaccurate final positioning, IK-related motions that collide with nearby objects, and premature success declarations. The last case separates the model’s interpretation of “done” from the benchmark’s actual completion criteria.

5 Limitations

Main limitations are long inference delays, unreliable orientation reasoning, coarse corrections near contact, and collisions or other unsafe execution. Explanations based on pretraining’s lack of rotation examples are hypotheses. Strong simulation success and a few physical tasks do not establish robust control for unfamiliar robots, fine manipulation, or settings requiring fast reactions.

6 Conclusion

A suitable action interface and demonstrations can expose useful manipulation capabilities in frozen VLMs. The practical contribution is a reusable decision-and-feedback loop; faster inference, more precise motion, and reliable completion and collision handling remain necessary for broader deployment. The project site provides further demonstrations.

Author affiliations

The authors are affiliated with Tsinghua University and Tencent Hunyuan.

[Uncaptioned image]
Concept demonstration

Open image ↗