Literature
Display
← All papers

Learning Human Intention for Robotic Manipulation

Chengyang Li · Kaiyi Xiong · Yuan Xu · Lei Qian · Yizhou Wang · Wentao Zhu

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

GazeVLA uses a predicted gaze location as an intermediate control signal learned from egocentric human demonstrations.

Refer to caption
Figure 1: A human-to-robot learning framework that leverages human intention to facilitate robotic manipulation without requiring robot-side intention annotations. The model is first pre-trained on large-scale egocentric videos, followed by post-training on a limited set of robot and human demonstrations. We execute an intention-action reasoning chain during inference, enabling long-horizon, fine-grained tasks and robust generalization to unseen settings.

At a glance

The model learns to predict a two-dimensional gaze point before generating an action chunk, using human gaze as a proxy for task-relevant attention. Large-scale human pretraining is followed by joint human–robot post-training without robot gaze labels. Simulation and six physical tasks show improved average success and transfer to variations demonstrated only by humans. The method still needs robot and aligned on-task human data; gaze-guided control is an operational representation, not direct access to a person’s latent intentions.

1 Introduction

Human demonstrations offer scalable information about objects, task transitions, and manipulation, but human and robot actions differ substantially. The proposed bridge is gaze: a compact spatial signal that often anticipates interaction targets. A policy first predicts that signal and then conditions motor generation on it. The reported physical gain over π0.5 is about 12.1 percentage points on average, while success varies sharply across tasks and training inputs differ between baselines.

2 Related Work

Prior human-to-robot methods transfer visual features, keypoints, trajectories, latent actions, or predicted future observations. This work instead models gaze jointly with human motion and transfers the learned conditioning mechanism during robot training. Related systems include EgoMimic [34], H-RDT [7], and gaze-based Look, Focus, Act [20]. The intermediate sequence called chain-of-thought here consists of gaze-coordinate tokens rather than a natural-language explanation.

Refer to caption
Figure 2: We curate a large-scale egocentric dataset aggregated from heterogeneous sources, containing hand and gaze annotations paired with validity masks. The dataset features a unified coordinate system and covers diverse scenes, actions, and objects. Long videos are clipped and filtered into short segments, resulting in over 150M high-quality motion frames.

3 Method

The method standardizes heterogeneous human datasets, trains a gaze-and-action model, and adapts it with matched human and robot demonstrations.

3.1 Data

Thirteen egocentric datasets contribute more than 150 million frames with hand and gaze annotations where available. A video-language model segments interactions, unreliable gaze samples are filtered, and validity masks accommodate missing labels. Two hands’ fingertip positions, wrist positions, and six-dimensional rotations form a 48-dimensional motion representation; gaze is one image-plane point. On-task human demonstrations are additionally collected through eye-tracking glasses. The large frame count does not mean every frame has both valid labels or that all footage is task-matched.

Refer to caption
Figure 3: Model Architecture. VLIA model takes an egocentric image and language instructions as inputs, employing a Chain-of-Thought reasoning process to capture the synergy between intention and action. The model first leverages a VLM to predict gaze tokens in an autoregressive manner, followed by continuous action generation using flow matching. The intention is explicitly modeled via gaze as 2D image coordinates.

3.2 VLIA Model

A PaliGemma vision–language backbone autoregressively predicts discretized gaze coordinates, then a flow-matching action expert uses the resulting cached features together with state and observations to produce continuous actions. Training combines gaze-token cross-entropy and an action velocity-matching loss. The factorization makes the predicted spatial target available to motor generation, but the model’s chosen target remains a learned proxy for intention. The appendix describes the specific backbone as PaliGemma-2, whereas the main text names PaliGemma with Gemma-2B.

Equations
πθ(at:t+H,it|st,ot,l)=πθ(it|ot,l)⋅πθ(at:t+H|it,st,ot,l)\pi_{\theta}(a_{t:t+H},i_{t}|s_{t},o_{t},l)=\pi_{\theta}(i_{t}|o_{t},l)\cdot\pi_{\theta}(a_{t:t+H}|i_{t},s_{t},o_{t},l)
ℒintent=𝔼[−∑n=1Nintentlogπθ(itn|ot,l,it<n)]\mathcal{L}_{\text{intent}}=\mathbb{E}\left[-\sum_{n=1}^{N_{\text{intent}}}\log\pi_{\theta}(i^{n}_{t}|o_{t},l,i^{<n}_{t})\right]
ℒaction=𝔼[||πθ(at:t+Hτ,τ,c)−(at:t+H−ϵ)||22]\mathcal{L}_{\text{action}}=\mathbb{E}\left[||\pi_{\theta}(a^{\tau}_{t:t+H},\tau,c)-(a_{t:t+H}-\epsilon)||^{2}_{2}\right]
ℒ=λaction​ℒaction+λintent​ℒintent\mathcal{L}=\lambda_{\text{action}}\mathcal{L}_{\text{action}}+\lambda_{\text{intent}}\mathcal{L}_{\text{intent}}

3.3 Training

Human pretraining begins with the vision/language backbone frozen, then jointly optimizes the model. Images and gaze coordinates receive synchronized augmentation, and human action sequences are downsampled toward robot control rates. Post-training samples human and robot data equally, pads robot actions to the shared dimension, and normalizes each embodiment separately. Robot examples lack gaze labels, so transfer relies on the shared representation and human supervision rather than annotated robot intentions.

Refer to caption
Figure 4: (Top): Original samples from the dataset. (Bottom): Counterfactual samples with modified instructions while keeping the same visual input. The results demonstrate the model’s ability to perform task-dependent reasoning.

4 Experiments

Experiments measure human gaze/motion prediction, simulation performance under visual changes, physical task success, and the effects of intention supervision and additional human data.

4.1 Human Pre-training Evaluation

Human prediction errors are reported as 4.71 cm for hand keypoints and 12.31 degrees for wrist rotation. The paper gives gaze error as 4.8% of the image diagonal and also eleven pixels for a 224-by-224 image; those statements are inconsistent because 4.8% of that diagonal is approximately fifteen pixels. Qualitative instruction changes produce different gaze/action predictions from the same image. The text does not provide a complete held-out split or uncertainty protocol for these prediction metrics.

4.2 Simulation Experiments

The AV-ALOHA benchmark [19] uses two manipulation arms plus an active-camera arm. A single multitask policy trains on one hundred demonstrations per task and is evaluated over one hundred trials per task. Compared methods include LFA, H-RDT, and π0.5, with distractor and lighting shifts added at evaluation.

Rounded average success is 49% in distribution, 28% with distractors, and 27% with lighting changes, versus 41%, 22%, and 23% for π0.5. Averaging the two out-of-distribution conditions gives approximately a 22% relative gain. The method does not win every task: LFA leads in-distribution pouring and threading, and π0.5 leads pouring under altered lighting. Fine insertion and hooking remain difficult in absolute terms.

ID OOD-Distractors OOD-Lighting
Task \\backslash Method LFA HRDT π0.5\pi_{0.5} Ours LFA HRDT π0.5\pi_{0.5} Ours LFA HRDT π0.5\pi_{0.5} Ours
Cube Transfer 87 89 94 100 12 28 25 35 0 12 32 36
Hook Package 23 13 20 32 8 0 11 14 0 0 10 19
Peg Insertion 10 17 15 18 7 2 6 9 0 0 7 16
Pour Test Tube 41 24 34 39 11 14 24 28 0 3 25 22
Slot Insertion 43 42 54 60 23 19 47 56 0 11 44 50
Thread Needle 56 46 33 43 21 23 19 23 0 12 20 21
Average 43 39 41 49 14 14 22 28 0 6 23 27
Table 1: Quantitative comparison between our method and baseline methods including LFA [20], HRDT [7] and π0.5\pi_{0.5} [32] on the AV-ALOHA [19] benchmark. Our method achieves the best overall performance and demonstrates improved robustness across most out-of-distribution settings.
Refer to caption
Figure 5: Qualitative evaluation of our method on the AV-ALOHA [19] benchmark. The green cross indicates the predicted intention point.

4.3 Real-World Experiments

Physical tests use bimanual ALOHA and a Unitree G1 with dexterous hands. Tasks include collecting fruit, stowing a remote, stacking blocks, tightening a screw, transferring a bottle, and typing four keys. Each method/task receives two rounds of twenty trials; error bars summarize variation across those two rounds, not many independent training seeds.

The figure’s rounded successes for the proposed method are 68%, 43%, 62%, 12%, 85%, and 48%, compared with π0.5’s 55%, 30%, 52%, 5%, 68%, and 35%. The average gain is about 12.1 points, but screw tightening remains near 12%. Comparisons combine several differences: the proposed model and EgoMimic use human plus robot post-training, while H-RDT and π0.5 use robot data only; π0.5 also retains a different joint-angle action representation.

Original figure
Figure 6: Quantitative comparison between our method and baseline methods including EgoMimic [34], HRDT [7] and π0.5\pi_{0.5} [32] on real-world experiments. Our method outperforms the baseline methods in long-horizon, fine-grained, and dexterous manipulation.
Refer to caption
Figure 7: Qualitative analysis of our method on real-world experiments. The green cross indicates the predicted intention.

4.4 Ablation Studies

In pick-and-place ablations, full training succeeds in 19/20 in-domain trials and 6/10, 8/10, and 6/10 for position, object, and scene changes. Removing intention gives 16/20 and 5/10, 6/10, 5/10. Removing human pretraining or human post-training causes larger losses on several variants. These small trials support the components’ utility without proving a general causal model of human intention.

Generalization variants are absent from robot demonstrations but present in on-task human demonstrations. A separate collection-time study favors adding human data, partly because ten minutes yields roughly one hundred human trajectories versus twenty teleoperated robot trajectories. At inference, predicted gaze and measured human gaze each give 65% fruit-task success, versus 50% for generic VLM keypoints, 40% with intention masked, and 25% with random points. This demonstrates dependence on the conditioning signal, while also exposing sensitivity to incorrect guidance.

Method ID OOD-Position OOD-Object OOD-Scene
Ours 19/20 6/10 8/10 6/10
Ours w/o Intention 16/20 5/10 6/10 5/10
Ours w/o Human Pretrain 13/20 1/10 2/10 0/10
Ours w/o Human Finetune 17/20 3/10 5/10 2/10
Table 2: Ablation studies on pick-and-place task. We evaluate generalization across position shifts, object categories, and scene variations. The results demonstrate that intention modeling consistently improves manipulation performance, while human data play a significant role in both the pre-training and post-training stages.
Refer to caption
Figure 8: Representative failure cases of Ours w/o Intention. Fine-grained manipulation tasks frequently exhibit action misalignment and execution failures without intention reasoning, highlighting the benefit of intention guidance.
Original figure
Figure 9: Comparison between human and robot data scaling on Bottle Transferring task. With the same initial amount of robot data, adding human data yields substantially larger performance gains than adding robot data.
Intention Source Success Rate
Predicted (Ours) 65%
Human Gaze 65%
VLM Keypoint 50%
No Intention 40%
Random Noise 25%
Table 3: Comparison of different intention sources.

5 Conclusion, Limitations and Future Directions

Gaze prediction can provide a useful shared spatial signal for human–robot learning and improve the evaluated tasks. The approach still requires task adaptation, aligned human examples, and robot demonstrations, so it is not zero-shot human-to-robot transfer. Sequential gaze prediction adds latency and can misdirect actions when its target conflicts with the instruction. More efficient inference and broader transfer remain future work.

Acknowledgments

The acknowledgments identify Zhejiang research funding and computing support from institutions in Ningbo.

Appendix A Data

The data appendix explains on-task capture, instruction-guided preprocessing, label validity, and alignment of motion representations.

A.1 Data Collection

Pupil glasses collect images, gaze, and camera calibration; hand motion is reconstructed with a bimanual capture pipeline and temporally filtered. Participants are explicitly asked to act in robot-like ways and attend to task-relevant objects. Task configurations and viewpoints are aligned with the robot before introducing new placements, objects, and backgrounds. This careful collection protocol helps transfer and limits claims about unrestricted internet video as interchangeable supervision.

A.2 Data Processing

A Qwen2.5 video-language model processes batches of thirty frames sampled at one hertz, assigning atomic-action boundaries and verb–noun descriptions. Heuristics remove saccades and low-confidence gaze, then cluster nearby fixations. Missing left/right hand or gaze labels are masked from supervision. Camera/wrist axes are manually standardized; keypoints support sources without MANO fits. End-effector pose plus gripper or dexterous-hand state represents robot action, zero-padded to 48 dimensions and separately normalized. The pipeline reduces representational mismatch but does not establish a unique physical correspondence between human and robot motion.

Appendix B Model

The architecture appendix specifies token ordering, gaze quantization, optimization, and spatial augmentation.

Original figure
Figure 10: Block-wise structured attention.

B.1 Model Architecture

The model uses one padded/resized 224-by-224 image and a flow-based action expert. Four attention blocks cover language/image, intention, state, and action; tokens attend bidirectionally within each block but cannot access future blocks. An intention encoder/decoder refines features and predicts spatial probabilities. The complete original attention diagram is retained.

Table 4: Hyperparameters for pre-training and post-training.
Hyperparameter    Pre-Training    Post-Training
Step    20000    30000
Batch Size    2048    32
Learning Rate    5e-5    1e-4
Learning Rate Scheduling    Constant    Cosine
Gradient Accumulation Steps    4    1
λaction\lambda_{\text{action}}    1    1
λintent\lambda_{\text{intent}}    0.1    0.1

B.2 Implementation Details

Human pretraining runs for twenty thousand steps at batch size 2,048 on eight A800 GPUs, with a reported 1,344 total GPU-hours. Post-training uses thirty thousand steps and batch size 32. The table weights action loss by one and intention loss by 0.1, with constant then cosine learning-rate schedules. Gaze uses 224 bins along each image axis and a dedicated linear decoder. Joint crops, translations, and rotations compensate for central human gaze and more dispersed robot-camera targets. The precise warm-up duration is not specified.

Appendix C Simulation Experiments

Simulation details define a shared multitask protocol and provide additional examples of attention-guided manipulation.

C.1 Training Configuration

All six tasks are trained jointly, with one hundred trajectories each. Control runs at 25 frames per second, and each prediction contains 25 actions executed open-loop for one second before the next inference. The simulation action vector has 21 dimensions, including both arms and active camera motion. LFA trains for one hundred thousand steps, while H-RDT and π0.5 train for thirty thousand with their stated batch sizes and dimensional adaptations. Equal demonstrations do not imply identical optimization budgets or inherited pretraining.

C.2 More Results

Additional sequences show attention near the needle opening and package hook, including distractor and lighting conditions. These examples illustrate the proposed spatial mechanism but do not establish universal robustness, especially given the low success rates on some precision tasks.

Refer to caption
Figure 11: More qualitative evaluation of our method in simulation. The green cross indicates the predicted intention point.

Appendix D Real-World Experiments

The physical appendix provides task definitions, collected demonstration counts, baseline adaptations, and additional views of execution.

D.1 Task Description

Demonstration counts vary by task: fruit collecting and screw tightening use one hundred robot and two hundred human examples; remote stowing and block stacking use fifty and one hundred; bottle transfer and typing use twenty and one hundred. Every method/task has forty evaluation trials. Success criteria also vary: for example, remote stowing is counted by the remote being inside the drawer, although the described sequence includes closing it. Complete sequence descriptions should therefore be distinguished from the actual success check.

Table 5: Real-World Task Descriptions and Evaluation Protocols
Task Name Task Description Evaluation Protocols
Fruit Collecting Pick up all fruits on the table sequentially and place them into the plate. 100 robot data; 200 human data; 2 rounds of 20 trials (40 total); success if all fruits are placed in the bowl.
Remote Stowing Open the drawer, pick up the remote control, place it into the drawer, and close the drawer. 50 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the remote control is inside the drawer.
Block Stacking First place the red block in the center, then stack the yellow block on top of it. 50 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the yellow block is on top of the red block.
Screw Tightening The left arm picks up the screwdriver and aligns it with the screw, while the right arm presses the electric screwdriver switch. 100 robot data; 200 human data; 2 rounds of 20 trials (40 total); success if the screw is tightened into the wooden block.
Bottle Transferring Use the dexterous hand to pick up the bottle and place it onto the plate. 20 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the bottle is stably placed on the plate.
Keyboard Typing Press the four letters “V”, “L”, “I”, and “A” sequentially on the keyboard. 20 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the screen displays “VLIA”.

D.2 Training Configuration

The proposed method and EgoMimic jointly post-train on human and robot data. H-RDT and π0.5 use only robot demonstrations under the implementations chosen by the authors. The latter uses joint-angle actions for compatibility; the other methods use the shared wrist/end-effector representation. These are material comparison differences rather than an isolated test of gaze conditioning alone.

D.3 More Results

Additional physical sequences retain synchronized third-person and egocentric views, showing predicted target points during navigation of task stages and fine manipulation. They complement the success statistics rather than replace failure-rate measurements.

Refer to caption
Figure 12: More qualitative evaluation of our method in real-world setting. The green cross indicates the predicted intention point. The left column shows the third-person view, while the right column presents the egocentric first-person view.
Refer to caption
Figure 13: Generalization evaluation setup on pick-and-place task. The orange circle indicates the initial object positions.

Appendix E Results

Further results separate adaptation to new visual settings, collection-time efficiency, alternate guidance sources, intervention behavior, and runtime.

E.1 Human Data Enable Robot to Generalize to Unseen Positions, Objects, and Scenes

Robot pick-and-place data contain only a strawberry within a five-centimeter placement radius. Evaluation expands the radius to eight centimeters, substitutes lemons or cherries, or adds a white tablecloth. Human adaptation data include these new settings, so this is cross-embodiment transfer of observed variations rather than generalization to variations unseen by both training streams.

[Uncaptioned image]
Figure 14: Evaluation protocols on Bottle Transferring task.
Table 6: Comparison between human and robot data scaling on Bottle Transferring task.
Data ID OOD
10m robot 4/5 9/15
20m robot 4/5 12/15
10m human + 10m robot 5/5 12/15
30m robot 5/5 12/15
20m human + 10m robot 5/5 13/15

E.2 Human Data Scales Robot Performance More Efficiently

The bottle study holds an initial ten minutes of robot data fixed, then adds human or robot collection time. Ten extra human minutes yield five of five in-domain and twelve of fifteen out-of-domain successes, matching thirty total robot minutes on these counts; twenty extra human minutes reach thirteen of fifteen out-of-domain successes. The twenty-minute robot condition has the same twelve of fifteen out-of-domain successes but four of five in-domain. Thus the apparent efficiency advantage depends partly on the small in-domain subset and on collecting many more human trajectories per minute. New bottle colors/geometries appear only in human demonstrations.

E.3 Effect of Different Intention Sources

Alternate inference inputs include model-predicted gaze, synchronized human gaze, VLM-generated task keypoints, a masked intention path, and random spatial points. The measured 65/65/50/40/25 percent ranking supports useful spatial conditioning. It does not isolate anticipatory timing as the sole cause of the gaze advantage, since the sources differ in more than timing and the intervention changes model inputs at inference.

Refer to caption
Figure 15: We evaluated the model’s adherence to intention. During reasoning in the Fruit Collecting task, we manually modified the predicted intention, and the results showed that the actions followed the guidance of the intention and grasped the corresponding objects. The green crosses denote the model predicted intentions, while the red crosses indicate the intervened intentions.

E.4 Synergy Between Intention and Action

Manually moving the intention point changes which fruit the policy grasps, directly demonstrating that this internal input affects action selection. Gaze transitions also track stages of typing, drawer use, and tool alignment in the shown sequences. This intervention establishes reliance on the supplied point within the model; it does not establish that the point completely represents a human’s latent intention.

E.5 Tradeoff between Sensitivity and Robustness on Intention Guidance

Strong sensitivity helps follow informative target points but makes the policy vulnerable when gaze is wrong or contradicts language. Performance drops when a left-target instruction is paired with a right-target point or random noise. The authors report that loss weighting adjusts this tradeoff, without a quantified calibration or recovery guarantee. Robustness to erroneous predicted intentions remains a practical limitation.

E.6 Inference Latency

On an Intel i9-14900KF and RTX 4090 in the ten-hertz setting, inference takes 44.384 ms versus 32.205 ms for direct action prediction, adding 12.179 ms. This is a specific model/hardware measurement, separate from the simulation’s one-second open-loop chunks and from total perception-to-actuation latency.

Concept demonstration

Open image ↗