Overview
GazeVLA uses a predicted gaze location as an intermediate control signal learned from egocentric human demonstrations.
At a glance
The model learns to predict a two-dimensional gaze point before generating an action chunk, using human gaze as a proxy for task-relevant attention. Large-scale human pretraining is followed by joint human–robot post-training without robot gaze labels. Simulation and six physical tasks show improved average success and transfer to variations demonstrated only by humans. The method still needs robot and aligned on-task human data; gaze-guided control is an operational representation, not direct access to a person’s latent intentions.
1 Introduction
Human demonstrations offer scalable information about objects, task transitions, and manipulation, but human and robot actions differ substantially. The proposed bridge is gaze: a compact spatial signal that often anticipates interaction targets. A policy first predicts that signal and then conditions motor generation on it. The reported physical gain over π0.5 is about 12.1 percentage points on average, while success varies sharply across tasks and training inputs differ between baselines.
2 Related Work
Prior human-to-robot methods transfer visual features, keypoints, trajectories, latent actions, or predicted future observations. This work instead models gaze jointly with human motion and transfers the learned conditioning mechanism during robot training. Related systems include EgoMimic [34], H-RDT [7], and gaze-based Look, Focus, Act [20]. The intermediate sequence called chain-of-thought here consists of gaze-coordinate tokens rather than a natural-language explanation.
3 Method
The method standardizes heterogeneous human datasets, trains a gaze-and-action model, and adapts it with matched human and robot demonstrations.
3.1 Data
Thirteen egocentric datasets contribute more than 150 million frames with hand and gaze annotations where available. A video-language model segments interactions, unreliable gaze samples are filtered, and validity masks accommodate missing labels. Two hands’ fingertip positions, wrist positions, and six-dimensional rotations form a 48-dimensional motion representation; gaze is one image-plane point. On-task human demonstrations are additionally collected through eye-tracking glasses. The large frame count does not mean every frame has both valid labels or that all footage is task-matched.
3.2 VLIA Model
A PaliGemma vision–language backbone autoregressively predicts discretized gaze coordinates, then a flow-matching action expert uses the resulting cached features together with state and observations to produce continuous actions. Training combines gaze-token cross-entropy and an action velocity-matching loss. The factorization makes the predicted spatial target available to motor generation, but the model’s chosen target remains a learned proxy for intention. The appendix describes the specific backbone as PaliGemma-2, whereas the main text names PaliGemma with Gemma-2B.
Equations
3.3 Training
Human pretraining begins with the vision/language backbone frozen, then jointly optimizes the model. Images and gaze coordinates receive synchronized augmentation, and human action sequences are downsampled toward robot control rates. Post-training samples human and robot data equally, pads robot actions to the shared dimension, and normalizes each embodiment separately. Robot examples lack gaze labels, so transfer relies on the shared representation and human supervision rather than annotated robot intentions.
4 Experiments
Experiments measure human gaze/motion prediction, simulation performance under visual changes, physical task success, and the effects of intention supervision and additional human data.
4.1 Human Pre-training Evaluation
Human prediction errors are reported as 4.71 cm for hand keypoints and 12.31 degrees for wrist rotation. The paper gives gaze error as 4.8% of the image diagonal and also eleven pixels for a 224-by-224 image; those statements are inconsistent because 4.8% of that diagonal is approximately fifteen pixels. Qualitative instruction changes produce different gaze/action predictions from the same image. The text does not provide a complete held-out split or uncertainty protocol for these prediction metrics.
4.2 Simulation Experiments
The AV-ALOHA benchmark [19] uses two manipulation arms plus an active-camera arm. A single multitask policy trains on one hundred demonstrations per task and is evaluated over one hundred trials per task. Compared methods include LFA, H-RDT, and π0.5, with distractor and lighting shifts added at evaluation.
Rounded average success is 49% in distribution, 28% with distractors, and 27% with lighting changes, versus 41%, 22%, and 23% for π0.5. Averaging the two out-of-distribution conditions gives approximately a 22% relative gain. The method does not win every task: LFA leads in-distribution pouring and threading, and π0.5 leads pouring under altered lighting. Fine insertion and hooking remain difficult in absolute terms.
| ID | OOD-Distractors | OOD-Lighting | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Task Method | LFA | HRDT | Ours | LFA | HRDT | Ours | LFA | HRDT | Ours | |||
| Cube Transfer | 87 | 89 | 94 | 100 | 12 | 28 | 25 | 35 | 0 | 12 | 32 | 36 |
| Hook Package | 23 | 13 | 20 | 32 | 8 | 0 | 11 | 14 | 0 | 0 | 10 | 19 |
| Peg Insertion | 10 | 17 | 15 | 18 | 7 | 2 | 6 | 9 | 0 | 0 | 7 | 16 |
| Pour Test Tube | 41 | 24 | 34 | 39 | 11 | 14 | 24 | 28 | 0 | 3 | 25 | 22 |
| Slot Insertion | 43 | 42 | 54 | 60 | 23 | 19 | 47 | 56 | 0 | 11 | 44 | 50 |
| Thread Needle | 56 | 46 | 33 | 43 | 21 | 23 | 19 | 23 | 0 | 12 | 20 | 21 |
| Average | 43 | 39 | 41 | 49 | 14 | 14 | 22 | 28 | 0 | 6 | 23 | 27 |
4.3 Real-World Experiments
Physical tests use bimanual ALOHA and a Unitree G1 with dexterous hands. Tasks include collecting fruit, stowing a remote, stacking blocks, tightening a screw, transferring a bottle, and typing four keys. Each method/task receives two rounds of twenty trials; error bars summarize variation across those two rounds, not many independent training seeds.
The figure’s rounded successes for the proposed method are 68%, 43%, 62%, 12%, 85%, and 48%, compared with π0.5’s 55%, 30%, 52%, 5%, 68%, and 35%. The average gain is about 12.1 points, but screw tightening remains near 12%. Comparisons combine several differences: the proposed model and EgoMimic use human plus robot post-training, while H-RDT and π0.5 use robot data only; π0.5 also retains a different joint-angle action representation.
4.4 Ablation Studies
In pick-and-place ablations, full training succeeds in 19/20 in-domain trials and 6/10, 8/10, and 6/10 for position, object, and scene changes. Removing intention gives 16/20 and 5/10, 6/10, 5/10. Removing human pretraining or human post-training causes larger losses on several variants. These small trials support the components’ utility without proving a general causal model of human intention.
Generalization variants are absent from robot demonstrations but present in on-task human demonstrations. A separate collection-time study favors adding human data, partly because ten minutes yields roughly one hundred human trajectories versus twenty teleoperated robot trajectories. At inference, predicted gaze and measured human gaze each give 65% fruit-task success, versus 50% for generic VLM keypoints, 40% with intention masked, and 25% with random points. This demonstrates dependence on the conditioning signal, while also exposing sensitivity to incorrect guidance.
| Method | ID | OOD-Position | OOD-Object | OOD-Scene |
|---|---|---|---|---|
| Ours | 19/20 | 6/10 | 8/10 | 6/10 |
| Ours w/o Intention | 16/20 | 5/10 | 6/10 | 5/10 |
| Ours w/o Human Pretrain | 13/20 | 1/10 | 2/10 | 0/10 |
| Ours w/o Human Finetune | 17/20 | 3/10 | 5/10 | 2/10 |
| Intention Source | Success Rate |
|---|---|
| Predicted (Ours) | 65% |
| Human Gaze | 65% |
| VLM Keypoint | 50% |
| No Intention | 40% |
| Random Noise | 25% |
5 Conclusion, Limitations and Future Directions
Gaze prediction can provide a useful shared spatial signal for human–robot learning and improve the evaluated tasks. The approach still requires task adaptation, aligned human examples, and robot demonstrations, so it is not zero-shot human-to-robot transfer. Sequential gaze prediction adds latency and can misdirect actions when its target conflicts with the instruction. More efficient inference and broader transfer remain future work.
Acknowledgments
The acknowledgments identify Zhejiang research funding and computing support from institutions in Ningbo.
Appendix A Data
The data appendix explains on-task capture, instruction-guided preprocessing, label validity, and alignment of motion representations.
A.1 Data Collection
Pupil glasses collect images, gaze, and camera calibration; hand motion is reconstructed with a bimanual capture pipeline and temporally filtered. Participants are explicitly asked to act in robot-like ways and attend to task-relevant objects. Task configurations and viewpoints are aligned with the robot before introducing new placements, objects, and backgrounds. This careful collection protocol helps transfer and limits claims about unrestricted internet video as interchangeable supervision.
A.2 Data Processing
A Qwen2.5 video-language model processes batches of thirty frames sampled at one hertz, assigning atomic-action boundaries and verb–noun descriptions. Heuristics remove saccades and low-confidence gaze, then cluster nearby fixations. Missing left/right hand or gaze labels are masked from supervision. Camera/wrist axes are manually standardized; keypoints support sources without MANO fits. End-effector pose plus gripper or dexterous-hand state represents robot action, zero-padded to 48 dimensions and separately normalized. The pipeline reduces representational mismatch but does not establish a unique physical correspondence between human and robot motion.
Appendix B Model
The architecture appendix specifies token ordering, gaze quantization, optimization, and spatial augmentation.
B.1 Model Architecture
The model uses one padded/resized 224-by-224 image and a flow-based action expert. Four attention blocks cover language/image, intention, state, and action; tokens attend bidirectionally within each block but cannot access future blocks. An intention encoder/decoder refines features and predicts spatial probabilities. The complete original attention diagram is retained.
| Hyperparameter | Pre-Training | Post-Training |
|---|---|---|
| Step | 20000 | 30000 |
| Batch Size | 2048 | 32 |
| Learning Rate | 5e-5 | 1e-4 |
| Learning Rate Scheduling | Constant | Cosine |
| Gradient Accumulation Steps | 4 | 1 |
| 1 | 1 | |
| 0.1 | 0.1 |
B.2 Implementation Details
Human pretraining runs for twenty thousand steps at batch size 2,048 on eight A800 GPUs, with a reported 1,344 total GPU-hours. Post-training uses thirty thousand steps and batch size 32. The table weights action loss by one and intention loss by 0.1, with constant then cosine learning-rate schedules. Gaze uses 224 bins along each image axis and a dedicated linear decoder. Joint crops, translations, and rotations compensate for central human gaze and more dispersed robot-camera targets. The precise warm-up duration is not specified.
Appendix C Simulation Experiments
Simulation details define a shared multitask protocol and provide additional examples of attention-guided manipulation.
C.1 Training Configuration
All six tasks are trained jointly, with one hundred trajectories each. Control runs at 25 frames per second, and each prediction contains 25 actions executed open-loop for one second before the next inference. The simulation action vector has 21 dimensions, including both arms and active camera motion. LFA trains for one hundred thousand steps, while H-RDT and π0.5 train for thirty thousand with their stated batch sizes and dimensional adaptations. Equal demonstrations do not imply identical optimization budgets or inherited pretraining.
C.2 More Results
Additional sequences show attention near the needle opening and package hook, including distractor and lighting conditions. These examples illustrate the proposed spatial mechanism but do not establish universal robustness, especially given the low success rates on some precision tasks.
Appendix D Real-World Experiments
The physical appendix provides task definitions, collected demonstration counts, baseline adaptations, and additional views of execution.
D.1 Task Description
Demonstration counts vary by task: fruit collecting and screw tightening use one hundred robot and two hundred human examples; remote stowing and block stacking use fifty and one hundred; bottle transfer and typing use twenty and one hundred. Every method/task has forty evaluation trials. Success criteria also vary: for example, remote stowing is counted by the remote being inside the drawer, although the described sequence includes closing it. Complete sequence descriptions should therefore be distinguished from the actual success check.
| Task Name | Task Description | Evaluation Protocols |
|---|---|---|
| Fruit Collecting | Pick up all fruits on the table sequentially and place them into the plate. | 100 robot data; 200 human data; 2 rounds of 20 trials (40 total); success if all fruits are placed in the bowl. |
| Remote Stowing | Open the drawer, pick up the remote control, place it into the drawer, and close the drawer. | 50 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the remote control is inside the drawer. |
| Block Stacking | First place the red block in the center, then stack the yellow block on top of it. | 50 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the yellow block is on top of the red block. |
| Screw Tightening | The left arm picks up the screwdriver and aligns it with the screw, while the right arm presses the electric screwdriver switch. | 100 robot data; 200 human data; 2 rounds of 20 trials (40 total); success if the screw is tightened into the wooden block. |
| Bottle Transferring | Use the dexterous hand to pick up the bottle and place it onto the plate. | 20 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the bottle is stably placed on the plate. |
| Keyboard Typing | Press the four letters “V”, “L”, “I”, and “A” sequentially on the keyboard. | 20 robot data; 100 human data; 2 rounds of 20 trials (40 total); success if the screen displays “VLIA”. |
D.2 Training Configuration
The proposed method and EgoMimic jointly post-train on human and robot data. H-RDT and π0.5 use only robot demonstrations under the implementations chosen by the authors. The latter uses joint-angle actions for compatibility; the other methods use the shared wrist/end-effector representation. These are material comparison differences rather than an isolated test of gaze conditioning alone.
D.3 More Results
Additional physical sequences retain synchronized third-person and egocentric views, showing predicted target points during navigation of task stages and fine manipulation. They complement the success statistics rather than replace failure-rate measurements.
Appendix E Results
Further results separate adaptation to new visual settings, collection-time efficiency, alternate guidance sources, intervention behavior, and runtime.
E.1 Human Data Enable Robot to Generalize to Unseen Positions, Objects, and Scenes
Robot pick-and-place data contain only a strawberry within a five-centimeter placement radius. Evaluation expands the radius to eight centimeters, substitutes lemons or cherries, or adds a white tablecloth. Human adaptation data include these new settings, so this is cross-embodiment transfer of observed variations rather than generalization to variations unseen by both training streams.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/1a/1a4e17890f1308278b3da52710460572081dfe3860cbd6e90965332bf1717a28.webp)
E.2 Human Data Scales Robot Performance More Efficiently
The bottle study holds an initial ten minutes of robot data fixed, then adds human or robot collection time. Ten extra human minutes yield five of five in-domain and twelve of fifteen out-of-domain successes, matching thirty total robot minutes on these counts; twenty extra human minutes reach thirteen of fifteen out-of-domain successes. The twenty-minute robot condition has the same twelve of fifteen out-of-domain successes but four of five in-domain. Thus the apparent efficiency advantage depends partly on the small in-domain subset and on collecting many more human trajectories per minute. New bottle colors/geometries appear only in human demonstrations.
E.3 Effect of Different Intention Sources
Alternate inference inputs include model-predicted gaze, synchronized human gaze, VLM-generated task keypoints, a masked intention path, and random spatial points. The measured 65/65/50/40/25 percent ranking supports useful spatial conditioning. It does not isolate anticipatory timing as the sole cause of the gaze advantage, since the sources differ in more than timing and the intervention changes model inputs at inference.
E.4 Synergy Between Intention and Action
Manually moving the intention point changes which fruit the policy grasps, directly demonstrating that this internal input affects action selection. Gaze transitions also track stages of typing, drawer use, and tool alignment in the shown sequences. This intervention establishes reliance on the supplied point within the model; it does not establish that the point completely represents a human’s latent intention.
E.5 Tradeoff between Sensitivity and Robustness on Intention Guidance
Strong sensitivity helps follow informative target points but makes the policy vulnerable when gaze is wrong or contradicts language. Performance drops when a left-target instruction is paired with a right-target point or random noise. The authors report that loss weighting adjusts this tradeoff, without a quantified calibration or recovery guarantee. Robustness to erroneous predicted intentions remains a practical limitation.
E.6 Inference Latency
On an Intel i9-14900KF and RTX 4090 in the ten-hertz setting, inference takes 44.384 ms versus 32.205 ms for direct action prediction, adding 12.179 ms. This is a specific model/hardware measurement, separate from the simulation’s one-second open-loop chunks and from total perception-to-actuation latency.