Literature
Display
← All papers

EagleVLA: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference

Zebin Yang · Qi Wang · Yunhe Wang · Xiurui Guo · Bo Yu · Shaoshan Liu · Jiafeng Xu · Hao Dong · Meng Li

Preprint posted · CoRL 2026 (accepted) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

EagleVLA combines predicted future visual-language features, adaptive model scheduling, and an optimized onboard inference engine to improve asynchronous robot control.

Table 5: Model architecture parameters of π\pi series VLA models.
Parameter Value
ViT (SigLIP So400m)
ViT layers 27
ViT embedding dimension 1152
ViT patch size 14
Input Image Size 224
LLM (gemma 2b)
LLM layers 18
LLM embedding dimension 2048
LLM attention heads 8
LLM head dimension 256
Action Expert (gemma 300m)
Action expert layers 18
Action expert embedding dimension 1024
Action expert attention heads 8
Action expert head dimension 256
Denoising steps 10

At a glance

A small correction model predicts the scene representation after already committed actions execute, helping the action expert compensate for inference delay. A learned confidence score schedules occasional full vision-language updates between cheaper action predictions. CUDA graph reuse, device-resident buffers, and denoising unrolling further accelerate execution. On Jetson Orin the reported action-update rate rises from 0.70 to 10.30 Hz, while a full visual-language-action pass still takes 271.1 ms. The faster update rate should therefore be distinguished from the rate of incorporating fresh camera observations.

1 Introduction

Onboard inference can take long enough that an action chunk is based on a scene that has changed before execution. Overlapping inference and action execution removes pauses but does not itself correct stale perception. EagleVLA, previously called Jetson-PI, predicts the effects of the committed actions in representation space, then allocates full visual processing only when its prediction is considered unreliable. The paper also compares power and a 500 Wh battery scenario including mechanical consumption; those scenario-dependent comparisons do not establish a universal robot endurance multiplier.

Refer to caption
Figure 1: (a) Battery life of three robots equipped with four different computing devices. We use a 500 Wh WILLQ AGV lithium battery pack, and robot’s mechanical power consumption is included. (b) Power consumption and control frequency of different VLA inference on four computing devices: Jetson Orin, Jetson Thor, RTX A6000, and RTX 4090.

2 Background

π₀.₅ [15] and related models pass vision-language key/value context to a flow-based action expert. real-time chunking [3] smooths asynchronous transitions by inpainting, while VLASH [35] uses predicted future robot state. EagleVLA additionally predicts environment features affected by committed actions. Its variable prediction horizon targets the time at which a newly inferred chunk begins, rather than a distant desired subgoal image.

Refer to caption
Figure 2: Comparison of different inference modules. Δ\Delta is VLA latency in terms of action steps. LL is the number of actions executed in each chunk.

3 When Asynchronous Inference Meets Onboard Computation

For a chunk executed over L steps and an inference delay of Δ steps, the paper analyzes response delays ranging from Δ to Δ+L. Longer chunks and stale inputs can reduce success. Continuous inference reduces gaps but remains limited by computation. EagleVLA addresses these issues jointly through future-state correction, selective visual-language invocation, and a more efficient execution engine. Feature prediction cannot reveal arbitrary external events that occur while fresh observations are skipped.

Refer to caption
Figure 3: (a)(b) Success rate changes with different Δ\Delta on LIBERO-Spatial and LIBERO-Goal using π0\pi_{0}. Here, L=H−ΔL=H-\Delta, where H=20H=20 is the action chunk size. (c)(d) Success rate changes with different LL values. using naive asynchronous inference on LIBERO-Spatial and LIBERO-Goal.

4 Method

The method comprises two-stage training of a future-feature correction model, confidence-driven scheduling, and system optimizations for fixed-shape VLA computation.

Refer to caption
Figure 4: Overview of EagleVLA.
VLA Device ViT LLM Action Expert Total
Orin-30W 322.4 856.4 1090.0 2268.8
Orin-50W 152.5 632.3 518.2 1403.0
π0\pi_{0} Thor 63.7 185.0 199.2 447.9
A6000 23.4 43.5 46.9 116.1
4090 13.1 32.2 22.4 69.5
Orin-30W 329.0 858.2 1102.2 2289.4
Orin-50W 152.3 631.0 536.8 1420.8
π0.5\pi_{0.5} Thor 61.7 187.0 209.2 457.9
A6000 23.8 51.2 50.2 128.2
4090 13.2 35.9 25.2 76.4
Table 1: Inference latency breakdown (ms) of different VLA models on various devices. We consider different power modes of Jetson Orin.

4.1 Foresight-Aligned Asynchronous Correction

A roughly forty-million-parameter module receives compressed final-layer VLM features and the action prefix guaranteed to execute during the delay. It predicts compressed features at the future execution time; this correction supplements the action expert’s existing context rather than regenerating images or every layer’s cache.

Stage one trains the compressor and action expert using actual future features, teaching the expert to use the correction. Stage two learns to predict those features from current features and committed actions, sampling different delays during training. Its confidence head is trained toward the negative feature-prediction error, as shown in Equation 1. Confidence is therefore an error-related scalar with negative thresholds, not a calibrated probability of task success or safety.

Refer to caption
Figure 5: Foresight-Aligned Asynchronous Correction.
Refer to caption
Figure 6: Scheduling Optimization.
Equation 1
Lp​r​e​d​i​c​t=‖h^t+Δ−ht+Δ‖2,Lt​o​t​a​l=Lp​r​e​d​i​c​t+λ​‖c^+Lp​r​e​d​i​c​t‖2,L_{predict}=\|\hat{h}_{t+\Delta}-h_{t+\Delta}\|_{2},~~~~~L_{total}=L_{predict}+\lambda\|\hat{c}+L_{predict}\|_{2}, (1)

4.2 Confidence-based Scheduling Optimization

When predicted confidence exceeds a threshold, the scheduler reuses visual-language context and invokes the correction module plus action expert. Otherwise it captures the current scene, refreshes the VLM cache, predicts across the longer full-model delay, and updates the action buffer. Rolling predicted hidden states forward reduces computation but accumulates errors, motivating fresh observations.

The paper’s roofline argument attributes poor parallel VLM/action-expert execution on Jetson devices to shared memory-bandwidth contention. It therefore favors selective sequential invocation rather than simultaneous execution. This is an analysis of the tested models and devices, not a general prohibition on parallel edge inference. The original hardware specification and timing tables are retained for inspection.

Table 2: Hardware specifications and roofline analysis of different computing devices.
RTX 4090 RTX A6000 Orin 32GB Orin 64GB Thor
Max Power (W) 450 300 50 50 130
TOPS 309 309 200 275 510
Memory 24GB GDDR6X 48GB GDDR6 48GB LPDDR5 48GB LPDDR5 48GB LPDDR5X
Bandwidth (GB/s) 1024 768 204.8 204.8 273
Balance Computational Intensity 618 804 1953 2685 3736
(TOPS/Bandwidth ×1000\times 1000)
VLM Compute Bound Compute Bound Bandwidth Bound Bandwidth Bound Bandwidth Bound
Action Expert Bandwidth Bound Bandwidth Bound Bandwidth Bound Bandwidth Bound Bandwidth Bound

4.3 System Design

Fixed image resolutions and padded language length allow repeated inference to reuse one CUDA graph. Fixed-size intermediate tensors remain in GPU buffers instead of returning through CPU memory. Multiple flow-denoising iterations are unrolled into one graph, trading a one-time construction cost for fewer repeated launches. These system changes preserve the modeled computation, but the complete method also requires correction-aware training and altered scheduling. They are complementary to quantization or pruning rather than substitutes for those techniques.

5 Experiments

Experiments test success under simulated delays, actual device latency, a physical clothes-handling task, and confidence-threshold behavior.

Table 3: Success rate comparison across different methods on four sub-datasets of LIBERO, using π0.5\pi_{0.5}. We report the SR of using Foresight-Aligned Asynchronous Correction alone (ours) and with Confidence-based Scheduling Optimization (+Sched). We estimate inference time of action expert as Δa​e=⌈Δ3⌉\Delta_{ae}=\lceil\frac{\Delta}{3}\rceil., which is a common ratio both on high-end GPUs such as RTX 4090 and on onboard GPUs such as Orin. We only train one model for all Δ\Delta values.
SPATIAL OBJECT GOAL LIBERO-10
Sync. 97.3 99.6 96.7 93.5
Δ\Delta VLASH RTC Ours +Sched VLASH RTC Ours +Sched VLASH RTC Ours +Sched VLASH RTC Ours +Sched
1 98.8 97.1 97.7 98.6 99.2 98.5 98.7 98.8 96.7 96.5 97.0 97.5 94.4 92.3 93.4 93.1
2 97.5 95.4 97.1 97.6 99.2 98.5 98.5 99.3 97.0 96.0 96.2 97.3 94.6 92.1 92.7 92.9
3 94.4 94.5 97.1 97.2 98.8 96.4 98.2 98.4 93.3 94.3 96.7 96.4 91.9 89.6 91.3 91.9
4 92.5 92.5 96.7 96.9 96.9 97.7 97.8 98.9 93.3 93.3 95.9 96.9 89.6 85.8 91.8 92.3
5 84.3 91.2 95.9 96.7 94.3 96.9 96.9 98.6 89.9 93.9 96.0 96.1 80.3 83.6 91.5 92.0
6 74.4 91.7 97.0 97.5 88.5 97.4 98.3 98.4 81.3 94.0 96.7 96.9 78.5 84.6 92.2 92.9
7 51.3 91.7 97.0 97.3 81.4 97.1 98.0 98.4 76.1 92.6 97.1 97.2 77.0 85.3 92.6 92.6
8 46.7 90.9 97.2 97.5 65.1 93.8 98.3 98.8 70.8 93.4 96.8 96.8 65.5 83.2 92.3 92.7
9 30.1 88.8 97.0 97.3 51.7 93.1 97.5 97.6 59.7 92.5 96.3 96.5 59.6 81.0 92.0 92.2
Avg 74.4 92.6 97.0 97.4 86.1 96.6 98.0 98.6 84.2 94.1 96.5 96.8 81.3 86.4 92.2 92.5
Table 4: Inference latency and control frequency on different devices, evaluated on LIBERO.
Jetson Orin
Method ViT (ms) LLM (ms) Action Expert (ms) Total (ms) Reaction Time (ms) Control Frequency (Hz)
Naive PI05 152.3 631.0 536.8 1420.8 1420.8 0.70
+Schedule opt. 152.3 631.0 536.8 1420.8 674.9 1.48
+Graph reuse 71.9 196.9 143.9 412.9 184.0 5.43
+Intermediate Buffer & Unroll 66.6 137.0 67.6 271.1 97.1 10.30
Jetson Thor
Method ViT (ms) LLM (ms) Action Expert (ms) Total (ms) Reaction Time (ms) Control Frequency (Hz)
Naive PI05 61.7 187.0 209.2 457.9 457.9 2.18
+Schedule opt. 61.7 187.0 209.2 457.9 244.7 4.08
+Graph reuse 32.0 92.3 115.7 240.0 134.6 7.43
+Intermediate Buffer & Unroll 29.0 54.9 48.9 132.8 61.1 16.37

5.1 Simulation Experiments

Four LIBERO suites [21] compare a zero-delay synchronous reference, chunk inpainting, future-robot-state inference, feature correction, and correction plus scheduling. One trained correction model handles all tested Δ values from one to nine. The scheduler comparison estimates action-expert delay as the ceiling of one third of the full-model delay.

With scheduling, average success across suites and delays is about 96.3%, versus 81.5% for VLASH and 92.4% for chunk inpainting: gains of approximately 14.8 and 3.9 percentage points. At the largest delay, the respective gaps are 45.6 and 7.0 points. Small-delay VLASH results occasionally exceed EagleVLA, so the advantage is chiefly robust performance as delay grows. The zero-delay synchronous reference is a task-quality reference, not a realistic onboard latency baseline.

5.2 Latency Evaluation

On Orin, naive total inference is 1,420.8 ms at a reported 0.70 Hz update rate. Scheduling alone leaves full inference unchanged but reduces the reported reaction/update interval. With all system optimizations, a complete pass is 271.1 ms, the reported reaction interval is 97.1 ms, and the update rate is 10.30 Hz. These yield the stated 14.71-fold rate gain, while full-pass latency improves by about 5.24-fold.

On Thor, corresponding totals improve from 457.9 to 132.8 ms, with a reported 61.1 ms reaction interval and 16.37 Hz update rate. The 9.20-fold comparison to vla.cpp [25] uses its cited 893 ms Orin latency against EagleVLA’s scheduled update interval. These figures measure different invocation patterns; they do not mean a fresh camera-to-action forward pass runs at every high-rate update.

Refer to caption
Method Picking Folding Placing
RTX 4090 (Baseline) 10/10 7/10 10/10
Jetson Orin (Naive Async) 6/10 0/10 5/10
EagleVLA (Ours) 10/10 8/10 9/10
Figure 7: Real-world results on 3 subtasks (picking, folding, placing) on different deployments.
Refer to caption
Figure 8: Impact of θ\theta, evaluated on LIBERO-Spatial on Jetson Orin.
Refer to caption
Figure 9: An example of confidence evolution. The confidence threshold is -0.2, and the instruction is “pick up the black bowl between the plate and the ramekin and place it on the plate”.

5.3 Real-world Experiments

A physical X2-W robot uses three 224×224 cameras, the XR-1 model on Orin, and 15 Hz action execution for cloth picking, folding, and placing. Each reported subtask has ten trials. EagleVLA scores 10/10, 8/10, and 9/10; the RTX 4090 reference scores 10/10, 7/10, and 10/10; naive Orin asynchronous execution scores 6/10, 0/10, and 5/10. These are subtask results, not a reported end-to-end success rate for the whole three-stage sequence. Fifteen-hertz actuator execution is also distinct from VLM observation refresh frequency.

5.4 Analysis of Confidence-Based Scheduling

An example trace shows confidence dropping during difficult grasp/place transitions, triggering visual updates. A LIBERO-Spatial threshold sweep trades more frequent VLM calls against longer feature-only prediction periods. The tested thresholds improve reported success and update rate relative to no scheduling, but one trace and a threshold sweep do not establish confidence calibration or reliable detection of unseen external disturbances.

6 Conclusion

Future-feature correction helps an asynchronous action expert account for its own committed motion, while scheduling and graph-level optimizations substantially increase onboard action-update throughput. The strongest evidence combines delay-stress simulation, measured device inference, and a small physical cloth-handling evaluation. Fresh visual observation latency and task-level reliability remain distinct from the headline update rate.

7 Limitations

The paper identifies limited onboard compute and bandwidth as continuing constraints as model size and batch size grow. Its method also depends on predicted future features between visual refreshes; faster action-expert invocation does not supply new sensory evidence. The reported experiments do not establish general disturbance-response guarantees or long-duration battery performance across robot workloads.

8 Acknowledgments

The authors acknowledge support from national and municipal research programs in China, including work on efficient embodied-intelligence edge hardware.

Appendix A Model Architecture of π series Models

The model appendix specifies a twenty-seven-layer visual encoder, eighteen-layer language model, and eighteen-layer action expert with ten flow-denoising steps. Images and language generate key/value context used by the expert’s attention while action noise is refined into a chunk. EagleVLA adds predicted future hidden-state information to this pipeline. Table 5 retains all dimensions and head counts.

Appendix B Architecture of Future Correction Module

The committed action sequence is encoded by a multilayer network and transformer, retaining its final token. A Q-Former compresses VLM features to four tokens. Their joined representation passes through two transformer blocks and separates into a future-feature head and an error-related confidence head. Figure 10 preserves the exact dataflow and dimensions.

Refer to caption
Figure 10: Architecture of future correction module. TT denotes the number of tokens in the VLM hidden sequence; T′T^{\prime} is the compressed token length after Q-Former compression (with T′<<TT^{\prime}<<T, T′=4T^{\prime}=4 in practice); dVLMd_{\text{VLM}} is the embedding dimension of the VLM; dAEd_{\text{AE}} is the embedding dimension of the action expert (DiT); tt and t+Δt+\Delta denotes time steps.

Appendix C Algorithm Details of Confidence-based Scheduling Optimization

Algorithm 1 maintains action, hidden-state, and key/value buffers while actions execute continuously. High confidence uses the shorter action-expert horizon and advances the predicted hidden state; low confidence observes the scene and refreshes context across the full-model horizon. The algorithm’s explicit observation step occurs only in the low-confidence branch, which is central to interpreting the throughput-versus-perception trade-off.

Algorithm 1 Confidence-based Scheduling Optimization for Asynchronous VLA Inference
1: VLM fvlmf_{\text{vlm}}, action expert factf_{\text{act}}, future correction module fcorrf_{\text{corr}}
2: Initial observation o0o_{0}, language instruction ll, confidence threshold θ\theta (θ<0.0\theta<0.0)
3: Inference latency of the whole VLA model Δ\Delta, inference latency of action expert Δa​e\Delta_{ae}
4: Continuous action execution stream
5: Initialize KV buffer Bkv←∅B_{\text{kv}}\leftarrow\emptyset, hidden state buffer Bh←∅B_{h}\leftarrow\emptyset, action buffer Ba←∅B_{a}\leftarrow\emptyset
6: Compute initial VLM output: h0,K​V0←fvlm​(o0,l)h_{0},KV_{0}\leftarrow f_{\text{vlm}}(o_{0},l)
7: Update BhB_{h} and BkvB_{\text{kv}}: Bh←h0,Bkv←K​V0B_{h}\leftarrow h_{0},B_{\text{kv}}\leftarrow KV_{0}
8: Update actions buffer BaB_{a}: Ba←fact​(h0,K​V0)B_{a}\leftarrow f_{\text{act}}(h_{0},KV_{0})
9: Initialize parameters c^←0.0,t←0\hat{c}\leftarrow 0.0,~t\leftarrow 0 ⊳\triangleright Initialize a large confidence
10: while task not finished do ⊳\triangleright Actions in BaB_{a} are continuously being executed in parallel
11:   if c^>θ\hat{c}>\theta then ⊳\triangleright Confidence high: skip VLM, use future correction only
12:    Retrieve committed Δa​e\Delta_{ae} actions {at,at+1,…,at+Δa​e−1}\{a_{t},a_{t+1},\dots,a_{t+\Delta_{ae}-1}\} from BaB_{a}
13:    Predict future VLM final-layer state: ht+Δa​e←fcorr​(Bh,at,…,at+Δa​e−1)h_{t+\Delta_{ae}}\leftarrow f_{\text{corr}}(B_{h},a_{t},\dots,a_{t+\Delta_{ae}-1})
14:    Obtain new confidence c^←fcorr.confidence​()\hat{c}\leftarrow f_{\text{corr}}.\text{confidence}()
15:    Update actions buffer: Ba←fact​(ht+Δa​e,Bkv)B_{a}\leftarrow f_{\text{act}}(h_{t+\Delta_{ae}},B_{\text{kv}})
16:    Update hidden state buffer: Bh←ht+Δa​eB_{h}\leftarrow h_{t+\Delta_{ae}}
17:   else⊳\triangleright Confidence low: invoke VLM to refresh environment context
18:    Observe current environment oto_{t}
19:    Retrieve committed Δ\Delta actions {at,at+1,…,at+Δ−1}\{a_{t},a_{t+1},\dots,a_{t+\Delta-1}\} from BaB_{a}
20:    Update VLM state: ht,K​Vt←fvlm​(ot,l)h_{t},KV_{t}\leftarrow f_{\text{vlm}}(o_{t},l)
21:    Update BkvB_{\text{kv}}: Bkv←K​VtB_{\text{kv}}\leftarrow KV_{t}
22:    Predict future VLM final-layer state: ht+Δ←fcorr​(ht,at,…,at+Δ−1)h_{t+\Delta}\leftarrow f_{\text{corr}}(h_{t},a_{t},\dots,a_{t+\Delta-1})
23:    Obtain new confidence c^←fcorr.confidence​()\hat{c}\leftarrow f_{\text{corr}}.\text{confidence}()
24:    Update actions buffer: Ba←fact​(ht+Δ,Bkv)B_{a}\leftarrow f_{\text{act}}(h_{t+\Delta},B_{\text{kv}})
25:    Update hidden state buffer: Bh←ht+ΔB_{h}\leftarrow h_{t+\Delta}
26:   end if
27:   t←t+(number of executed actions)t\leftarrow t+\text{(number of executed actions)}
28: end while
29: return
Table 6: Training hyperparameters for each stage of our method.
Parameter Stage 1 Stage 2
Batch size 32 32
Learning rate 3e-4 3e-4
Optimizer AdamW AdamW
Number of traning steps 30,000 25,000
Learning rate schedule Cosine Cosine
Warmup steps 500 500
GPU type NVIDIA A100 NVIDIA A100

Appendix D Training Details

Both training stages use batch size thirty-two, AdamW, a 3×10⁻⁴ learning rate, cosine scheduling, and five hundred warmup steps on A100 hardware. Stage one uses thirty thousand steps and stage two twenty-five thousand. Table 6 retains the full reported configuration.

Concept demonstration

Open image ↗