Overview
This reading edition follows the complete 17-page arXiv v1 by Zhenyang Feng, Jimin Heo, Erik B. Sudderth and Unnat Jain at the University of California, Irvine, with all five appendices and the linked project demonstrations.
At a glance
TEMPO adds two temporal inputs to a pretrained vision-language-action policy: a frozen-video motion representation and a compact history of the robot’s commands. These address moving-object ambiguity and visually similar task phases that require different actions.
Four real-robot tasks support the approach. Bottle Handover rises from 44% with RTC to 74% with TEMPO; the full untrimmed Flick Catch and Wine Pour tasks expose failures hidden by trimmed evaluations. Pouring is measured by retained bead mass, so its percentage is not a binary rollout-success rate.
1 Introduction
A robot intercepting a moving object needs information about its recent trajectory. It also needs task progress: reaching toward an object and withdrawing after release can look similar while demanding opposite actions. Faster execution can help timing without resolving either ambiguity.
TEMPO combines a video-derived motion token with coarse command-history tokens on an asynchronous VLASH backbone. The added conditioning paths have about two million parameters. The separate pretrained video encoder remains an additional computational resource.
The experiments concern four tasks and three related pretrained backbones. They support the value of temporal conditioning in these settings; they do not test every VLA architecture, systematically vary model scale, or prove that representation alone explains every dynamic-control failure.
2 Related Work
Task-specific dynamic controllers model scenarios such as table tennis, tossing and catching. General pretrained policies offer a different route to skill transfer. Asynchronous inference methods, including RTC and VLASH, reduce delays between observation and execution; other work anticipates a future observation.
TEMPO adds recent motion and task history to those timing improvements. Its closest comparison is the shared VLASH pipeline, which helps isolate the effect of conditioning. Comparisons with other architectures can also reflect differences in their implementations and inference procedures.
3 Motion Ambiguity and State Aliasing
Motion ambiguity: a current image may locate an object without specifying its velocity. State aliasing: similar current observations can belong to different phases, such as acquiring a plate and returning it after catching.
The untrimmed Flick Catch and Wine Pour baselines score zero because they repeatedly approach and withdraw without committing. TEMPO reaches 68% Flick Catch success and 97.6% retained pour mass. These are separately trained untrimmed conditions, not the same evaluations as the main trimmed table.
Figure 2 retains the original joint-command and human-hand comparison. The evidence illustrates failures of the tested policies; correlations with task position can still make some motion statistically predictable from one frame.
| Method | Flick Catch | Wine Pour |
|---|---|---|
| RTC [3] | 0% | 0% |
| VLASH [27] | 0% | 0% |
| TEMPO (Ours) | 68% | 97.6% |
4 Temporal Encoding for Motion-aware Policy (TEMPO)
The architecture supplies two complementary histories. TEMPOMOT describes recent visual dynamics; TEMPOACT summarizes the robot’s preceding commands. Figure 3 retains the full streaming cache, prefix-token and action-expert conditioning paths.
4.1 TEMPO MOT : Captures Object and Scene Dynamics
A frozen video encoder extracts patch features. Current-frame features query cached features from the preceding W−1 frames through cross-attention, producing the motion representation in Equation 1. Each frame’s features are computed once and reused. The numerical window length W is not stated.
SAM 2.1-Tiny is the default. Bottle Handover reaches 74% with SAM, 76% with Reducio, 60% with VidTwin and 52% with VideoLaVIT, compared with 38% for VLASH. SAM is selected for streaming speed rather than the highest handover score. Appendix E tests linear velocity readouts from these encoder features.
Equation 1
| (1) |
4.2 TEMPO ACT : Resolves State Aliasing
The command history is partitioned into ten ordered buckets, each averaging 30 consecutive commands: 300 commands in total, nominally ten seconds at 30 Hz. Equation 2 retains the exact mean aggregation and zero padding before the episode starts. Entirely pre-start buckets receive a padding flag.
This is a coarse history of proprioceptive commands, rather than a full high-frequency trajectory or a learned task-stage label. Increasing bucket length extends the horizon without increasing the ten-token count, while discarding ordering within each bucket.
Equation 2
| (2) |
4.3 Plug-and-Play Temporal Integration
The motion representation and ten history tokens are projected into the vision-language model’s prefix beside current multiview images and instructions. The flattened history also passes through a multilayer perceptron to condition the action expert through an adaptive root-mean-square normalization residual.
The added conditioning path is zero initialized, preserving the pretrained policy’s initial behavior. Fine-tuning keeps the flow-matching objective unchanged. The reported two million parameters, or 0.08%, describe added conditioning parameters, rather than the complete video encoder’s size or resource cost.
The original architecture drawing labels the modulation path FiLM, while the accompanying text specifies an AdaRMS residual. Both original descriptions are retained; the paper does not reconcile those labels.
4.4 Real-Time Deployment via Parallel Encoding
The frozen motion encoder runs in a background thread at about 11 ms per camera frame on an RTX PRO 6000. The policy reads the latest motion token. Median policy forward times are 33.8 ms for VLASH, 35.3 ms for TEMPO and 127.7 ms for RTC.
The additional policy-path latency is about 1.5 ms, while encoder work happens concurrently. These measurements distinguish policy forward time from end-to-end sensor age and asynchronous action execution; a constant-size window does not establish zero deployment overhead.
5 Experiments
Table 4 retains all four tasks and all methods. Each condition uses 50 real-world rollouts. Three columns report binary success; Wine Pour reports the proportion of bead mass retained in the pot. Their average combines different outcome types and should not be read as an overall probability of task completion.
TEMPO’s entries are 74%, 66%, 66% and 98.1%, averaging 76.0. TEMPO+UVT reaches 88%, 88%, 76% and 96.7%, averaging 87.2. The additional training approach helps the first three tasks while slightly reducing retained pour mass.
| Method | Bottle Handover | Drop Catch | Flick Catch | Wine Pour | Average |
|---|---|---|---|---|---|
| RTC [3] | 44% | 80% | 48% | 80.8% | 63.2% |
| VLASH [27] | 38% | 38% | 20% | 94.0% | 47.5% |
| TEMPO (Ours) | 74% | 66% | 66% | 98.1% | 76.0% |
| TEMPO+UVT [11] | 88% | 88% | 76% | 96.7% | 87.2% |
5.1 Experimental Setup
Bottle Handover intercepts a bottle held by a walking person. Drop Catch tracks a moving hand before catching a dropped ball. Flick Catch first acquires a Velcro plate, tracks a hand, then switches to the flicked ball. Wine Pour picks up a bottle, follows a moving pot while pouring beads, and returns the bottle.
The main quantitative comparison trims the release-and-retract phases from Flick Catch and Wine Pour, removing the most conspicuous state aliasing; training and evaluation both use trimmed versions. Appendix A separately evaluates full tasks. The authors call the trimmed advantage a lower bound, but these separately trained conditions do not establish a formal bound.
Motion, changing targets, sequential subtasks and ambiguous phases motivate the task design. They also make the results specific to these staged human–robot interactions and the Velcro/bead apparatus.
5.2 Experimental Results
On Bottle Handover, TEMPO improves over RTC by 30 percentage points and over VLASH by 36 points. Its 74% success corresponds to 37 of 50 trials. Figure 5 preserves every success/failure count.
RTC remains stronger on Drop Catch: 80% versus TEMPO’s 66%, consistent with that task’s timing demands. TEMPO outperforms both baselines on trimmed Flick Catch and retained pour mass. Table 5 also reports handover improvements for three related backbones: 26→56%, 44→80% and 38→74%.
The results support combining temporal representation with responsive inference. They do not show uniform dominance on every task, and the paper supplies no confidence intervals or repeated-seed uncertainty for these 50-rollout comparisons.
6 Analysis and Ablations
Removing either signal reduces Bottle Handover performance: command history alone scores 52%, motion alone 56%, and both 74%, versus VLASH’s 38%. The inputs are complementary in this ablation; neither is necessary for every successful trial.
Table 7 probes policy hidden states for pixel-space object velocity using a multilayer perceptron. TEMPO’s average R² is 0.47, versus 0.19 for RTC and 0.21 for VLASH, with the largest gains on Flick Catch and Wine Pour. This test uses a nonlinear probe, despite nearby prose describing motion as linearly decodable. Appendix E provides a separate linear probe of motion encoders.
TEMPO-Bench contains over 50,000 annotated frames and a multiple-choice variant. Frame count is not an independent-episode count. Probe train/test trajectory splits and uncertainty are not specified here, and the paper does not report multiple-choice benchmark results.
| Configuration | Handover |
|---|---|
| TEMPOACT only | 52% |
| TEMPOMOT only | 56% |
| TEMPO (full) | 74% |
| Task | RTC [3] | VLASH [27] | TEMPO |
|---|---|---|---|
| Bottle Handover | 0.46 | 0.43 | 0.52 |
| Drop Catch | 0.31 | 0.28 | 0.33 |
| Flick Catch | 0.03 | 0.57 | |
| Wine Pour | 0.06 | 0.09 | 0.44 |
| Average | 0.19 | 0.21 | 0.47 |
7 Conclusion
Temporal conditioning improves the tested policies by supplying recent visual motion and coarse task history. Asynchronous execution remains useful, especially for the timing-dominated Drop Catch task.
The contribution is supported by task outcomes, signal ablations and velocity probes. Broader architecture coverage, controlled scale comparisons, more varied tasks and a fuller account of deployment resources remain needed to generalize the strongest claims.
Acknowledgments
The authors thank CoRL reviewers and the area chair for feedback, and Basavasagar Patil for an early draft review.
Original project demonstrations
The authors’ project page supplies qualitative task comparisons, method and setup demonstrations, state-aliasing examples and interactive explorer clips. All 19 original video files are retained unchanged, with local preview images; the four explorer showcase frames are also preserved.
These selected demonstrations illustrate the paper’s argument and should be interpreted alongside its complete quantitative tables, rather than as an additional success-rate estimate. The policy code and benchmark repository remain linked for implementation and data access.




Appendix
The appendix adds untrimmed-task results, hardware and training details, task-specific failure counts, a within-episode input-freezing experiment and linear probes of alternative motion encoders. All five appendices and their original figures and tables are retained below.
Appendix A Additional Failure Mode Analysis
Untrimmed episodes include approaching, performing the central task, and withdrawing. The repeated approach/retract observations can make a single-frame policy oscillate. Figure A1 retains the full rollout sequence and joint/gripper traces.
Both baselines score zero on untrimmed Flick Catch and Wine Pour, while TEMPO scores 68% catch success and 97.6% retained pour mass. The main trimmed-task values are 66% and 98.1%, respectively. The appendix comparison tests jointly changed training/evaluation conditions, rather than isolating a single test-time phase intervention.
Appendix B Experimental Setup Details
Appendix B documents the bimanual platform, cameras, task geometry, human release protocol and compute. These details help interpret the staged physical tasks and the meaning of the latency measurements.
B.1 Hardware Platform
The platform uses two table-mounted I2RT YAM Ultra arms, one fixed third-person camera and two wrist cameras. Images are 224 pixels at 30 Hz; policies execute at 30 Hz where possible.
The text describes the arms as having seven degrees of freedom, while its hardware reference is titled “Yam ultra 6-dof robotic arm.” The original discrepancy is retained rather than silently resolved.
B.2 Evaluation Protocol
Drop Catch releases tennis-ball-sized Velcro balls 13–18 inches above the table, approximately 33–46 cm. Flick Catch uses five or six identical balls on an 80×20×21.6 cm shelf. Velcro balls and plates make successful interception easier to retain after contact.
The releasing person closes their eyes and wears headphones during the ball protocol to reduce anticipation of the robot. This control should not be assumed to apply to every human action in all four tasks.
B.3 Training Details
Training uses eight RTX PRO 6000 GPUs for approximately two hours per policy, with effective batch size 128. Reported iteration times are 0.49 s for VLASH, 0.46 s for RTC and 0.50 s for TEMPO.
RTC is the authors’ reimplementation because an open implementation was unavailable. The paper does not state task demonstration counts, the motion-window length W or detailed probe trajectory splits, limiting exact reproduction from this document alone.
Appendix C Per-Task Failure Mode Breakdown
The authors visually classify each of 50 rollouts by success or the dominant failure. Figures A2 and A3 preserve the complete counts and legends. Categories differ across tasks, so similarly colored segments should be interpreted using each figure’s own legend.
Flick Catch (Fig. A2 ).
Flick Catch has three stages: grasp the plate, track the hand and catch the ball. VLASH fails at hand tracking in 37 of 50 trials. RTC reaches the tracking stage but loses the ball in 19 trials. TEMPO reduces those counts to nine and four, respectively, and succeeds in 33 trials.
The breakdown is consistent with improved motion awareness. It is a manually assigned, dominant-cause description of these rollouts, rather than an independent intervention on each failure mechanism.
Drop Catch (Fig. A3 ).
RTC succeeds in 40 Drop Catch trials, with six lagging failures and four overshoots. VLASH moves before the human hand in 23 trials and succeeds in 19. TEMPO reduces premature movement to six trials and succeeds in 33.
TEMPO’s overshoot count rises from VLASH’s six to seven. The source interprets reduced premature movement as evidence that command history stabilizes task progress, while RTC’s higher overall score shows the continued importance of timing.
Appendix D Input Ablation
For one representative Flick Catch episode, the authors freeze either temporal input at its first-frame value and measure predicted-action mean squared error over time, with exponential smoothing. Freezing motion features hurts most during rapid object motion; freezing history hurts most during the horizontal sweep across the balls before pickup/flick.
At 2.7 s, error rises from 0.019 to 0.059 when motion is frozen, versus 0.033 when history is frozen. At 5.3 s, history freezing gives 0.091, motion freezing 0.051 and the full model 0.020.
This is an open-loop, single-episode prediction analysis. Freezing also changes the input distribution; it does not by itself establish dataset-wide causal necessity or measure closed-loop task success.
Appendix E Motion Encoder Selection
A linear probe predicts annotated pixel-space object velocity from each motion encoder. Average R² is 0.34 for VideoLaVIT, 0.39 for Reducio, 0.46 for SAM 2 and 0.48 for VidTwin, versus 0.08 from object position alone. All per-task values and the four original trajectory images are retained.
The results show useful velocity information in the learned features. Position alone still scores 0.24 on Bottle Handover, illustrating task correlations rather than literal absence of all motion information. SAM 2.1-Tiny balances a strong probe score with streaming speed; VidTwin has the highest probe average and Reducio the highest handover success.
| Initialization for TEMPOMOT | Bottle Handover | Drop Catch | Flick Catch | Wine Pour | Average |
|---|---|---|---|---|---|
| Object Position | 0.24 | 0.03 | 0.03 | 0.02 | 0.08 |
| VideoLaVIT [15] | 0.45 | 0.25 | 0.28 | 0.36 | 0.34 |
| Reducio [28] | 0.50 | 0.36 | 0.38 | 0.32 | 0.39 |
| SAM 2 [24] | 0.58 | 0.37 | 0.50 | 0.39 | 0.46 |
| VidTwin [30] | 0.59 | 0.44 | 0.43 | 0.45 | 0.48 |