Literature
Display
← All papers

TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

Zhenyang Feng · Jimin Heo · Erik B. Sudderth · Unnat Jain

Preprint posted · CoRL 2026 (accepted) ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

This reading edition follows the complete 17-page arXiv v1 by Zhenyang Feng, Jimin Heo, Erik B. Sudderth and Unnat Jain at the University of California, Irvine, with all five appendices and the linked project demonstrations.

At a glance

TEMPO adds two temporal inputs to a pretrained vision-language-action policy: a frozen-video motion representation and a compact history of the robot’s commands. These address moving-object ambiguity and visually similar task phases that require different actions.

Four real-robot tasks support the approach. Bottle Handover rises from 44% with RTC to 74% with TEMPO; the full untrimmed Flick Catch and Wine Pour tasks expose failures hidden by trimmed evaluations. Pouring is measured by retained bead mass, so its percentage is not a binary rollout-success rate.

1 Introduction

A robot intercepting a moving object needs information about its recent trajectory. It also needs task progress: reaching toward an object and withdrawing after release can look similar while demanding opposite actions. Faster execution can help timing without resolving either ambiguity.

TEMPO combines a video-derived motion token with coarse command-history tokens on an asynchronous VLASH backbone. The added conditioning paths have about two million parameters. The separate pretrained video encoder remains an additional computational resource.

The experiments concern four tasks and three related pretrained backbones. They support the value of temporal conditioning in these settings; they do not test every VLA architecture, systematically vary model scale, or prove that representation alone explains every dynamic-control failure.

Original complete multi-panel figure layout, including task labels
Figure 1: TEMPO makes a pretrained VLA motion-aware. Four dynamic tasks (left) in which the target object is in motion throughout the episode. A standard VLA, conditioned on a single observation, cannot determine how the scene is evolving (center, red; “? m/s”) and cannot disambiguate perceptually identical observations that arise at different phases of the same task, producing failed grasps and mistimed actions. TEMPO resolves both failure modes by augmenting the policy with two compact signals: TEMPOMOT, a motion representation from a frozen video foundation model, and TEMPOACT, a compact proprioceptive history providing subtask context. Together they enable reliable task completion across all four tasks (right, green).

2 Related Work

Task-specific dynamic controllers model scenarios such as table tennis, tossing and catching. General pretrained policies offer a different route to skill transfer. Asynchronous inference methods, including RTC and VLASH, reduce delays between observation and execution; other work anticipates a future observation.

TEMPO adds recent motion and task history to those timing improvements. Its closest comparison is the shared VLASH pipeline, which helps isolate the effect of conditioning. Comparisons with other architectures can also reflect differences in their implementations and inference procedures.

3 Motion Ambiguity and State Aliasing

Motion ambiguity: a current image may locate an object without specifying its velocity. State aliasing: similar current observations can belong to different phases, such as acquiring a plate and returning it after catching.

The untrimmed Flick Catch and Wine Pour baselines score zero because they repeatedly approach and withdraw without committing. TEMPO reaches 68% Flick Catch success and 97.6% retained pour mass. These are separately trained untrimmed conditions, not the same evaluations as the main trimmed table.

Figure 2 retains the original joint-command and human-hand comparison. The evidence illustrates failures of the tested policies; correlations with task position can still make some motion statistically predictable from one frame.

Original complete multi-panel figure layout, including task labels
Figure 2: Robot joint action (blue) does not track the human hand (red). Side-by-side comparison of a Drop Catch rollout from the baseline (VLASH, left) and TEMPO (right). The baseline oscillates left-to-right continuously while the human hand moves in a single direction, whereas TEMPO closely tracks the hand.
Method Flick Catch Wine Pour
RTC [3] 0% 0%
VLASH [27] 0% 0%
TEMPO (Ours) 68% 97.6%
Table 1: Success on the untrimmed Flick Catch and Wine Pour, where each episode retains the release-and-retract phase that produces state aliasing. Both asynchronous baselines fail every rollout; TEMPO completes the task by using TEMPOACT to disambiguate phases the current frame cannot.

4 Temporal Encoding for Motion-aware Policy (TEMPO)

The architecture supplies two complementary histories. TEMPOMOT describes recent visual dynamics; TEMPOACT summarizes the robot’s preceding commands. Figure 3 retains the full streaming cache, prefix-token and action-expert conditioning paths.

Original complete multi-panel figure layout, including task labels
Figure 3: TEMPO architecture. TEMPO augments a pretrained VLM-based policy with two temporal inputs. TEMPOMOT cross-attends the current frame’s patch features 𝐟t\mathbf{f}_{t} against a streaming cache ℱt\mathcal{F}_{t} of features from the preceding W−1W-1 frames, producing a motion code 𝐦t\mathbf{m}_{t} that encodes scene dynamics. TEMPOACT mean pools the robot’s recent proprioceptive command history into KK fixed-size buckets, yielding a compact summary 𝐜t\mathbf{c}_{t}. Both signals attach to the pretrained backbone with near-zero overhead: 𝐦t\mathbf{m}_{t} and 𝐜t\mathbf{c}_{t} are projected to the VLM’s token dimension and appended to its prefix, and 𝐜t\mathbf{c}_{t} additionally conditions the action expert via an AdaRMS residual.

4.1 TEMPO MOT : Captures Object and Scene Dynamics

A frozen video encoder extracts patch features. Current-frame features query cached features from the preceding W−1 frames through cross-attention, producing the motion representation in Equation 1. Each frame’s features are computed once and reused. The numerical window length W is not stated.

SAM 2.1-Tiny is the default. Bottle Handover reaches 74% with SAM, 76% with Reducio, 60% with VidTwin and 52% with VideoLaVIT, compared with 38% for VLASH. SAM is selected for streaming speed rather than the highest handover score. Appendix E tests linear velocity readouts from these encoder features.

Equation 1
𝐦t=ϕ(ot−W+1:t)=CrossAttn(𝐟t,ℱt),\mathbf{m}_{t}\;=\;\phi(o_{t-W+1:t})\;=\;\operatorname{CrossAttn}\bigl(\mathbf{f}_{t},\;\mathcal{F}_{t}\bigr), (1)
Motion encoder ϕ\phi Handover
SAM 2.1-Tiny [24] (default) 74%
Reducio [28] 76%
VidTwin [30] 60%
VideoLaVIT [15] 52%
Table 2: Swapping the motion encoder ϕ\phi keeps Bottle Handover success comfortably above the VLASH baseline (38%38\%).

4.2 TEMPO ACT : Resolves State Aliasing

The command history is partitioned into ten ordered buckets, each averaging 30 consecutive commands: 300 commands in total, nominally ten seconds at 30 Hz. Equation 2 retains the exact mean aggregation and zero padding before the episode starts. Entirely pre-start buckets receive a padding flag.

This is a coarse history of proprioceptive commands, rather than a full high-frequency trajectory or a learned task-stage label. Increasing bucket length extends the horizon without increasing the ten-token count, while discarding ordering within each bucket.

Equation 2
𝐜t=(q¯t(1),…,q¯t(K)),q¯t(k)=1L​∑τ∈Bkqτ∈ℝdq,qτ:=0​for ​τ<episode start.\mathbf{c}_{t}=\bigl(\bar{q}^{(1)}_{t},\dots,\bar{q}^{(K)}_{t}\bigr),\qquad\bar{q}^{(k)}_{t}=\frac{1}{L}\sum_{\tau\in B_{k}}q_{\tau}\in\mathbb{R}^{d_{q}},\qquad q_{\tau}:=0\ \text{for }\tau<\text{episode start}. (2)

4.3 Plug-and-Play Temporal Integration

The motion representation and ten history tokens are projected into the vision-language model’s prefix beside current multiview images and instructions. The flattened history also passes through a multilayer perceptron to condition the action expert through an adaptive root-mean-square normalization residual.

The added conditioning path is zero initialized, preserving the pretrained policy’s initial behavior. Fine-tuning keeps the flow-matching objective unchanged. The reported two million parameters, or 0.08%, describe added conditioning parameters, rather than the complete video encoder’s size or resource cost.

The original architecture drawing labels the modulation path FiLM, while the accompanying text specifies an AdaRMS residual. Both original descriptions are retained; the paper does not reconcile those labels.

4.4 Real-Time Deployment via Parallel Encoding

The frozen motion encoder runs in a background thread at about 11 ms per camera frame on an RTX PRO 6000. The policy reads the latest motion token. Median policy forward times are 33.8 ms for VLASH, 35.3 ms for TEMPO and 127.7 ms for RTC.

The additional policy-path latency is about 1.5 ms, while encoder work happens concurrently. These measurements distinguish policy forward time from end-to-end sensor age and asynchronous action execution; a constant-size window does not establish zero deployment overhead.

Method Median
VLASH [27] 33.8 ms
RTC [3] 127.7 ms
TEMPO (Ours) 35.3 ms
Table 3: Inference latency comparison across methods.

5 Experiments

Table 4 retains all four tasks and all methods. Each condition uses 50 real-world rollouts. Three columns report binary success; Wine Pour reports the proportion of bead mass retained in the pot. Their average combines different outcome types and should not be read as an overall probability of task completion.

TEMPO’s entries are 74%, 66%, 66% and 98.1%, averaging 76.0. TEMPO+UVT reaches 88%, 88%, 76% and 96.7%, averaging 87.2. The additional training approach helps the first three tasks while slightly reducing retained pour mass.

Method Bottle Handover Drop Catch Flick Catch Wine Pour Average
RTC [3] 44% 80% 48% 80.8% 63.2%
VLASH [27] 38% 38% 20% 94.0% 47.5%
TEMPO (Ours) 74% 66% 66% 98.1% 76.0%
TEMPO+UVT [11] 88% 88% 76% 96.7% 87.2%
Table 4: Success rates across the four dynamic tasks. Each entry is the percentage of 5050 real-world rollouts in which the policy completes the task, evaluated under varied object speeds, appearances, and approach directions. For Wine Pour task, instead of recording binary success or failure, we measure the percentage of the ”wine” mass retained in the pot at the end of the pour.

5.1 Experimental Setup

Bottle Handover intercepts a bottle held by a walking person. Drop Catch tracks a moving hand before catching a dropped ball. Flick Catch first acquires a Velcro plate, tracks a hand, then switches to the flicked ball. Wine Pour picks up a bottle, follows a moving pot while pouring beads, and returns the bottle.

The main quantitative comparison trims the release-and-retract phases from Flick Catch and Wine Pour, removing the most conspicuous state aliasing; training and evaluation both use trimmed versions. Appendix A separately evaluates full tasks. The authors call the trimmed advantage a lower bound, but these separately trained conditions do not establish a formal bound.

Motion, changing targets, sequential subtasks and ambiguous phases motivate the task design. They also make the results specific to these staged human–robot interactions and the Velcro/bead apparatus.

Original complete multi-panel figure layout, including task labels
Figure 4: Task Setup. We benchmark all models across four comprehensive dynamic tasks.

5.2 Experimental Results

On Bottle Handover, TEMPO improves over RTC by 30 percentage points and over VLASH by 36 points. Its 74% success corresponds to 37 of 50 trials. Figure 5 preserves every success/failure count.

RTC remains stronger on Drop Catch: 80% versus TEMPO’s 66%, consistent with that task’s timing demands. TEMPO outperforms both baselines on trimmed Flick Catch and retained pour mass. Table 5 also reports handover improvements for three related backbones: 26→56%, 44→80% and 38→74%.

The results support combining temporal representation with responsive inference. They do not show uniform dominance on every task, and the paper supplies no confidence intervals or repeated-seed uncertainty for these 50-rollout comparisons.

Original chart; see complete figure caption
Figure 5: Failure-mode breakdown on Bottle Handover, we categorized and visually examined each episode in deployment rollouts and attributed each model’s failure to different categories.
VLA backbone Base + TEMPO
π0.5\pi_{0.5} [21] 26% 56%
RTC [3] 44% 80%
VLASH [27] 38% 74%
Table 5: TEMPO is plug-and-play across VLA backbones. Adding TEMPOMOT and TEMPOACT improves Bottle Handover on all backbones, including the non-asynchronous π0.5\pi_{0.5}.

6 Analysis and Ablations

Removing either signal reduces Bottle Handover performance: command history alone scores 52%, motion alone 56%, and both 74%, versus VLASH’s 38%. The inputs are complementary in this ablation; neither is necessary for every successful trial.

Table 7 probes policy hidden states for pixel-space object velocity using a multilayer perceptron. TEMPO’s average R² is 0.47, versus 0.19 for RTC and 0.21 for VLASH, with the largest gains on Flick Catch and Wine Pour. This test uses a nonlinear probe, despite nearby prose describing motion as linearly decodable. Appendix E provides a separate linear probe of motion encoders.

TEMPO-Bench contains over 50,000 annotated frames and a multiple-choice variant. Frame count is not an independent-episode count. Probe train/test trajectory splits and uncertainty are not specified here, and the paper does not report multiple-choice benchmark results.

Configuration Handover
TEMPOACT only 52%
TEMPOMOT only 56%
TEMPO (full) 74%
Table 6: Both temporal signals contribute to the task. Removing either TEMPOMOT or TEMPOACT reduces Bottle Handover success; the two together outperform either alone.
Task RTC [3] VLASH [27] TEMPO
Bottle Handover 0.46 0.43 0.52
Drop Catch 0.31 0.28 0.33
Flick Catch −0.07-0.07 0.03 0.57
Wine Pour 0.06 0.09 0.44
Average 0.19 0.21 0.47
Table 7: Motion probing of the trained VLA hidden state. MLP probe on object velocity across our dynamic tasks (R2R^{2}). On Flick Catch and Wine Pour, where motion is the only usable cue, the asynchronous baselines’ representations collapse to near zero while TEMPO’s remains informative.

7 Conclusion

Temporal conditioning improves the tested policies by supplying recent visual motion and coarse task history. Asynchronous execution remains useful, especially for the timing-dominated Drop Catch task.

The contribution is supported by task outcomes, signal ablations and velocity probes. Broader architecture coverage, controlled scale comparisons, more varied tasks and a fuller account of deployment resources remain needed to generalize the strongest claims.

Acknowledgments

The authors thank CoRL reviewers and the area chair for feedback, and Basavasagar Patil for an early draft review.

Original project demonstrations

The authors’ project page supplies qualitative task comparisons, method and setup demonstrations, state-aliasing examples and interactive explorer clips. All 19 original video files are retained unchanged, with local preview images; the four explorer showcase frames are also preserved.

These selected demonstrations illustrate the paper’s argument and should be interpreted alongside its complete quantitative tables, rather than as an additional success-rate estimate. The policy code and benchmark repository remain linked for implementation and data access.

State aliasing: baseline
State aliasing: TEMPO
Bottle Handover: baseline
Bottle Handover: TEMPO
Drop Catch: baseline
Drop Catch: TEMPO
Method illustration
Wine Pour: TEMPO
Wine Pour: VLASH
Trimmed Flick Catch rollout
State-aliasing illustration
Four task setups
Project overview
Flick Catch: baseline
Flick Catch: TEMPO
Wine Pour explorer: approaching phase
Wine Pour explorer: withdrawing phase
Flick Catch explorer: approaching phase
Flick Catch explorer: withdrawing phase
Original pour explorer still at 8.7 seconds
Original pour state-aliasing explorer frame at 8.7 s.
Original pour explorer still at 13.1 seconds
Original pour state-aliasing explorer frame at 13.1 s.
Original spartan explorer still at 1.7 seconds
Original spartan state-aliasing explorer frame at 1.7 s.
Original spartan explorer still at 15.7 seconds
Original spartan state-aliasing explorer frame at 15.7 s.

Appendix

The appendix adds untrimmed-task results, hardware and training details, task-specific failure counts, a within-episode input-freezing experiment and linear probes of alternative motion encoders. All five appendices and their original figures and tables are retained below.

Appendix A Additional Failure Mode Analysis

Untrimmed episodes include approaching, performing the central task, and withdrawing. The repeated approach/retract observations can make a single-frame policy oscillate. Figure A1 retains the full rollout sequence and joint/gripper traces.

Both baselines score zero on untrimmed Flick Catch and Wine Pour, while TEMPO scores 68% catch success and 97.6% retained pour mass. The main trimmed-task values are 66% and 98.1%, respectively. The appendix comparison tests jointly changed training/evaluation conditions, rather than isolating a single test-time phase intervention.

Original complete multi-panel figure layout, including task labels
Figure A1: VLASH rollout on the untrimmed Flick Catch task. The task is to grip the velcro plate and follow the human hand. VLASH’s gripper hovers near the plate without committing to grasp or retract, because a single-frame observation cannot differentiate reaching for the plate from releasing it. RTC exhibits the same failure (see project page).

Appendix B Experimental Setup Details

Appendix B documents the bimanual platform, cameras, task geometry, human release protocol and compute. These details help interpret the staged physical tasks and the meaning of the latency measurements.

B.1 Hardware Platform

The platform uses two table-mounted I2RT YAM Ultra arms, one fixed third-person camera and two wrist cameras. Images are 224 pixels at 30 Hz; policies execute at 30 Hz where possible.

The text describes the arms as having seven degrees of freedom, while its hardware reference is titled “Yam ultra 6-dof robotic arm.” The original discrepancy is retained rather than silently resolved.

B.2 Evaluation Protocol

Drop Catch releases tennis-ball-sized Velcro balls 13–18 inches above the table, approximately 33–46 cm. Flick Catch uses five or six identical balls on an 80×20×21.6 cm shelf. Velcro balls and plates make successful interception easier to retain after contact.

The releasing person closes their eyes and wears headphones during the ball protocol to reduce anticipation of the robot. This control should not be assumed to apply to every human action in all four tasks.

B.3 Training Details

Training uses eight RTX PRO 6000 GPUs for approximately two hours per policy, with effective batch size 128. Reported iteration times are 0.49 s for VLASH, 0.46 s for RTC and 0.50 s for TEMPO.

RTC is the authors’ reimplementation because an open implementation was unavailable. The paper does not state task demonstration counts, the motion-window length W or detailed probe trajectory splits, limiting exact reproduction from this document alone.

Appendix C Per-Task Failure Mode Breakdown

The authors visually classify each of 50 rollouts by success or the dominant failure. Figures A2 and A3 preserve the complete counts and legends. Categories differ across tasks, so similarly colored segments should be interpreted using each figure’s own legend.

Original chart; see complete figure caption
Figure A2: Failure-mode breakdown on Flick Catch. Each of the 5050 rollouts is decomposed into success plus three failure modes corresponding to the three task stages (grab plate, track hand, catch ball). VLASH’s failures concentrate in hand-tracking and RTC struggles most at tracking the ball’s trajectory (see project page); TEMPO cuts both failure modes.

Flick Catch (Fig. A2 ).

Flick Catch has three stages: grasp the plate, track the hand and catch the ball. VLASH fails at hand tracking in 37 of 50 trials. RTC reaches the tracking stage but loses the ball in 19 trials. TEMPO reduces those counts to nine and four, respectively, and succeeds in 33 trials.

The breakdown is consistent with improved motion awareness. It is a manually assigned, dominant-cause description of these rollouts, rather than an independent intervention on each failure mechanism.

Original chart; see complete figure caption
Figure A3: Failure-mode breakdown on Drop Catch. RTC scores highest as Drop Catch is dominated by reactive timing rather than motion or state ambiguity. VLASH’s dominant failure is premature movement before the human’s hand begins to move; TEMPO cuts this failure by nearly 4×4\times.

Drop Catch (Fig. A3 ).

RTC succeeds in 40 Drop Catch trials, with six lagging failures and four overshoots. VLASH moves before the human hand in 23 trials and succeeds in 19. TEMPO reduces premature movement to six trials and succeeds in 33.

TEMPO’s overshoot count rises from VLASH’s six to seven. The source interprets reduced premature movement as evidence that command history stabilizes task progress, while RTC’s higher overall score shows the continued importance of timing.

Original chart; see complete figure caption
Figure A4: Both TEMPOMOT and TEMPOACT are necessary. Per-frame action MSE for the full TEMPO model (blue) vs. the two single-signal ablations, evaluated on the Flick Catch task. Freezing TEMPOMOT to its first-frame value (green) hurts most at 2.72.7 s, the beginning of the task where human hand starts to move. Freezing TEMPOACT (red) hurts most at 5.35.3 s, the middle of the task where the finger sweeps horizontally across the balls before picking one and flicking it. Thus, each signal plays an important role at a different moment in the task.

Appendix D Input Ablation

For one representative Flick Catch episode, the authors freeze either temporal input at its first-frame value and measure predicted-action mean squared error over time, with exponential smoothing. Freezing motion features hurts most during rapid object motion; freezing history hurts most during the horizontal sweep across the balls before pickup/flick.

At 2.7 s, error rises from 0.019 to 0.059 when motion is frozen, versus 0.033 when history is frozen. At 5.3 s, history freezing gives 0.091, motion freezing 0.051 and the full model 0.020.

This is an open-loop, single-episode prediction analysis. Freezing also changes the input distribution; it does not by itself establish dataset-wide causal necessity or measure closed-loop task success.

Appendix E Motion Encoder Selection

A linear probe predicts annotated pixel-space object velocity from each motion encoder. Average R² is 0.34 for VideoLaVIT, 0.39 for Reducio, 0.46 for SAM 2 and 0.48 for VidTwin, versus 0.08 from object position alone. All per-task values and the four original trajectory images are retained.

The results show useful velocity information in the learned features. Position alone still scores 0.24 on Bottle Handover, illustrating task correlations rather than literal absence of all motion information. SAM 2.1-Tiny balances a strong probe score with streaming speed; VidTwin has the highest probe average and Reducio the highest handover success.

Original complete multi-panel figure layout, including task labels
Initialization for TEMPOMOT Bottle Handover Drop Catch Flick Catch Wine Pour Average
Object Position 0.24 0.03 0.03 0.02 0.08
VideoLaVIT [15] 0.45 0.25 0.28 0.36 0.34
Reducio [28] 0.50 0.36 0.38 0.32 0.39
SAM 2 [24] 0.58 0.37 0.50 0.39 0.46
VidTwin [30] 0.59 0.44 0.43 0.45 0.48
Figure A5: Linear probe of candidate initializations for TEMPOMOT. Top: one example frame per task showing the object of interest (red dot) and its recent trajectory (yellow line); the pixel-space velocity of this object is the regression target the linear probe must predict. Bottom: per-task velocity R2R^{2} for each candidate initialization, along with the average. Higher is better; the raw object-position baseline (R2=0.08R^{2}=0.08 on average) cannot recover velocity from a single position, while every learned motion encoder does so substantially better.
Concept demonstration

Open image ↗