Literature
Display
← All papers

TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks

Zhi Cao · Howard Ji · Kevin Zhang · Kuangzhi Ge · Li Fei-Fei · Jiajun Wu · Huang Huang

Preprint posted · arXiv (preprint) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

TrAct links candidate robot actions to imagined visual outcomes through jointly predicted point tracks.

Refer to caption
Figure 1: Overview of TrAct. TrAct uses VLAT (left, purple), pretrained on large-scale cross-embodiment data, to propose candidate action–track pairs. TWM (middle, gray) rolls out the visual outcome for each candidate conditioned on the tracks, and VLAC (middle, green) selects the highest-reward rollout; the robot executes the paired action. Visual tracks thus serve as an intermediate interface between future prediction and robot control. We evaluate TrAct on diverse simulation and physical robot tasks (right).

At a glance

TrAct generates paired action chunks and image-space tracks, predicts videos conditioned on those tracks, and uses a learned visual-language critic to select an action. On the proposed shifted-task simulation benchmark, success rises from 27% for the π0.5 baseline to 55%; the action-conditioned world-model variant reaches 49%. Physical task/background trials reach 76% versus 49% and 66% respectively. The method adds substantial pretraining and inference work, with a reported 2.431 seconds of computation per sixteen-step action chunk.

1 Introduction

Robot commands do not directly specify how pixels move: their visual consequences depend on embodiment, geometry, and contact. TrAct predicts image-space motion as an intermediate representation to make future-video generation easier, then executes the robot action paired with the selected track. Tracks are a shared visual interface, but they do not themselves guarantee that the paired action will realize the imagined future.

2 Related Work

Related work uses video models for prediction, tracks or flow for transferable motion representations, and visual-language critics for action selection. TrAct combines these components in a propose–predict–score loop. Its contribution is the action–track coupling and track-conditioned visual prediction within that loop, rather than a new physical simulator or a verified action-to-video correspondence.

3 Methods

The framework contains a joint action/track policy, a conditional video generator, and a critic that scores task progress from predicted videos.

Refer to caption
Figure 2: Overview of the TrAct framework. Given an observation oto_{t} and language instruction ll, the VLAT model jointly predicts KK candidate actions and corresponding visual tracks. Each set of predicted tracks conditions a world model to generate future video frames. A VLM-based reward model scores the predicted outcomes and selects the action whose rollout best fits the instruction.

3.1 Policy Architecture

The policy extends π0.5 [5] with separate track and action projections sharing a flow-matching action expert. Track tokens precede actions, so actions can attend to tracks while the reverse is masked. The agent view predicts seven gripper points; the wrist view predicts those seven points plus a 5 × 5 scene grid to represent camera-relative scene motion.

The video model uses Stable Video Diffusion [36] with a temporal ControlNet receiving rasterized track heat maps. The action-conditioned comparison instead injects encoded action vectors and view embeddings through attention. Both train on ground-truth conditioning but receive predicted candidates at evaluation. A task-finetuned visual-language action critic [33] scores imagined outcomes from zero to fifty. This is a learned progress score, not a calibrated success probability.

Equations 1–3
πθ​(ot,l)⟶{(ai,τi)}i=1K,\pi_{\theta}(o_{t},l)\;\longrightarrow\;\bigl\{(a_{i},\;\tau_{i})\bigr\}_{i=1}^{K}, (1)
ℒVLAT=ℒflow​(a,a∗)+ℒflow​(τ,τ∗),\mathcal{L}_{\text{VLAT}}=\mathcal{L}_{\text{flow}}(a,a^{*})+\mathcal{L}_{\text{flow}}(\tau,\tau^{*}), (2)
v^i=pϕ​(ot,τi)∈ℝH×C×Wimg×Himg,\hat{v}_{i}=p_{\phi}(o_{t},\tau_{i})\in\mathbb{R}^{H\times C\times W_{\text{img}}\times H_{\text{img}}}, (3)

3.2 Pretraining

Pretraining transfers visual motion structure from robot and human data, with different supervision available for each source.

VLAT.

The main policy uses 76,000 DROID trajectories [39] and a 150,000-clip EgoDex subset [40], mixed 1:2, for thirty thousand steps at batch size sixty-four on four H100 accelerators. Robot poses and calibration project gripper keypoints, while CoTracker [16] supplies image-point trajectories. Human video supervises visual tracks only; it does not supply executable robot action labels.

World Model.

Both video-model variants pretrain on 76,000 DROID trajectories for thirty thousand steps at batch size sixty-four, then use the same simulation or physical fine-tuning data. This keeps the world-model comparison more controlled than the direct comparison against an official pretrained policy checkpoint.

3.3 Inference

The policy samples twenty candidates in simulation or sixteen on hardware, using increased initial noise and temperature-scaled resampling for diversity. Each candidate’s tracks condition a video rollout; the critic selects the highest-scoring rollout and the corresponding action chunk is executed. The process reobserves after the full sixteen-step chunk, not after each individual predicted action.

Equation 4
i∗=arg⁡maxi∈{1,…,K}​Rψ​(v^i,l),i^{*}=\arg\max_{i\in\{1,\ldots,K\}}\;R_{\psi}(\hat{v}_{i},\,l), (4)

4 Simulation Experiments

Four variants compare the official policy, joint action/track prediction alone, action-conditioned video selection, and track-conditioned selection. The largest headline improvement includes policy architecture, additional pretraining, and selection changes. Comparing the two video-selection variants holds the candidate generator and critic fixed and more directly tests the conditioning interface.

4.1 LIBERO-INTEGRAL Benchmark

LIBERO-INTEGRAL is a twenty-task suite constructed from ten robustness cases and ten UR5 replacements of LIBERO-10 tasks. Robustness cases cover object swaps, object/task changes, difficult camera views, and altered initial robot states. It is a selected benchmark assembled by the authors, not the complete union of all source robustness tasks.

Model Finetuning.

The π0.5 baseline uses its official LIBERO checkpoint. Other variants fine-tune on two thousand LIBERO episodes; the joint policy trains for five thousand steps and each video model for twenty-five thousand. The critic is fine-tuned on predicted rollouts and completion rewards. Projected gripper points and tracked wrist-view grid points supervise the sixteen-step predictions.

4.2 Video Generation Results

On one hundred generated evaluation videos, track conditioning improves all five listed video-quality metrics in both views. Agent-view peak signal-to-noise ratio rises from 15.12 to 24.51 and perceptual distance falls from 0.438 to 0.106. These measure image/video resemblance; better perceptual scores alone do not verify contact dynamics or that selected actions achieve the predicted motion.

Refer to caption
Figure 3: Qualitative video comparison. TWM produces more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
Table 1: Video generation quality of action-conditioned and track-conditioned world models. Results are reported for real-world and simulation scenes under agent-view and wrist-view cameras. Act. denotes action-conditioned world model, and Track denotes track-conditioned world model. ↑\uparrow indicates higher is better and ↓\downarrow indicates lower is better. Best results for each scene are in bold.
Metric Real / Agent Real / Wrist Sim / Agent Sim / Wrist
Act. Track Act. Track Act. Track Act. Track
PSNR ↑\uparrow 20.70 21.27 16.89 20.62 15.12 24.51 16.25 22.71
SSIM ↑\uparrow 0.767 0.802 0.621 0.738 0.482 0.843 0.668 0.822
LPIPS ↓\downarrow 0.126 0.111 0.372 0.225 0.438 0.106 0.452 0.182
FID ↓\downarrow 105 95 176 130 248 156 222 92
FVD ↓\downarrow 148 105 246 140 129 38 207 162

4.3 Robot Experiment Results

Each simulation task uses ten rollout episodes. All variants predict and execute sixteen-step action chunks. Standard-suite results and the harder shifted-task results address different performance regimes.

Table 2: Success rates (%) on four standard LIBERO benchmark suites. Each suite contains 100 evaluation episodes. The best result in each column is shown in bold.
Method LIBERO-10 Goal Object Spatial Avg.
π0.5\pi_{0.5} 94.0 96.0 98.0 99.0 96.8
VLAT 94.0 98.0 100.0 100.0 98.0
VLAT+AWM 94.0 98.0 100.0 100.0 98.0
TrAct 94.0 100.0 99.0 100.0 98.3
Table 3: Results on LIBERO-INTEGRAL. Scores denote average task success rates. LIBERO-INTEGRAL combines robustness variations from LIBERO-PRO and LIBERO-Plus, and cross-embodiment transfer with a UR5 robot. Best results in each column are bolded.
Method Swap Object Task Camera RobotInit Cross-Emb. Avg.
π0.5\pi_{0.5} 0.40 0.35 0.15 0.45 0.45 0.17 0.27
VLAT 0.50 0.40 0.35 0.45 0.55 0.42 0.44
VLAT+AWM 0.65 0.50 0.50 0.45 0.55 0.44 0.49
TrAct 0.65 0.60 0.50 0.60 0.65 0.50 0.55

Standard LIBERO Suites.

Across the four standard suites, average success is 96.8% for π0.5, 98.0% for the joint policy and action-conditioned selection, and 98.3% for TrAct. Gains are small near saturation. TrAct is not best in every individual suite: the joint policy reaches 100% on the object suite versus TrAct’s 99%.

LIBERO-INTEGRAL.

On LIBERO-INTEGRAL, average success is 0.27 for π0.5, 0.44 for the joint policy, 0.49 with action-conditioned selection, and 0.55 with track-conditioned selection. UR5-task success rises from 0.17 to 0.50. TrAct improves or ties the action-conditioned variant in most cases, but not all individual tasks; the full per-task table is retained. Additional evaluation-seed runs support the approximately six-point selection gain without measuring independent training-seed variability.

5 Physical Robot Experiments

Physical experiments add a π0.5-plus-critic/action-world-model baseline, allowing the effect of the candidate generator to be examined separately from visual selection.

5.1 Physical Setup

A Franka Panda uses a fixed external camera and a wrist camera. Four hundred demonstrations at fifteen frames per second cover four training tasks. Evaluation changes the manipulated object, noodle length, cabinet opening, or pouring orientation across five tasks, each on the original and a visually different background. These are controlled manipulation variations, not entirely unrelated skill families.

5.2 Model Finetuning

Both policies fine-tune on the same four hundred demonstrations for five thousand steps. Both video models use twenty-five thousand fine-tuning steps, and the critic uses four thousand. Their differing pretraining histories remain part of the overall system comparison.

5.3 Video Generation Results

The physical-video comparison also uses one hundred evaluation rollouts and favors track conditioning on all reported metrics. Wrist-view perceptual distance falls from 0.372 to 0.225 and video distribution distance from 246 to 140. These aggregate video metrics complement rather than replace physical task completion.

5.4 Robot Experiment Results

Each task/background combination has ten trials, giving one hundred attempts per model across the ten conditions. Average success is 49% for π0.5, 65% for π0.5 with action-world-model selection, 55% for the joint policy, 66% for the joint policy with action-conditioned selection, and 76% for TrAct. Against the latter action-conditioned variant, track selection improves 76% to 82% on the original background and 56% to 70% on the changed background. Some individual conditions tie, and repeated physical training/evaluation confidence intervals are not reported.

Table 4: Success rates on five unseen physical robot manipulation tasks, each evaluated under a regular training background (R) and a visually different non-training background (D). We compare five models. The best result for each row is shown in bold.
Task Bg. π0.5\pi_{0.5} π0.5\pi_{0.5}+VLAC VLAT VLAT+AWM TrAct
Put the green block in the bowl R 0.5 0.7 0.6 0.7 0.7
D 0.4 0.5 0.4 0.5 0.7
Put the half pink noodle in the bowl R 0.6 0.9 0.8 1.0 1.0
D 0.6 0.8 0.6 0.8 0.8
Put the hotdog bun in the bowl R 0.6 0.8 0.6 0.8 0.8
D 0.4 0.6 0.5 0.6 0.6
Close the cabinet door R 0.5 0.6 0.5 0.6 0.8
D 0.3 0.4 0.4 0.4 0.7
Pour the pasta into the pan R 0.5 0.7 0.6 0.7 0.8
D 0.5 0.5 0.5 0.5 0.7
Average 0.49 0.65 0.55 0.66 0.76
Refer to caption
Figure 4: Rollout Demonstrations of two tasks: put the half-length pink noodle in the bowl (left) and close the right cabinet door (right). Each step is overlaid with track predictions.

6 Conclusion

The main practical limitation is computation: with sixteen candidates in parallel on L40S hardware, one planning call takes 2.431 seconds, dominated by 1.810 seconds of video generation. That call produces a sixteen-step action chunk; it is not a 2.431-millisecond low-level controller. Candidate count trades computation for choice quality. Other open issues include adaptive tracking points, longer horizons, broader manipulation, and the reliability of imagined outcomes and learned critic scores.

Appendix A Pseudo-code for TrAct Inference Procedure

The retained algorithm gives the full inference loop: propose action–track pairs, generate and score videos, execute the selected chunk, then observe again. The original nested-loop indentation is preserved.

Algorithm 1 TrAct Inference for WM-Guided Action Selection
1: Observation oto_{t}, language instruction ll, number of candidates KK
2: VLAT policy πθ\pi_{\theta}, track-conditioned world model pϕp_{\phi}, VLAC scorer RψR_{\psi}
3: while task not finished do
4:   {(ai,τi)}i=1K←Propose​(πθ,ot,l,K)\{(a_{i},\tau_{i})\}_{i=1}^{K}\leftarrow\textsc{Propose}(\pi_{\theta},o_{t},l,K) ⊳\triangleright sample candidate action–track pairs
5:   for i=1i=1 to KK do
6:    v^i←pϕ​(ot,τi)\hat{v}_{i}\leftarrow p_{\phi}(o_{t},\tau_{i}) ⊳\triangleright roll out future video with predicted tracks
7:    si←Rψ​(v^i,l)s_{i}\leftarrow R_{\psi}(\hat{v}_{i},l) ⊳\triangleright score predicted outcome
8:   end for
9:   i⋆←arg⁡maxi⁡sii^{\star}\leftarrow\arg\max_{i}s_{i}
10:   Execute selected action chunk ai⋆a_{i^{\star}}
11:   Observe new state ot+Ho_{t+H}
12:   t←t+Ht\leftarrow t+H
13: end while

Appendix B Pretraining Data and Unified Track Representation

The track representation uses fixed view-specific slots and masks so different robot and human datasets can share visual supervision.

Unified Track Slot Layout.

Slots are fixed token positions, not learned object identities. DROID uses seven gripper points in the external view and seven plus twenty-five background points in the wrist view. EgoDex supplies fourteen hand points plus twenty-five background points in its single egocentric view. Missing streams or keypoints are masked. The larger-mixture appendix adds BridgeData V2 tracks obtained from images because camera extrinsics are unavailable.

Track Tokenization and Masked Objective

Learned view embeddings distinguish camera streams without adding embodiment labels. Masks suppress unavailable track slots and action dimensions in attention and losses. Human clips train only the track branch; robot datasets additionally train actions in their valid control format. This shares visual supervision while keeping action labels embodiment-dependent.

Equations
zt,v,i=ϕ⁡(τt,v,i)+eview​(v)+eslot​(i)+etime​(t),z_{t,v,i}=\phi(\tau_{t,v,i})+e_{\mathrm{view}}(v)+e_{\mathrm{slot}}(i)+e_{\mathrm{time}}(t),
ℒ=ℒtrack+𝟙robot​ℒaction,\mathcal{L}=\mathcal{L}_{\text{track}}+\mathbb{1}_{\text{robot}}\mathcal{L}_{\text{action}},

Appendix C Inference Hyperparameters

The main settings use twenty simulation candidates with resampling parameter three and sixteen physical candidates with parameter four; both use noise scale two. The latency table reports 52 ms for the joint policy, 1,810 ms for the video model, and 569 ms for the critic at sixteen candidates. It does not report a complete candidate-count latency curve or controller-level real-time throughput.

Table 5: Inference latency breakdown per planning step, measured with K=16K{=}16 candidates, rolled out in parallel, on NVIDIA L40S GPUs. Each planning step yields a 16-step action chunk.
Component VLAT World Model VLAC Total
Latency (ms) 52 1810 569 2431

Appendix D Additional Qualitative Video Comparison

Additional complete video-frame montages compare both views and show examples of spatial drift, shape distortion, and motion inconsistency in action-conditioned predictions. These are qualitative examples, with the aggregate video metrics reported separately.

Refer to caption
Figure 5: Additional Qualitative video comparison. TWM continues to produce more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.

Appendix E Physical Experiment Setup

The complete setup and rollout figures retain all five evaluation variations under both backgrounds. They show the scope of the physical generalization claim and the sixteen-step predicted tracks, alongside the four training-task examples.

Refer to caption
Figure 6: Overview of the physical experiments. We use a base and wrist-view camera and finetune the model on four training tasks. We then test five out-of-distribution tasks, as well as an unseen background for each task.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 7: Rollout examples Rollout with four key frames showing 16-horizon track predictions under the training background (left) and unseen background (right). Task list from top row to bottom row: put the half-length pink noodle in the bowl (1), put the hotdog bun in the bowl (2), put the green brick in the bowl (3), pour the angled scooper’s pasta into the pan (4), and close the partially open right cabinet door (5).

Appendix F Simulation Results

Simulation supplements list the selected tasks, full per-task outcomes, evaluation-seed repeats, and candidate-count ablation.

F.1 Simulation Experiment Setup

The ten robustness cases contain two examples from each selected variation category. Ten additional tasks substitute a UR5 for the Franka embodiment. The retained rollout montages and task table identify every case, making the benchmark’s selection and coverage explicit.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 8: Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: pick the alphabet soup and place it in the basket (1), put both the cream cheese box and the butter in the basket (2), put both moka pots on the stove (3), put the yellow and white mug in the microwave and close it (4), pick the akita black bowl next to the ramekin and place it on the plate (5), put the plate on the stove (6), put the white mug on the left plate and put the yellow and white mug on the right plate (7), put the wine bottle on the top of the drawer (8), put both the cream cheese box and the butter in the basket (9), and pick up the book and place it in the back compartment of the caddy (10).
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 9: UR5 Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: put both the alphabet soup and the cream cheese box in the basket (1), put both the alphabet soup and the tomato sauce in the basket (2), put the bowl on the stove (3), pick the orange juice and place it in the basket (4), pick the akita black bowl between the plate and the ramekin and place it on the plate (5), pick the akita black bowl next to the plate and place it on the plate (6), pick the ketchup and place it in the basket (7), pick the tomato sauce and place it in the basket (8), pick the akita black bowl on the cookies box and place it on the plate (9), and put the bowl on the plate (10).

F.2 Per-Task LIBERO-INTEGRAL Results

TrAct is best or tied best on eighteen of twenty tasks. It is worse than action-conditioned selection for ketchup pickup and for the bowl-on-cookies-box case. These exceptions qualify the aggregate generalization benefit and show that improved world modeling does not guarantee improvement on every action-selection problem.

Table 6: Results on the LIBERO-INTEGRAL benchmark. The benchmark includes robustness variations from LIBERO-PRO and LIBERO-Plus, and the last ten tasks evaluate cross-embodiment generalization by replacing the Franka Panda with a UR5 robot. Scores denote task success rates. The best result for each row is shown in bold.
Task Var. 𝝅0.5\bm{\pi_{0.5}} VLAT VLAT+AWM TrAct
Pick the alphabet soup and place it in the basket Swap 0.4 0.5 0.7 0.7
Put both the cream cheese box and the butter in the basket Swap 0.4 0.5 0.6 0.6
Put both moka pots on the stove Object 0.5 0.5 0.6 0.7
Put the yellow and white mug in the microwave and close it Object 0.2 0.3 0.4 0.5
Pick the akita black bowl next to the ramekin and place it on the plate Task 0.1 0.3 0.5 0.5
Put the plate on the stove Task 0.2 0.4 0.5 0.5
Put the white mug on the left plate and put the yellow and white mug on the right plate Camera 0.4 0.3 0.3 0.5
Put the wine bottle on the top of the drawer Camera 0.5 0.6 0.6 0.7
Put both the cream cheese box and the butter in the basket RobotInit 0.4 0.5 0.5 0.6
Pick up the book and place it in the back compartment of the caddy RobotInit 0.5 0.6 0.6 0.7
Put both the alphabet soup and the cream cheese box in the basket UR5 0.1 0.2 0.2 0.3
Put both the alphabet soup and the tomato sauce in the basket UR5 0.3 0.6 0.6 0.7
Put the bowl on the stove UR5 0.2 0.3 0.3 0.5
Pick the orange juice and place it in the basket UR5 0.0 0.3 0.4 0.5
Pick the akita black bowl between the plate and the ramekin and place it on the plate UR5 0.1 0.5 0.5 0.6
Pick the akita black bowl next to the plate and place it on the plate UR5 0.1 0.3 0.3 0.5
Pick the ketchup and place it in the basket UR5 0.0 0.1 0.2 0.1
Pick the tomato sauce and place it in the basket UR5 0.0 0.2 0.2 0.2
Pick the akita black bowl on the cookies box and place it on the plate UR5 0.5 0.9 0.9 0.8
Put the bowl on the plate UR5 0.4 0.8 0.8 0.8
Average – 0.27 0.44 0.49 0.55

F.3 Statistical Robustness and Candidate Scaling

The additional analyses keep trained models fixed while varying evaluation randomness or the number of candidate proposals.

Table 7: LIBERO-INTEGRAL success rates across three independent evaluation seeds. SE denotes the standard error over seeds and CI the corresponding 95% confidence interval.
Method Mean Success SE 95% CI
VLAT+AWM 0.490 0.006 [0.47, 0.51]
TrAct 0.547 0.003 [0.53, 0.56]
Table 8: Effect of the number of sampled candidates KK on LIBERO-INTEGRAL average success rate, using VLAC selection over track-conditioned world model rollouts.
Selection Method 𝑲=𝟓\bm{K=5} 𝑲=𝟏𝟎\bm{K=10} 𝑲=𝟐𝟎\bm{K=20}
VLAC selection 0.52 0.54 0.55

Multiple Seeds.

Across three evaluation seeds, action-conditioned selection averages 0.490 with standard error 0.006, and TrAct averages 0.547 with standard error 0.003. The source reports nonoverlapping 95% intervals of [0.47, 0.51] and [0.53, 0.56]. These are intervals over a small number of evaluation seeds with fixed checkpoints; they do not include training randomness or validate the physical result statistically.

Number of Sampled Candidates.

Five, ten, and twenty candidates yield 0.52, 0.54, and 0.55 average success. The smaller candidate set retains most of the measured benefit, but no matching latency measurements are provided for those three settings. The main simulation score uses twenty candidates.

Appendix G Larger Pretraining Mixture Experiments

The final appendix tests a larger pretraining configuration on a selected difficult subset rather than the entire benchmark.

Table 9: Additional results on the five most difficult UR5 cross-embodiment tasks from LIBERO-INTEGRAL. TrAct+ denotes TrAct pretrained with the larger DROID+BridgeData V2+EgoDex mixture. Scores denote task success rates. The best result in each row is shown in bold.
Task 𝝅0.5\bm{\pi_{0.5}} TrAct TrAct+
Put both the alphabet soup and the cream cheese box in the basket (UR5) 0.1 0.3 0.3
Pick the ketchup and place it in the basket (UR5) 0.0 0.1 0.2
Put the bowl on the stove (UR5) 0.2 0.5 0.6
Pick the orange juice and place it in the basket (UR5) 0.0 0.5 0.5
Pick the tomato sauce and place it in the basket (UR5) 0.0 0.2 0.3
Average 0.06 0.32 0.38

Larger Pretraining Mixture.

TrAct+ adds sixty thousand BridgeData V2 trajectories and expands EgoDex from 150,000 to about 300,000 clips. It also doubles steps to sixty thousand and batch size to 128 on eight H100s, so data diversity and optimization budget change together. On five UR5 tasks selected because the original model performs poorly, average success rises from 0.32 to 0.38. This is subset evidence for the larger configuration, not an isolated data-scaling effect or a full-benchmark 38% score.

Concept demonstration

Open image ↗