At a glance
Actions from Video Dense Correspondences (AVDC) predicts a task-directed video from an initial image and text, tracks pixels between generated frames, and fits rigid object or camera motion using initial depth. Robot-specific controllers then execute the inferred motion. Training the video predictor needs no action annotations, while deployment still needs depth, camera calibration, and suitable control primitives.
The matching v1 paper reports 43.1% mean success in Meta-World, 31.3% in iTHOR navigation, and 90% over 40 human-to-simulated-robot pushing trials. Its physical-robot appendix reports eight failures in ten trials after finetuning. These distinct outcomes qualify the successful qualitative demonstrations.
1 Introduction
Generated images can describe desired behavior across different embodiments, but a video is not directly a motor command. Prior systems often learn inverse dynamics from action-labeled data. AVDC replaces that learned mapping with geometric motion estimation from dense correspondences and existing grasping, pushing, inverse-kinematics, and position-control routines.
The approach assumes rigid object motion under a fixed manipulation camera, or camera motion in a static navigation scene. A factorized video architecture reduces training cost for small task-specific datasets; the one-day claim does not cover every dataset used in the paper.
2 Related Work
Video-based robot learning includes visual representation learning, reward or subgoal extraction, dynamics prediction, and generated visual plans. UniPi (Du et al., 2023) uses an inverse-dynamics model to translate plans into actions. AVDC instead builds on dense-correspondence geometry to estimate motion without action labels for that learned mapping. It avoids online interaction during video-policy training, while execution relies on engineered robot knowledge.
3 Actions from Video Dense Correspondences
The pipeline has three stages: text-conditioned RGB video generation, optical-flow prediction between adjacent generated frames, and rigid-motion fitting using the current depth image. The inferred object or camera transformations are converted to actions by domain-specific routines.
3.1 Text-Conditioned Video Generation
A conditional diffusion model predicts a short video from the initial frame and text. The paper sets T = 8, although the main description calls these future frames while Appendix H’s sampling includes the initial frame in T. The original denoising objective is retained.
The U-Net concatenates the initial RGB image along the channels of each noisy future frame. A fixed CLIP text encoder supplies conditioning. Each residual block factorizes spatial and temporal convolution: spatial operations act independently at each time, followed by temporal operations at each spatial position. Appendix F gives the full architecture and training parameters.
Equations
3.2 Flow Prediction
The pretrained GMFlow model (Xu et al., 2022) predicts dense pixel correspondences between consecutive synthesized frames. Integrating these fields tracks scene points through the imagined motion. A variant trained to diffuse flow fields directly performs worse in most tasks; the authors suggest sparse or irregular flow distributions make that prediction harder, without establishing this explanation experimentally.
3.3 Action Regression from Flows and Depths
Depth is required for the current observation, not every generated future frame. With known intrinsics, visible pixels become initial 3D points. Their later 2D correspondences constrain a rigid transform through the retained reprojection objective. This is geometric fitting rather than a learned inverse-dynamics network.
Predict object-centric motion
For manipulation, an object mask selects tracked points. The fit minimizes projected-coordinate error under a rigid transform; masks may come from a human or a segmentation model. A grasp or contact point is transformed into successive subgoals, then a position controller executes them. The particular grasp-versus-push heuristics are specified in Appendix H.
For navigation, the same construction tracks the surrounding static scene and inverts its apparent motion to obtain camera motion. The resulting transform is mapped to forward, left, right, or stop actions. Different assumptions are used for the moving object and moving camera settings.
A stalled manipulation trajectory triggers a new plan when robot movement stays below 1 mm for 15 consecutive steps without completing the task. Navigation uses a correspondence-retention trigger instead. Replanning refreshes the current RGB-D observation; it is not continuous high-frequency visual feedback.
Equations
4 Experiments
Evaluation spans simulated manipulation, simulated navigation, human-to-robot pushing transfer, and a small physical-robot trial. Action labels available in simulation train the baselines but are withheld from AVDC’s video learning. Ground-truth or manually provided masks and initial depth remain important execution inputs.
4.1 Baselines and Variants of AVDC
Behavior-cloning baselines receive 15,216 labeled frame–action pairs in Meta-World or 5,757 in iTHOR. They use ResNet-18 image features, camera identity, CLIP text, and a three-layer action predictor, initialized either from scratch or R3M. The UniPi adaptation uses AVDC-generated videos and a learned inverse-dynamics/frame-switching model.
AVDC variants directly predict flow, omit replanning, or use the full two-stage pipeline. Table 1 additionally lists an AVDC (ID) row with the same overall score as the UniPi adaptation; its separate mechanism is not explained in the accompanying variant list, so it is retained without inventing a distinction.
4.2 Meta-World
Eleven Meta-World tasks use a Sawyer arm, three camera views, and five demonstrations per task/view: 165 videos total. Evaluation uses 25 trials for each camera and task, randomizing initial arm and object positions. AVDC receives the target object’s ground-truth mask.
Full AVDC averages 43.1%, compared with 19.6% without replanning, 13.7% for direct flow, 16.2% for behavior cloning from scratch, 15.4% for R3M initialization, and 6.1% for the listed UniPi adaptation. Performance varies widely: door closing reaches 89.3%, while assembly is 6.7% and hammering 8.0%. Direct flow exceeds the full model on top-down button pressing, so the full system is not best on every variant comparison.
More replanning improves the reported curves. Optical-flow error is a prominent failure source, especially for small objects at low resolution, where small pixel errors become large 3D motion errors.
| door-open | door-close | basketball | shelf-place | btn-press | btn-press-top | |
| BC-Scratch | 21.3% | 36.0% | 0.0% | 0.0% | 34.7% | 12.0% |
| BC-R3M | 1.3% | 58.7% | 0.0% | 0.0% | 36.0% | 4.0% |
| UniPi (With Replan) | 0.0% | 36.0% | 0.0% | 0.0% | 6.7% | 0.0% |
| AVDC (ID) | 0.0% | 36.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| AVDC (Flow) | 0.0% | 0.0% | 0.0% | 0.0% | 1.3% | 40.0% |
| AVDC (No Replan) | 30.7% | 28.0% | 21.3% | 8.0% | 34.7% | 17.3% |
| AVDC (Full) | 72.0% | 89.3% | 37.3% | 18.7% | 60.0% | 24.0% |
| faucet-close | faucet-open | handle-press | hammer | assembly | Overall | |
| BC-Scratch | 18.7% | 17.3% | 37.3% | 0.0% | 1.3% | 16.2% |
| BC-R3M | 18.7% | 22.7% | 28.0% | 0.0% | 0.0% | 15.4% |
| UniPi (With Replan) | 4.0% | 9.3% | 13.3% | 4.0% | 0.0% | 6.1% |
| AVDC (ID) | 4.0% | 9.3% | 13.3% | 4.0% | 0.0% | 6.1% |
| AVDC (Flow) | 42.7% | 0.0% | 66.7% | 0.0% | 0.0% | 13.7% |
| AVDC (No Replan) | 12.0% | 17.3% | 41.3% | 0.0% | 5.3% | 19.6% |
| AVDC (Full) | 53.3% | 24.0% | 81.3% | 8.0% | 6.7% | 43.1% |
4.3 iTHOR
iTHOR evaluates twelve target-object tasks across four room types, with twenty episodes per object. The robot chooses forward motion, left/right rotation, or done. Success requires the target in view and within 1.5 m under the evaluation budget; no object mask is used.
AVDC averages 31.3%, versus 2.1% and 0.4% for the two cloning baselines. The prose says the baselines fail entirely, but Table 2 records these small nonzero rates. Navigation actions come from whole-scene motion, with duplicated final training frames teaching a stopping cue.
| Room | BC-Scratch | BC-R3M | AVDC |
| Kitchen | 1.7% | 0.0% | 26.7% |
| Living Room | 3.3% | 0.0% | 23.3% |
| Bedroom | 1.7% | 1.7% | 38.3% |
| Bathroom | 1.7% | 0.0% | 36.7% |
| Overall | 2.1% | 0.4% | 31.3% |
4.4 Cross-Embodiment Learning: From Human Videos to Robot Execution
A video model trained for 10,000 steps on human pushing videos controls a simulated robot without finetuning and succeeds in 36 of 40 trials. The text reports 198 training videos, while Figure 4’s caption says 195. This is a specific pushing-transfer result, not an unrestricted cross-embodiment guarantee.
4.5 Real-World Franka Emika Panda Arm with Bridge Dataset
The physical setup uses a Franka Panda and a calibrated, fixed RealSense D435. A model pretrained on Bridge’s WidowX videos does not transfer adequately to this camera and scene, so it is finetuned on twenty local human demonstrations. Evaluation assumes a top grasp, no required object reorientation, and manual target segmentation.
Figures 9–10 show successful examples. The complete v1 quantitative result is in Appendix H.3: eight failures in ten pick-and-place trials, implying two successes. Six failures select the wrong object or destination and two lose the object during generated intermediate frames. This small result does not support reliable general physical manipulation.
5 Discussion
Occlusion, changing illumination, large motion, and low image resolution degrade correspondence tracking. Different robot hands still require suitable grasp/contact modules, and RGB demonstrations do not directly supply forces. The physical experiment also exposes semantic planning and video-continuity failures.
The contribution is a way to connect learned visual plans to geometric control without action-labeled policy training. Its demonstrated reach depends on rigid-motion assumptions, depth and calibration, masks for manipulation, and engineered action conversion; richer contact and force reasoning remain open.
Appendix
The appendices retain implementation details, additional comparisons, compute costs, and the complete physical-trial failure analysis.
Appendix A Code
The authors identify the codebase_AVDC directory in the submitted supplementary archive as the reproduction code. The project page links the associated materials.
Appendix B Extended Qualitative Results
The supplementary website contains generated-video and task-execution examples for Meta-World, iTHOR, Visual Pusher, Bridge, and the physical Panda setup. These qualitative videos supplement the retained static figures.
Appendix C Zero-Shot Generalization on Real-World Scenes
Bridge-only models can generate plausible videos for some unseen kitchen scenes, as shown in the zero-shot video examples. This concerns visual prediction; the physical robot evaluation still uses twenty local finetuning videos.
Appendix D Object mask with segmentation models
Replacing ground-truth Meta-World masks with Language Segment-Anything, built from GroundingDINO and Segment Anything, lowers mean task success from 43.1% to 34.5%, an 8.6-percentage-point loss. Figure 11 shows both correct and incorrect masks, making the effect of segmentation quality explicit.
Appendix E Additional Ablation Studies
The additional ablations evaluate generated final-frame pixel error. They measure video reconstruction quality rather than independently establishing downstream action success.
E.1 First-Frame Conditioning
Concatenating the initial image along RGB channels outperforms adding it as a separate time frame during early Bridge training. Each plotted point averages final-frame MSE over 4,000 generated samples; error bars are standard errors. Multimodal video quality is only partly represented by this pixel metric.
E.2 Text Encoder
The 63-million-parameter CLIP text encoder and a 110-million-parameter T5-base alternative show similar final-frame MSE in this comparison. The plot does not establish that their language generalization or action performance is interchangeable.
Appendix F Model Architecture and Training Detail
The text encoder, attention pooling, video architecture, and optimization settings are specified separately. Tables 3–4 retain all numerical hyperparameters.
F.1 Text Encoder
A frozen CLIP text encoder produces tokens, which a learned Perceiver attention-pooling module aggregates into conditioning added to the diffusion timestep embedding. The implementation does not use text cross-attention inside the video model. Table 3 gives its pooling dimensions and layers.
| Parameter | Value |
| layers | 2 |
| num_attn_heads | 8 |
| num_head_channels | 64 |
| num_output_tokens | 64 |
| num_output_tokens_from_pooled | 4 |
| max_seq_len | 512 |
| ff_expansion_factor | 4 |
F.2 Video Diffusion Model
All models use zero dropout, 32 channels per attention head, 100 training/inference diffusion steps, a cosine schedule, v-prediction, L2 loss, minimum-SNR gamma 5, and learning rate 10⁻⁴. Exponential moving averages update every ten steps with decay 0.999.
Table 4 reports 201M/109M/166M parameters for Meta-World/iTHOR/Bridge, respectively, with image resolutions 128×128, 64×64, and 48×64. The appendix’s v-prediction implementation and the main text’s noise-prediction objective are both retained rather than silently treated as identical parameterizations.
| Meta-World | iTHOR | Bridge | |
| num_parameters | 201M | 109M | 166M |
| resolution | (128, 128) | (64, 64) | (48, 64) |
| base_channels | 128 | 128 | 160 |
| num_res_block | 2 | 3 | 3 |
| attention_resolutions | (8, 16) | (4, 8) | (4, 8) |
| channel_mult | (1, 2, 3, 4, 5) | (1, 2, 4) | (1, 2, 4) |
| batch_size | 16 | 32 | 32 |
| training_timesteps | 60k | 80k | 180k |
Appendix G Hardware and complexity analysis
Reported efficiency depends on the dataset, hardware, diffusion sampler, and number of replans. Training time and online planning latency should be interpreted separately.
G.1 Training cost
Training uses four 32 GB V100 GPUs. Small Meta-World and iTHOR models take about 24 hours; Bridge pretraining takes 48 hours followed by four hours of local finetuning. The runtime appendix describes approximately 40,000 Bridge videos, whereas the main dataset description specifies 33,078; these source counts are not reconciled.
G.2 Inference cost
On an RTX 3080 Ti, video synthesis takes about 10.57 s, optical flow 0.28 s per adjacent pair, action regression 1.31 s, and controller execution 1.53 s on average. The stated 1.51 s per frame is part of the generation report, not a real-time controller rate.
G.3 Replanning cost
A Meta-World planning round takes about 18 s. Allowing five replans gives one to six planning rounds, or approximately 18–108 s of planning per episode on the stated GPU.
G.4 Improving Inference Efficiency with Denoising Diffusion Implicit Models
Using DDIM with ten denoising steps accelerates video generation by about tenfold while reducing Meta-World success from 43.1% to 37.5%, a 5.6-percentage-point drop. The speedup applies to generation, not automatically the entire execution pipeline. Distillation, lighter flow models, and parallel action fitting are suggested further improvements.
Appendix H Details on Experimental Setup
These details specify the task-dependent training samples, tracking filters, action heuristics, replanning triggers, and physical hardware that connect a generated video to execution.
H.1 Meta-World Experimental Setup
The Meta-World system combines goal-directed frame sampling, masked rigid-motion fitting, and a fixed grasp-or-push mode.
H.1.1 Learning the Diffusion Model
Training samples a current frame, uniformly chosen intermediate frames from the remaining demonstration, and the final frame. Keeping the terminal state anchors the goal while varying intermediate spacing. This appendix describes T total frames, unlike the main text’s future-frame notation.
H.1.2 Calculating Object Rigid Transformations and Inferring Actions
The implementation samples 500 points from the object mask and uses their centroid as a contact point. RANSAC rejects inconsistent 2D correspondences before fitting 3D rigid motion and transforming the contact into successive subgoals.
A vertical excursion above 10 cm selects grasping; otherwise the policy pushes. Grasping closes the gripper at the contact and follows subgoals. Pushing initializes the hand behind the object along the direction toward the first subgoal more than 10 cm away. These domain heuristics supply action knowledge absent from the actionless training videos.
H.1.3 Replanning Strategy
Replanning regenerates video from the current image and recomputes subgoals with the current mask and depth. It keeps the initially selected grasp/push mode. Pushing resets the hand’s approach position; grasping proceeds toward the updated subgoals.
H.2 iTHOR Experimental Setup
Navigation uses consecutive video frames, whole-scene correspondences, a discrete action mapper, and a different replanning criterion.
H.2.1 Learning the Diffusion Model
A training segment contains a sampled current frame and consecutive subsequent frames, without manipulation’s temporal skipping. If the demonstration ends early, the last frame is repeated to represent stopping at the target.
H.2.2 Calculating Scene Transformations and Inferring Actions
Scene tracking uses an optical-flow magnitude threshold of one pixel and removes points that leave the image; unlike manipulation, it does not apply RANSAC. The source’s wording about thresholding moving points is ambiguous, so the precise inclusion direction is not inferred here.
The fitted scene transform moves an imaginary point one metre ahead. Motion below 1 mm maps to done; otherwise horizontal displacement below 25 cm maps to forward motion, and larger signed displacement selects left or right rotation. These thresholds define a heuristic action conversion.
H.2.3 Replanning Strategy
A new navigation plan is generated when fewer than 10% of the initially sampled correspondences remain. Video prediction, flow, and action inference are then repeated from the current observation.
H.3 Real-World Franka Emika Panda Arm Experimental Setup
The Panda uses a two-jaw parallel gripper and joint-position control. Camera intrinsics and extrinsics map predicted motion into world coordinates. Twenty human demonstrations move fruit into plates or bowls, with utensils as distractors.
A human specifies the target mask. Assuming a top grasp without reorientation, the controller uses initial and final predicted object poses with inverse kinematics to generate the arm trajectory. Small tracked objects make recovered 3D rotations unreliable.
The matching v1 PDF explicitly reports eight failures out of ten trials. Seventy-five percent of those failures select the wrong object or target; the remaining quarter produce a discontinuous video in which the object disappears. Successful screenshots must be read alongside this full quantitative outcome.