Overview
This edition reviews arXiv v1 dated 31 December 2025, including all ten appendix sections. All five authors are from Stanford; Li Fei-Fei and Ruohan Zhang share advising, and Wenlong Huang is the corresponding author. Figure numbering follows the PDF, correcting the HTML’s restart after its teaser.
At a glance
Dream2Flow generates a task video, reconstructs the manipulated object’s 3D motion and asks a robot to reproduce that motion. Object trajectories bridge human-led video generation and robot-specific action spaces. Pushing uses learned particle dynamics, real grasped manipulation uses rigid-grasp optimization, and simulated door opening uses reinforcement learning.
The video and perception models require no task-specific demonstrations. This does not mean the whole system is untrained: pushing dynamics learns from 500 simulated transitions, and door policies train in simulation. Hardware evidence includes rigid, articulated and deformable tasks, with lower success on cloth and several qualitative additional cases.
I Introduction
Video models often depict human-object interactions better than robot interactions. Directly copying a generated hand trajectory would impose the wrong embodiment, while the object’s desired state change can be shared across robots. Dream2Flow therefore extracts object motion and lets a controller solve for the required robot actions.
The interface supports motion planning and reward-based policy training, but each domain still needs a suitable dynamics model or simulator, calibrated observation and robot-specific controls. Hand reconstruction is also used to select grasps, so the pipeline retains some human-interaction information beyond object flow.
The author project page provides videos and interactive flow visualizations. The manuscript’s zero-shot language refers to using pretrained video/perception models and avoiding task-specific human demonstrations; it should not obscure the learned dynamics, task-specific RL or designed prompts used below.
II Related Works
The paper places object flow between high-level intent and embodiment-specific control, drawing on task specification, dense tracking and video prediction.
II-A Task Specification in Manipulation
Task specifications range from symbolic goals and costs to language policies, goal images, forces, object descriptors and foundation-model-generated programs or spatial maps. Dream2Flow expresses the desired change as a time-indexed object trajectory, leaving the controller to realize it.
II-B 2D/3D Flow in Robotics
Optical flow, point tracks and scene flow can condition policies or provide motion targets. Dream2Flow follows the object-centric branch: reconstruct a generated trajectory independently of the actor, then track it under robot constraints. It builds on actionless-video correspondences such as AVDC and differs from fitting one rigid pose at every frame.
II-C Video Models for Robotics
Video models can supply auxiliary supervision, rewards, simulated observations or imagined demonstrations. Here their role is to propose a desired object motion; a separate dynamics model or RL simulator determines robot actions. The video generator is not itself the low-level controller.
III Method
Figure 2 summarizes three stages: generate a video from the instruction and robot-free initial view, recover the object’s calibrated 3D tracks, then optimize or learn controls that follow those tracks.
III-A Problem Formulation
Inputs are an instruction, RGB-D observation and known camera intrinsics/extrinsics. A masked set of object points is tracked through generated frames and lifted into robot coordinates. State contains these points and the robot state; actions may be pushes, end-effector poses or joint controls.
The general objective sums squared corresponding-point error plus a control cost, subject to predicted dynamics and the measured initial state. Video time can be warped or matched to the closest observed shape. Reliable point identity and visibility are essential; the displayed generic cost omits an explicit visibility mask or normalization, although the implementation discussion uses only available tracks.
Equations
III-B Extracting 3D Object Flows from Videos
The initial image excludes the robot, and prompts request human manipulation with a stationary camera. SpatialTrackerV2 supplies monocular video depth. A global scale and offset align its first frame to measured depth; the same calibration is then applied throughout. Camera motion and changing depth bias can violate this approximation.
Grounding DINO localizes the named object, SAM 2 segments it, and CoTracker3 tracks sampled mask pixels with visibility flags. Calibrated depth and projection lift visible tracks into 3D. This reconstructs predicted motion, which may still be physically impossible or geometrically wrong.
Equation
III-C Action Inference with 3D Object Flow
Pushing: a primitive specifies a start point, unit direction and distance. A learned particle model predicts scene-point displacement for sampled contact-making pushes. Random shooting selects the lowest tracking cost and replans after execution, using a nearby future video subgoal.
Hardware: AnyGrasp proposes grasps and HaMer estimates the generated thumb position to select the intended contact. After grasping, points assigned to the grasped part move rigidly with the end effector; other points stay fixed. PyRoki optimizes a smooth reachable path. This approximation is deliberately limited for deformable objects.
Door RL: Soft Actor-Critic learns controls using reward for reaching the object and advancing along its reference flow. The same object target supports Panda, floating-base Spot and a GR1 right arm/hand, but separate policies and action spaces are trained. This is transfer of a reward specification, not one unchanged policy across robots.
Equation
IV Experiments
Experiments test the motion representation, baseline interfaces, reward transfer, video generator and dynamics choice. Sample counts and success definitions differ across the three domains.
IV-A Tasks
Figure 3 shows initial and final configurations. The evaluations use explicit geometric thresholds, which are less demanding than unrestricted household task completion.
IV-A 1 Push-T
Simulated OmniGibson Push-T succeeds when the block reaches the central forward-facing goal within 2 cm translation and 15° rotation. The video prompt includes a goal image for this task.
IV-A 2 Put Bread in Bowl
The hardware bread task uses a fake bread piece and green bowl in randomized positions. Success requires the bread to end inside the bowl.
IV-A 3 Open Oven
The toaster oven is placed on a semicircular arc facing the robot base. Success requires an opening angle of at least 60°.
IV-A 4 Cover Bowl
The scarf begins folded in a randomized pose beside a blue bowl. Success requires covering at least 25% of the bowl’s top, not complete coverage.
IV-A 5 Open Door
Robosuite door opening requires turning the handle and opening the door by at least 17° before timeout. Positions and orientations vary between trials.
IV-B Properties of 3D Object Flow as a Video-Control Interface
Push-T tests ten initial states and ten random-shooting seeds per state, for one hundred trials. Wan2.1 obtains 52 successes. The paper reports six morphed videos but does not clearly specify that video count’s denominator or whether videos are regenerated for every repeated controller trial.
Hardware evaluation uses ten trials per primary task. Six robustness variants add five trials each. Figure 4’s reference bread success is 80%; the long bread variant drops to 40%, and several others fall to 60%. Small samples and absent uncertainty limit the claim of robust generalization.
Figure 5 demonstrates different language tasks in one scene. The four additional in-the-wild cases are qualitative rollouts, without a reported success-rate study. Figure 7’s sixty-trial breakdown records twelve video failures, four extraction failures, four execution failures and forty successes. Six video failures are morphing and six hallucination; the four execution failures occur in cloth covering. These counts describe that reported series, not every baseline or video-model ablation trial.
IV-C How does Dream2Flow perform compared to alternative interfaces?
Table I compares three video-to-control interfaces on the same three hardware tasks. Dream2Flow scores 8/10 bread, 8/10 oven and 3/10 cloth, versus AVDC 7/0/2 and RIGVID 6/6/1. The advantage is consistent across these rows, but cloth remains difficult and ten trials do not establish a broad statistical guarantee.
IV-C 1 AVDC
The AVDC baseline uses dense optical flow and depth to recover a rigid trajectory. This implementation additionally uses video depth and fits transforms relative to the initial frame because the authors found that less noisy. Results therefore concern this adaptation, not an untouched reproduction of the original system.
IV-C 2 RIGVID
The RIGVID adaptation replaces its 6D pose tracker with rigid transforms fit to Dream2Flow’s visible 3D tracks, enabling comparison on deformable objects. Few visible points can destabilize those transforms, while per-point tracking costs permit smooth interpolation through weakly observed intervals.
This comparison tests adapted representations under shared components. The authors attribute failures to tracking, noisy rigid transforms and deformation; there is no separate controlled decomposition of each cause.
IV-D How effective is 3D object flow as a reward for learning sensimotor policies?
Across one hundred random door poses, object-state versus flow rewards yield Panda 99/100 versus 100/100, Spot 99/100 versus 100/100, and GR1 96/100 versus 94/100. Performance is comparable, with a small GR1 decline rather than universal improvement.
Spot can reposition its base, while GR1 pulls with its palm/finger region. These are simulated policies; the GR1 experiment uses its right arm, not complete humanoid locomotion. Reward computation uses simulator door angle to transform object particles, rather than relying on online visual tracking.
| Reward Type | Franka | Spot | GR1 |
|---|---|---|---|
| Object State | 99/100 | 99/100 | 96/100 |
| 3D Object Flow | 100/100 | 100/100 | 94/100 |
IV-E How does the choice of video model affect Dream2Flow in simulation and in real-world tasks?
Wan2.1 reaches 52/100 Push-T and 2/10 oven; Kling 2.1 reaches 31/100 and 4/10; Veo 3 reaches 8/10 oven. Veo 3 was not tested on Push-T because its interface at evaluation time lacked a goal-frame input.
Kling produces more Push-T morphing; Wan produces more camera motion in oven videos, and both can depict articulation about the wrong axis. Generator quality is task-dependent, and these historical results do not compare current model versions.
IV-F How does the choice of dynamics model affect the performance of Dream2Flow?
With the same Push-T flow guidance, a learned pose model trained on the same transitions reaches 12/100, a translation-only heuristic 17/100, and particle dynamics 52/100. This supports per-point prediction for the tested rotational pushes, without proving its superiority for every dynamics family or domain.
| Dynamics Model Type | Success Rate |
|---|---|
| Pose | 12/100 |
| Heuristic | 17/100 |
| Particle | 52/100 |
V Conclusion
Object trajectories provide a useful interface between video intent and robot-specific control, with stronger results than the adapted rigid-motion baselines in this study. The evidence spans several controlled tasks and qualitative cases, rather than unrestricted open-world manipulation.
The main dependencies are feasible video motion, accurate depth/tracks, valid grasp/part selection and a dynamics model suited to the task. Real-world processing takes minutes, and learning or engineering the downstream controller remains necessary.
Acknowledgments
Support includes Stanford HAI, Schmidt Futures Senior Fellows and ONR MURI grants N00014-21-1-2801 and N00014-22-1-2740.
-A Video Generation Prompts
The three exact prompt templates and seven task substitutions are retained below. Real tasks require “by one hand” for thumb-based grasp selection and stationary-camera wording for depth calibration. Robustness prompts replace bread with donut or long bread.
Push-T includes both start and goal images. The simulated door prompt uses self-opening without a hand. All evaluated generators receive the same task wording; Kling additionally uses relevance 0.7 and negative prompt “fast motion, morphing, camera motion”. These designed prompts are part of the method, not arbitrary natural-language task inputs.
Real World Task Language Prompt <TASK> by one hand. The camera holds a still pose, not zooming in or out.
- Put Bread in Bowl: The bread is grabbed and placed into the green bowl
- Open Oven: The toaster oven is opened
- Cover Bowl: The scarf is lifted by a corner and directly dragged over the blue bowl in one smooth motion without flinging the cloth or any dynamic / fast motions
- Pull Out Chair: The chair is pulled out straight from under the table to the right by grabbing the middle
- Open Drawer: The partially opened drawer is opened all the way out
- Sweep Pasta: The brush with a green handle moves left to right to push the pasta into the compost bin
- Recycle Can: The can is grabbed and dropped into the recycling bin
Push-T Task Language Prompt The T-shaped block slides and rotates smoothly to the center of the wooden platform.
Open Door Task Language Prompt The door opens all the way by itself to the right. The camera holds a still pose, not zooming in or out.
-B Particle Dynamics Model
The Push-T model uses a small Point Transformer V3 between input/output MLPs. Each point has fourteen features: XYZ, RGB, surface normal and five push parameters. Four virtual RGB-D cameras form the scene cloud; points outside the platform are removed and one random point is retained per 1.5-cm voxel.
Training uses 500 random-push transitions, with simulator object poses providing point correspondences before/after each push. Visual tracking is suggested as an alternative but not the supervision actually used. This is domain-specific dynamics training even though human task demonstrations are absent.
-C Push-T Planning Details
After each push, online CoTracker3 updates object tracks and measured depth lifts them to 3D. Nearest-neighbor matching associates those tracked particles with the downsampled scene particles used by the dynamics model. Occluded tracks can become visible again once the gripper lifts.
Find the closest reference shape at t*, then target min(t*+20, t_end). This intermediate target helps avoid reaching approximately the right location with the wrong rotation. Random shooting chooses a predicted low-cost contact-making push and repeats until tolerance or a push limit. The candidate count and maximum pushes are not specified here.
Equation
-D Grasp Selection
AnyGrasp proposes up to forty grasps: twenty from the camera-frame cloud and twenty from a virtual overhead view. Choose the earliest frame where a detected thumb lies within 2 cm of a candidate grasp. If none qualifies, use the grasp closest to the inferred movable-part points.
The heuristic exploits a generated human contact location, but hand-estimation errors or missing grasp proposals can select the wrong part. Figure 10 preserves the detected thumb and chosen-grasp examples.
-E Movable Part Flow Filtering for Rigid-Grasp Dynamics Model
Tracks moving at least one pixel per timestep on average are labeled movable. This empirical image-space threshold separates moving oven doors from stationary bodies, but its physical meaning depends on view and scale.
A one-pixel tracking error can lift a point onto background depth. SAM 2 therefore propagates a movable-part mask from positive moving-point and negative stationary-point prompts; tracks outside the mask are invalidated. This filters errors without recovering all occluded motion.
-F Real World Planning Details
Hardware planning optimizes seven joint angles per waypoint, with flow weight 10, reachability weight 100, pose-smoothness weight 1 and manipulability weight 0.01. The retained equations show the objective; exact cost implementations and visibility weighting are not fully specified.
Forward-kinematics poses are fitted by a B-spline and sampled using the source’s at-least-1-cm and/or 20° spacing rule. PyBullet IK converts poses to target joints and Deoxys joint impedance executes them. The described hardware stage follows this planned path; push-style visual replanning is not specified. Reachability costs alone do not establish collision or contact feasibility.
Equations 1–2
| (1) |
| (2) |
-G In-the-Wild Tasks
Figure 11 presents successful examples of four additional tasks. The appendix supplies their success definitions but no repeated-trial success totals.
-G 1 Pull Out Chair
Pull a rolling chair horizontally at least 5 cm from under a table. The short required displacement reflects configurations where further motion is difficult for the fixed-base Franka.
-G 2 Open Drawer
Extend a partially open drawer to at least 90% of its full travel. Opening from a fully closed state is not the evaluated case.
-G 3 Sweep Pasta
Grasp a supported brush and sweep all four dried pasta pieces into a compost bin. This small case illustrates moving multiple separate objects rather than a general granular-material benchmark.
-G 4 Recycle Can
Move an upright aluminum can, initially between recycling and trash bins, into the recycling bin.
-H Open Door Reinforcement Learning Details
Separate SAC policies are trained for Panda, floating-base Spot and a GR1 arm using the same reference object flow. Simulation supplies embodiment-specific interactions and state.
-H 1 Hyperparameters
SAC uses learning rate 3×10⁻⁴, discount 0.99, batch 256, replay capacity one million, target rate 0.005 updated every step, and two hidden layers of 256 units. Training lasts 5,000 iterations, or 10,000 for GR1’s larger action space; episodes last at most 500 steps. Iteration counts are not a reported environment-transition budget.
| Hyperparameter | Value |
| Learning rate | |
| Discount factor () | 0.99 |
| Batch size | 256 |
| Buffer size | |
| Target update rate () | 0.005 |
| Target network update freq | 1 |
| Hidden layers | 2 |
| Hidden units per layer | 256 |
| Training iterations | 5000 |
| Episode horizon | 500 |
-H 2 Reward Functions
The state reward combines a distance-to-handle term capped at 0.25 with a handle-rotation term capped at 0.25; opening the hinge beyond 0.3 radians overrides them with reward 1. The absolute handle angle in the formula makes its negative clipping bound inactive.
The flow reward gives up to 0.75 for the index of the nearest reference shape and up to 0.25 for proximity to the current mean object point. Matching uses unsquared Euclidean point distances in the door frame, unlike the squared generic planning cost. It rewards nearest-frame progress, not explicitly monotonically increasing temporal progress.
For efficiency, the simulator’s door angle transforms initial object particles instead of tracking them visually. This is privileged object-state information in reward computation, so the experiment does not establish equivalent learning from noisy online point tracks. All seven numbered equations and the mean/reward definitions are retained.
Equations 3–7
| (3) |
| (4) |
| (5) |
| (6) |
| (7) |
-I Video Generation Failures
Figure 12 shows bread morphing into crackers and a hallucinated second bowl. The first disrupts point identity; the second can direct a valid-looking robot trajectory to the wrong physical location. A visually plausible video is therefore insufficient for safe or correct execution.
-J Limitations
Rigid-grasp dynamics limits hardware tasks; scaling learned particle dynamics to physical scenes remains future work. Extracting flow takes 3–11 minutes, mainly video generation, so the full pipeline is not an online reactive controller.
One generated viewpoint handles severe occlusion poorly, especially when a hand covers a small object. Better 3D tracking or 4D reconstruction is proposed but not evaluated here. Small hardware sample counts, limited cloth success and qualitative in-the-wild evidence further bound the generalization claim.