Literature
Display
← All papers

Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

Karthik Dharmarajan · Wenlong Huang · Jiajun Wu · Li Fei-Fei · Ruohan Zhang

· ICRA 2026 ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

This edition reviews arXiv v1 dated 31 December 2025, including all ten appendix sections. All five authors are from Stanford; Li Fei-Fei and Ruohan Zhang share advising, and Wenlong Huang is the corresponding author. Figure numbering follows the PDF, correcting the HTML’s restart after its teaser.

Complete original Figure 1.
Fig. 1: Dream2Flow leverages off-the-shelf video generation models to produce videos of the task being performed in the same scene of the robot. Dream2Flow then extracts a 3D object flow from the motion in the video, allowing for downstream planning and execution with a robot across a wide variety of tasks.

At a glance

Dream2Flow generates a task video, reconstructs the manipulated object’s 3D motion and asks a robot to reproduce that motion. Object trajectories bridge human-led video generation and robot-specific action spaces. Pushing uses learned particle dynamics, real grasped manipulation uses rigid-grasp optimization, and simulated door opening uses reinforcement learning.

The video and perception models require no task-specific demonstrations. This does not mean the whole system is untrained: pushing dynamics learns from 500 simulated transitions, and door policies train in simulation. Hardware evidence includes rigid, articulated and deformable tasks, with lower success on cloth and several qualitative additional cases.

I Introduction

Video models often depict human-object interactions better than robot interactions. Directly copying a generated hand trajectory would impose the wrong embodiment, while the object’s desired state change can be shared across robots. Dream2Flow therefore extracts object motion and lets a controller solve for the required robot actions.

The interface supports motion planning and reward-based policy training, but each domain still needs a suitable dynamics model or simulator, calibrated observation and robot-specific controls. Hand reconstruction is also used to select grasps, so the pipeline retains some human-interaction information beyond object flow.

The author project page provides videos and interactive flow visualizations. The manuscript’s zero-shot language refers to using pretrained video/perception models and avoiding task-specific human demonstrations; it should not obscure the learned dynamics, task-specific RL or designed prompts used below.

II Related Works

The paper places object flow between high-level intent and embodiment-specific control, drawing on task specification, dense tracking and video prediction.

II-A Task Specification in Manipulation

Task specifications range from symbolic goals and costs to language policies, goal images, forces, object descriptors and foundation-model-generated programs or spatial maps. Dream2Flow expresses the desired change as a time-indexed object trajectory, leaving the controller to realize it.

II-B 2D/3D Flow in Robotics

Optical flow, point tracks and scene flow can condition policies or provide motion targets. Dream2Flow follows the object-centric branch: reconstruct a generated trajectory independently of the actor, then track it under robot constraints. It builds on actionless-video correspondences such as AVDC and differs from fitting one rigid pose at every frame.

II-C Video Models for Robotics

Video models can supply auxiliary supervision, rewards, simulated observations or imagined demonstrations. Here their role is to propose a desired object motion; a separate dynamics model or RL simulator determines robot actions. The video generator is not itself the low-level controller.

III Method

Figure 2 summarizes three stages: generate a video from the instruction and robot-free initial view, recover the object’s calibrated 3D tracks, then optimize or learn controls that follow those tracks.

Complete original Figure 2.
Fig. 2: An overview of Dream2Flow. Given a task instruction and an initial RGB-D observation, an image-to-video model synthesizes video frames conditioned on the instruction. We additionally obtain object masks, video depth, and point tracking from vision foundation models, which are used to reconstruct 3D object flow. Finally, a robot policy generates executable actions that track the 3D object flow using trajectory optimization or reinforcement learning.

III-A Problem Formulation

Inputs are an instruction, RGB-D observation and known camera intrinsics/extrinsics. A masked set of object points is tracked through generated frames and lifted into robot coordinates. State contains these points and the robot state; actions may be pushes, end-effector poses or joint controls.

The general objective sums squared corresponding-point error plus a control cost, subject to predicted dynamics and the measured initial state. Video time can be warped or matched to the closest observed shape. Reliable point identity and visibility are essential; the displayed generic cost omits an explicit visibility mask or normalization, although the implementation discussion uses only available tracks.

Equations
min{ut∈𝒰}\displaystyle\min_{\{u_{t}\in\mathcal{U}\}} ∑t=0H−1λtask​(x^tobj,P~t)+λcontrol​(x^t,ut)\displaystyle\sum_{t=0}^{H-1}\lambda_{\text{task}}\big(\hat{x}^{\text{obj}}_{t},\tilde{P}_{t}\big)+\lambda_{\text{control}}(\hat{x}_{t},u_{t})
s.t.\displaystyle\text{s.t.} x^t+1=f(x^t,ut),x^0=x0,\displaystyle\hat{x}_{t+1}=f(\hat{x}_{t},u_{t}),\quad\hat{x}_{0}=x_{0},
λtask​(x^tobj,P~t)=∑i=1n‖x^tobj​[i]−P~t​[i]‖22,\displaystyle\lambda_{\text{task}}\big(\hat{x}^{\text{obj}}_{t},\tilde{P}_{t}\big)=\sum_{i=1}^{n}\big\|\hat{x}^{\text{obj}}_{t}[i]-\tilde{P}_{t}[i]\big\|_{2}^{2},

(I0∈ℝH×W×3,D0∈ℝH×W)(I_{0}\in\mathbb{R}^{H\!\times\!W\!\times\!3},\,D_{0}\in\mathbb{R}^{H\!\times\!W})

u0:H−1∈𝒰Hu_{0:H-1}\in\mathcal{U}^{H}

P1:T∈ℝT×n×3P_{1:T}\in\mathbb{R}^{T\times n\times 3}

xt=(xtobj,rt)x_{t}=(x_{t}^{\text{obj}},r_{t})

xtobj∈ℝn×3x_{t}^{\text{obj}}\in\mathbb{R}^{n\times 3}

x^t+1=f⁡(x^t,ut)\hat{x}_{t+1}=f(\hat{x}_{t},u_{t})

x^0=x0\hat{x}_{0}=x_{0}

P~t∈ℝn×3\tilde{P}_{t}\in\mathbb{R}^{n\times 3}

III-B Extracting 3D Object Flows from Videos

The initial image excludes the robot, and prompts request human manipulation with a stationary camera. SpatialTrackerV2 supplies monocular video depth. A global scale and offset align its first frame to measured depth; the same calibration is then applied throughout. Camera motion and changing depth bias can violate this approximation.

Grounding DINO localizes the named object, SAM 2 segments it, and CoTracker3 tracks sampled mask pixels with visibility flags. Calibrated depth and projection lift visible tracks into 3D. This reconstructs predicted motion, which may still be physically impossible or geometrically wrong.

Equation

Zt=s⋆​Z~t+b⋆Z_{t}=s^{\star}\tilde{Z}_{t}+b^{\star}

V∈{0,1}T×nV\!\in\!\{0,1\}^{T\times n}

III-C Action Inference with 3D Object Flow

Pushing: a primitive specifies a start point, unit direction and distance. A learned particle model predicts scene-point displacement for sampled contact-making pushes. Random shooting selects the lowest tracking cost and replans after execution, using a nearby future video subgoal.

Hardware: AnyGrasp proposes grasps and HaMer estimates the generated thumb position to select the intended contact. After grasping, points assigned to the grasped part move rigidly with the end effector; other points stay fixed. PyRoki optimizes a smooth reachable path. This approximation is deliberately limited for deformable objects.

Door RL: Soft Actor-Critic learns controls using reward for reaching the object and advancing along its reference flow. The same object target supports Panda, floating-base Spot and a GR1 right arm/hand, but separate policies and action spaces are trained. This is transfer of a reward specification, not one unchanged policy across robots.

Equation

(cx,cy)(c_{x},c_{y})

(Δ​cx,Δ​cy)(\Delta c_{x},\Delta c_{y})

x~t∈ℝN×14\tilde{x}_{t}\in\mathbb{R}^{N\times 14}

IV Experiments

Experiments test the motion representation, baseline interfaces, reward transfer, video generator and dynamics choice. Sample counts and success definitions differ across the three domains.

IV-A Tasks

Figure 3 shows initial and final configurations. The evaluations use explicit geometric thresholds, which are less demanding than unrestricted household task completion.

Complete original Figure 3.
Fig. 3: Evaluation Tasks. (a) The initial states of each task used in the evaluation trials. (b) For one of the initial states, the corresponding final state after the robot performs the task. For Push-T only, the desired final state is the same for all initial states.

IV-A 1 Push-T

Simulated OmniGibson Push-T succeeds when the block reaches the central forward-facing goal within 2 cm translation and 15° rotation. The video prompt includes a goal image for this task.

IV-A 2 Put Bread in Bowl

The hardware bread task uses a fake bread piece and green bowl in randomized positions. Success requires the bread to end inside the bowl.

IV-A 3 Open Oven

The toaster oven is placed on a semicircular arc facing the robot base. Success requires an opening angle of at least 60°.

IV-A 4 Cover Bowl

The scarf begins folded in a randomized pose beside a blue bowl. Success requires covering at least 25% of the bowl’s top, not complete coverage.

IV-A 5 Open Door

Robosuite door opening requires turning the handle and opening the door by at least 17° before timeout. Positions and orientations vary between trials.

IV-B Properties of 3D Object Flow as a Video-Control Interface

Push-T tests ten initial states and ten random-shooting seeds per state, for one hundred trials. Wan2.1 obtains 52 successes. The paper reports six morphed videos but does not clearly specify that video count’s denominator or whether videos are regenerated for every repeated controller trial.

Hardware evaluation uses ten trials per primary task. Six robustness variants add five trials each. Figure 4’s reference bread success is 80%; the long bread variant drops to 40%, and several others fall to 60%. Small samples and absent uncertainty limit the claim of robust generalization.

Figure 5 demonstrates different language tasks in one scene. The four additional in-the-wild cases are qualitative rollouts, without a reported success-rate study. Figure 7’s sixty-trial breakdown records twelve video failures, four extraction failures, four execution failures and forty successes. Six video failures are morphing and six hallucination; the four execution failures occur in cloth covering. These counts describe that reported series, not every baseline or video-model ablation trial.

Complete original Figure 4.
Fig. 4: Robustness evaluations. Relative performance across instance, background, and task variations, showing Dream2Flow remains robust under various different settings.
Complete original Figure 5.
Fig. 5: Multiple tasks in the same scene. With different language goals, Dream2Flow adapts object-flow targets to produce distinct behaviors in the same environment.

IV-C How does Dream2Flow perform compared to alternative interfaces?

Table I compares three video-to-control interfaces on the same three hardware tasks. Dream2Flow scores 8/10 bread, 8/10 oven and 3/10 cloth, versus AVDC 7/0/2 and RIGVID 6/6/1. The advantage is consistent across these rows, but cloth remains difficult and ten trials do not establish a broad statistical guarantee.

Task AVDC RIGVID Dream2Flow
Bread in Bowl 7/10 6/10 8/10
Open Oven 0/10 6/10 8/10
Cover Bowl 2/10 1/10 3/10
TABLE I: Comparisons of intermediate representations on real robot. Dream2Flow outperforms AVDC and RIGVID across three tasks by following 3D object flow rather than rigid transforms alone.

IV-C 1 AVDC

The AVDC baseline uses dense optical flow and depth to recover a rigid trajectory. This implementation additionally uses video depth and fits transforms relative to the initial frame because the authors found that less noisy. Results therefore concern this adaptation, not an untouched reproduction of the original system.

IV-C 2 RIGVID

The RIGVID adaptation replaces its 6D pose tracker with rigid transforms fit to Dream2Flow’s visible 3D tracks, enabling comparison on deformable objects. Few visible points can destabilize those transforms, while per-point tracking costs permit smooth interpolation through weakly observed intervals.

This comparison tests adapted representations under shared components. The authors attribute failures to tracking, noisy rigid transforms and deformation; there is no separate controlled decomposition of each cause.

IV-D How effective is 3D object flow as a reward for learning sensimotor policies?

Across one hundred random door poses, object-state versus flow rewards yield Panda 99/100 versus 100/100, Spot 99/100 versus 100/100, and GR1 96/100 versus 94/100. Performance is comparable, with a small GR1 decline rather than universal improvement.

Spot can reposition its base, while GR1 pulls with its palm/finger region. These are simulated policies; the GR1 experiment uses its right arm, not complete humanoid locomotion. Reward computation uses simulator door angle to transform object particles, rather than relying on online visual tracking.

Reward Type Franka Spot GR1
Object State 99/100 99/100 96/100
3D Object Flow 100/100 100/100 94/100
TABLE II: Comparison of policies trained using different rewards. The policies trained using the 3D object flow reward perform comparably to those trained with the object state reward across different embodiments.
Complete original Figure 6.
Fig. 6: Rollouts from policies trained using 3D object flow as a reward. Different embodiments such as the (a) Panda, (b) Spot, or (c) GR1 use different strategies to open the door. The Spot is able move its base for better reachability while the GR1 uses the area between its fingers and palm to pull for better stability.

IV-E How does the choice of video model affect Dream2Flow in simulation and in real-world tasks?

Wan2.1 reaches 52/100 Push-T and 2/10 oven; Kling 2.1 reaches 31/100 and 4/10; Veo 3 reaches 8/10 oven. Veo 3 was not tested on Push-T because its interface at evaluation time lacked a goal-frame input.

Kling produces more Push-T morphing; Wan produces more camera motion in oven videos, and both can depict articulation about the wrong axis. Generator quality is task-dependent, and these historical results do not compare current model versions.

Video Generation Model Push-T Open Oven
Wan2.1 [89] 52/100 2/10
Kling 2.1 31/100 4/10
Veo 3 [90] - 8/10
TABLE III: Effect of video generator. Veo 3 excels on real-world domains such as “Open Oven”, while Wan 2.1 performs better on simulated domains such as “Push-T”.

IV-F How does the choice of dynamics model affect the performance of Dream2Flow?

With the same Push-T flow guidance, a learned pose model trained on the same transitions reaches 12/100, a translation-only heuristic 17/100, and particle dynamics 52/100. This supports per-point prediction for the tested rotational pushes, without proving its superiority for every dynamics family or domain.

Dynamics Model Type Success Rate
Pose 12/100
Heuristic 17/100
Particle 52/100
TABLE IV: Dynamics model ablation. Particle dynamics substantially outperform pose and heuristic models, highlighting the importance of per-point predictions.
Complete original Figure 7.
Fig. 7: Failure breakdown on real-robot experiments. Common causes include video artifacts (object morphing, object hallucination), tracking errors, and grasp selection mismatches.

V Conclusion

Object trajectories provide a useful interface between video intent and robot-specific control, with stronger results than the adapted rigid-motion baselines in this study. The evidence spans several controlled tasks and qualitative cases, rather than unrestricted open-world manipulation.

The main dependencies are feasible video motion, accurate depth/tracks, valid grasp/part selection and a dynamics model suited to the task. Real-world processing takes minutes, and learning or engineering the downstream controller remains necessary.

Acknowledgments

Support includes Stanford HAI, Schmidt Futures Senior Fellows and ONR MURI grants N00014-21-1-2801 and N00014-22-1-2740.

-A Video Generation Prompts

The three exact prompt templates and seven task substitutions are retained below. Real tasks require “by one hand” for thumb-based grasp selection and stationary-camera wording for depth calibration. Robustness prompts replace bread with donut or long bread.

Push-T includes both start and goal images. The simulated door prompt uses self-opening without a hand. All evaluated generators receive the same task wording; Kling additionally uses relevance 0.7 and negative prompt “fast motion, morphing, camera motion”. These designed prompts are part of the method, not arbitrary natural-language task inputs.

Real World Task Language Prompt

<TASK> by one hand. The camera holds a still pose, not zooming in or out.
  • Put Bread in Bowl: The bread is grabbed and placed into the green bowl
  • Open Oven: The toaster oven is opened
  • Cover Bowl: The scarf is lifted by a corner and directly dragged over the blue bowl in one smooth motion without flinging the cloth or any dynamic / fast motions
  • Pull Out Chair: The chair is pulled out straight from under the table to the right by grabbing the middle
  • Open Drawer: The partially opened drawer is opened all the way out
  • Sweep Pasta: The brush with a green handle moves left to right to push the pasta into the compost bin
  • Recycle Can: The can is grabbed and dropped into the recycling bin
Push-T Task Language Prompt

The T-shaped block slides and rotates smoothly to the center of the wooden platform.
Complete original Figure 8.
Fig. 8: Example Push-T image prompt. To generate sufficiently accurate motions for the Push-T task, the visual component of the prompt includes both a start and end frame.
Open Door Task Language Prompt

The door opens all the way by itself to the right. The camera holds a still pose, not zooming in or out.

-B Particle Dynamics Model

The Push-T model uses a small Point Transformer V3 between input/output MLPs. Each point has fourteen features: XYZ, RGB, surface normal and five push parameters. Four virtual RGB-D cameras form the scene cloud; points outside the platform are removed and one random point is retained per 1.5-cm voxel.

Training uses 500 random-push transitions, with simulator object poses providing point correspondences before/after each push. Visual tracking is suggested as an alternative but not the supervision actually used. This is domain-specific dynamics training even though human task demonstrations are absent.

Complete original Figure 9.
Fig. 9: Particle dynamics model. The particle dynamics model primarily composed of a Point Transformer V3 backbone used in the Push-T task takes as input a set of feature-augmented particles and outputs delta position predictions for all particles in the scene.

-C Push-T Planning Details

After each push, online CoTracker3 updates object tracks and measured depth lifts them to 3D. Nearest-neighbor matching associates those tracked particles with the downsampled scene particles used by the dynamics model. Occluded tracks can become visible again once the gripper lifts.

Find the closest reference shape at t*, then target min(t*+20, t_end). This intermediate target helps avoid reaching approximately the right location with the wrong rotation. Random shooting chooses a predicted low-cost contact-making push and repeats until tolerance or a push limit. The candidate count and maximum pushes are not specified here.

Equation

x^t+1=x^t+Δ​x^t+1\hat{x}_{t+1}=\hat{x}_{t}+\Delta\hat{x}_{t+1}

min⁡(t⋆+L,tend)\min(t^{\star}+L,t_{\text{end}})

L=20L=20

-D Grasp Selection

AnyGrasp proposes up to forty grasps: twenty from the camera-frame cloud and twenty from a virtual overhead view. Choose the earliest frame where a detected thumb lies within 2 cm of a candidate grasp. If none qualifies, use the grasp closest to the inferred movable-part points.

The heuristic exploits a generated human contact location, but hand-estimation errors or missing grasp proposals can select the wrong part. Figure 10 preserves the detected thumb and chosen-grasp examples.

Complete original Figure 10.
Fig. 10: Grasp selection with thumb position. Dream2Flow selects a grasp closest to the position of the thumb in the video, if such a grasp exists within 2cm. The red sphere indicates the thumb position predicted by HaMer, with the red grasp being the selected one.

-E Movable Part Flow Filtering for Rigid-Grasp Dynamics Model

Tracks moving at least one pixel per timestep on average are labeled movable. This empirical image-space threshold separates moving oven doors from stationary bodies, but its physical meaning depends on view and scale.

A one-pixel tracking error can lift a point onto background depth. SAM 2 therefore propagates a movable-part mask from positive moving-point and negative stationary-point prompts; tracks outside the mask are invalidated. This filters errors without recovering all occluded motion.

Complete original Figure 11.
Fig. 11: In-the-Wild Task Rollouts. The Franka successfully pulls out a chair, opens a partially opened drawer, sweeps pasta into the compost bin, and recycles an aluminum can.

-F Real World Planning Details

Hardware planning optimizes seven joint angles per waypoint, with flow weight 10, reachability weight 100, pose-smoothness weight 1 and manipulability weight 0.01. The retained equations show the objective; exact cost implementations and visibility weighting are not fully specified.

Forward-kinematics poses are fitted by a B-spline and sampled using the source’s at-least-1-cm and/or 20° spacing rule. PyBullet IK converts poses to target joints and Deoxys joint impedance executes them. The described hardware stage follows this planned path; push-style visual replanning is not specified. Reachability costs alone do not establish collision or contact feasibility.

Equations 1–2
∑t=0H−1λtask​(x^tobj,P~t)+λcontrol​(x^t,ut)\sum_{t=0}^{H-1}\lambda_{\text{task}}\big(\hat{x}^{\text{obj}}_{t},\tilde{P}_{t}\big)+\lambda_{\text{control}}(\hat{x}_{t},u_{t}) (1)
λcontrol​(x^t,ut)=wr​𝒞r​(𝐪t)+ws​𝒞s​(𝐪t,𝐪t−1)+wm​𝒞m​(𝐪t)\lambda_{\text{control}}(\hat{x}_{t},u_{t})=w_{r}\mathcal{C}_{r}(\mathbf{q}_{t})+w_{s}\mathcal{C}_{s}(\mathbf{q}_{t},\mathbf{q}_{t-1})+w_{m}\mathcal{C}_{m}(\mathbf{q}_{t}) (2)

𝐪∈ℝ7\mathbf{q}\in\mathbb{R}^{7}

wr=100w_{r}=100

ws=1w_{s}=1

wm=0.01w_{m}=0.01

wf=10w_{f}=10

-G In-the-Wild Tasks

Figure 11 presents successful examples of four additional tasks. The appendix supplies their success definitions but no repeated-trial success totals.

-G 1 Pull Out Chair

Pull a rolling chair horizontally at least 5 cm from under a table. The short required displacement reflects configurations where further motion is difficult for the fixed-base Franka.

-G 2 Open Drawer

Extend a partially open drawer to at least 90% of its full travel. Opening from a fully closed state is not the evaluated case.

-G 3 Sweep Pasta

Grasp a supported brush and sweep all four dried pasta pieces into a compost bin. This small case illustrates moving multiple separate objects rather than a general granular-material benchmark.

-G 4 Recycle Can

Move an upright aluminum can, initially between recycling and trash bins, into the recycling bin.

-H Open Door Reinforcement Learning Details

Separate SAC policies are trained for Panda, floating-base Spot and a GR1 arm using the same reference object flow. Simulation supplies embodiment-specific interactions and state.

-H 1 Hyperparameters

SAC uses learning rate 3×10⁻⁴, discount 0.99, batch 256, replay capacity one million, target rate 0.005 updated every step, and two hidden layers of 256 units. Training lasts 5,000 iterations, or 10,000 for GR1’s larger action space; episodes last at most 500 steps. Iteration counts are not a reported environment-transition budget.

Hyperparameter Value
Learning rate 3×10−43\times 10^{-4}
Discount factor (γ\gamma) 0.99
Batch size 256
Buffer size 10610^{6}
Target update rate (τ\tau) 0.005
Target network update freq 1
Hidden layers 2
Hidden units per layer 256
Training iterations 5000
Episode horizon 500
TABLE V: SAC Hyperparameters for Open Door Task.

-H 2 Reward Functions

The state reward combines a distance-to-handle term capped at 0.25 with a handle-rotation term capped at 0.25; opening the hinge beyond 0.3 radians overrides them with reward 1. The absolute handle angle in the formula makes its negative clipping bound inactive.

The flow reward gives up to 0.75 for the index of the nearest reference shape and up to 0.25 for proximity to the current mean object point. Matching uses unsquared Euclidean point distances in the door frame, unlike the squared generic planning cost. It rewards nearest-frame progress, not explicitly monotonically increasing temporal progress.

For efficiency, the simulator’s door angle transforms initial object particles instead of tracking them visually. This is privileged object-state information in reward computation, so the experiment does not establish equivalent learning from noisy online point tracks. All seven numbered equations and the mean/reward definitions are retained.

Equations 3–7
rreach=0.25⋅(1−tanh⁡(10⋅dgripper, handle))r_{\text{reach}}=0.25\cdot(1-\tanh(10\cdot d_{\text{gripper, handle}})) (3)
rrot=clip​(0.25⋅|θhandle|0.5​π,−0.25,0.25)r_{\text{rot}}=\text{clip}\left(0.25\cdot\frac{|\theta_{\text{handle}}|}{0.5\pi},-0.25,0.25\right) (4)
rparticle=0.75⋅t⋆tendr_{\text{particle}}=0.75\cdot\frac{t^{\star}}{t_{\text{end}}} (5)
t⋆=argmint∈{1​…​tend}1n​∑i=1n‖x^tobj​[i]−Pt​[i]‖2t^{\star}=\argmin_{t\in\{1\dots t_{\text{end}}\}}\frac{1}{n}\sum_{i=1}^{n}\|\hat{x}_{t}^{\text{obj}}[i]-P_{t}[i]\|_{2} (6)
ree=0.25⋅(1−tanh⁡(10⋅‖e​edoor−x¯‖2))r_{\text{ee}}=0.25\cdot(1-\tanh(10\cdot\|ee_{\text{door}}-\bar{x}\|_{2})) (7)

rreach+rrotr_{\text{reach}}+r_{\text{rot}}

θhinge>0.3\theta_{\text{hinge}}>0.3

x¯=1n​∑i=1nx^tobj​[i]\bar{x}=\frac{1}{n}\sum_{i=1}^{n}\hat{x}_{t}^{\text{obj}}[i]

Rflow=rparticle+reeR_{\text{flow}}=r_{\text{particle}}+r_{\text{ee}}

-I Video Generation Failures

Figure 12 shows bread morphing into crackers and a hallucinated second bowl. The first disrupts point identity; the second can direct a valid-looking robot trajectory to the wrong physical location. A visually plausible video is therefore insufficient for safe or correct execution.

Complete original Figure 12.
Fig. 12: Video Generation Failure Examples. (a) The bread undergoes object morphing, where it turns into a stack of crackers, causing the downstream tracking to fail. (b) Another green bowl appears due to model hallucination, causing a downstream execution failure where the bread is dropped onto the surface of the workspace instead of the original green bowl.

-J Limitations

Rigid-grasp dynamics limits hardware tasks; scaling learned particle dynamics to physical scenes remains future work. Extracting flow takes 3–11 minutes, mainly video generation, so the full pipeline is not an online reactive controller.

One generated viewpoint handles severe occlusion poorly, especially when a hand covers a small object. Better 3D tracking or 4D reconstruction is proposed but not evaluated here. Small hardware sample counts, limited cloth success and qualitative in-the-wild evidence further bound the generalization claim.

Concept demonstration

Open image ↗