Overview
DreamCan turns robot videos into sparse visual plans by selecting frames that combine task progress, dataset-supported reachability and temporal spacing. A fine-tuned 5B video model generates those plans; a separate 0.1B controller executes them. This edition restores the missing opening overview and retains all ten original figures, five tables, six equations and 35 references. Read the planning-only evaluations separately from closed-loop robot success, and the small controller size separately from the much larger pretrained planner.
At a glance
The planner learns from action-free videos annotated automatically with useful, reachable-looking subgoals. Dynamic programming chooses training frames, then a goal-conditioned controller follows the generated image sequence with periodic replanning. DreamCan reaches 91.2% reported LIBERO-Long success and 86% PushT success, while benefiting from more suboptimal videos and test-time search. Physical success is lower—40% box folding, 65% unseen-combination packing and 55% line arrangement—but exceeds the tested alternatives. Reachability is an offline-data proxy, not a guarantee that a particular controller can execute every generated plan.
I Introduction
Video goals can express contact geometry and intermediate configurations that are awkward to specify in words. Dense futures may overconstrain execution, while uniformly sampled frames can miss important transitions or preserve detours. Inspired by SayCan’s separation of relevance and affordance, DreamCan scores task utility and transition reachability without a predefined skill vocabulary. Selected video frames supervise the planner, which communicates with an independently trained controller through images alone. Action-free planner learning still depends on video instructions, pretrained visual/progress models and a separately trained action-generating policy; it does not make the complete system annotation- or action-data-free.
II Related Work
Related work covers how to discover sparse task states and how learned world models support control. DreamCan’s main distinction is its supervision for choosing the temporal granularity of explicit image goals.
II-A Keystate Planning for Robot Manipulation Tasks
Manual keyframes and embodiment-specific cues can provide useful subgoals but are costly or unavailable in video-only datasets. Semantic milestones from vision-language models need not be executable. Compared with KeyIn, SuSIE and latent backward planning, DreamCan explicitly combines estimated transition ease with task progress and spacing when choosing training frames.
II-B World Models for Robot Manipulation Tasks
Latent dynamics models support policy optimization or planning, while video and world-action models predict observations or joint observation/action futures. DreamCan uses a video model only for sparse explicit image goals. The planner and controller share neither gradients nor hidden activations, although both rely on compatible goal-image semantics. This modular interface permits separate upgrades; it does not remove the need to match generated goals to controller capability.
III Preliminary and Motivation
The method is a hierarchy from language and current images to visual subgoals, then from observations and those goals to low-level action chunks. The motivation is to make each visual target informative without prescribing every intermediate motion.
III-A Preliminaries
The planner generates a short ordered sequence of goal images from the current observation and instruction. A goal-conditioned policy receives multiple future goals and produces an action chunk; subsequent observations allow periodic replanning. Planner training uses videos, while the controller is trained independently with action supervision. The image sequence is their sole interface, so planner improvement can be evaluated with a fixed controller.
Equations 1–2
| (1) |
| (2) |
III-B Motivation: What Makes a Good Visual Plan?
PushT illustrates the tradeoff. Language may omit the exact contact point or relative geometry; a dense sequence can drift from actual execution; uniform frame selection can place a difficult transition beside an almost redundant one. Useful subgoals should be reachable from preceding states and advance the task. Neither closeness alone nor apparent final-goal utility alone suffices.
IV Method
Three steps create the hierarchy: estimate reachability and progress from videos, select sparse training targets by dynamic programming, then fine-tune a pretrained video generator and pair it with a goal-conditioned controller.
IV-A Dataset-Supported Reachability
Reachability estimates discounted first-arrival time using a goal-conditioned value function learned from dataset transitions. It is conditioned on data coverage and the assumption that the eventual controller can perform behavior similar to that recorded.
Formulation
With reward −1 before reaching the target and zero at termination, normalized value is one plus (one minus discount) times the value. Equivalently, it is the expected discount raised to the first-hitting time, with zero contribution when the goal is never reached. A higher score combines faster arrival and more reliable arrival; it is not a fixed-horizon success probability.
Equation 3
| (3) | ||||
Learning from videos
Frozen RADIO features encode observations and goals. A temporal Transformer combines observation history, goal and language to learn action-free goal-conditioned values using an upper expectile of 0.75, discount 0.95 and target-update rate 0.005. The goal-state boundary is zero value. Half the sampled goals come from geometrically offset future frames and half from other trajectories; cross-trajectory goals are not automatically labeled impossible. The upper expectile emphasizes better dataset transitions, following ideas from implicit Q-learning and HIQL. This remains dataset-supported reachability rather than an online measurement of the current controller.
IV-B Task Utility Value Function
Most tasks adapt RoboMeter with LoRA. Spatial tasks supplement progress estimates with vision-language outputs and SAM masks. For successful demonstrations, smoothing, start/completion normalization and a running maximum produce a [0,1] progress score. The running maximum records the best progress reached so far, so it can remain high during a later regression; it is an annotation signal rather than the instantaneous physical success state.
IV-C Subgoal Selection by Dynamic Programming
Selection minimizes summed negative-log reachability and a temporal-gap penalty, minus utility at the final selected frame. Intermediate utility differences telescope, so utility mainly chooses a useful endpoint while reachability places intermediate steps. A minimum gap constrains increasing frame indices. Backward dynamic programming stores the best next index for each remaining budget and runs in O(KL²) after scoring; backtracking recovers K selected frames and appending the demonstration’s terminal frame yields K+1 targets. The original permits the final selected index to equal the terminal index, so the written rule can duplicate that endpoint; no undocumented deduplication is assumed. Figure 3 illustrates four selected goals and states twelve in experiments, while Figure 6’s box-folding row labels a goal g15; that source horizon discrepancy remains visible.
Equations 4–5
| (4) |
| (5) | ||||
IV-D Subgoal Generation with a Pretrained Video Model
The high-level generator is pretrained Wan2.2-TI2V-5B fine-tuned to output selected subgoal sequences. Its 5B scale matters when interpreting comparisons advertised in terms of the separate 0.1B controller.
Sparse latent construction
Wan’s causal video autoencoder compresses four frames per temporal slot. Each selected goal image is repeated four times after the initial observation so different goals occupy distinct latent slots instead of being mixed by temporal compression. Equation 6 retains the exact input/latent construction.
Equation 6
| (6) |
Closed-loop planner fine-tuning
LoRA adapts the diffusion Transformer with flow matching while freezing the video autoencoder and text encoder. Training starts from observations sampled throughout a trajectory and conditions on subsequent selected goals, exposing the planner to intermediate states needed for closed-loop replanning. Instruction dropout is 0.5 in domains with limited linguistic variety.
Prefix-conditioned generation for test-time search.
Thirty percent of training examples condition only on the observation. Otherwise a random clean prefix of one through K−4 goal latents is fixed, and denoising loss applies only to the remaining suffix. Prefix positions receive diffusion time zero. At test time, this allows search to commit chosen goals and regenerate the tail. The original alternates between K selected goals and K+1 terminal-appended targets; detailed configuration is needed to resolve the exact sequence length.
IV-E Goal-Conditioned Low-Level Policy
The 0.1B flow-matching Transformer receives current images, proprioception, instruction and several future subgoals. Current and goal images use frozen RADIO features; compressed goal tokens combine with state and language tokens as an action-generator prefix. The controller itself is not pretrained in the reported comparison, but its visual representation and the 5B planner are pretrained. Low policy parameter count therefore does not describe the total deployed system.
V Experiments
Evaluation separates visual-plan validity, end-to-end task success, benefits from extra suboptimal video, search/steering, local subgoal reachability and physical manipulation. These outcomes use different protocols and should not be conflated.
V-A Long-horizon Task Planning Results
Language-Table training uses 1,424 line, 1,303 region and 1,378 shape videos, resized to 320×240 with fifteen subgoals; the text specifies 100 test videos for each task. Plan success requires the instructed final arrangement, sustained object integrity and plausible tracked transitions. DreamCan scores 86/100/52% versus VLP’s 56/78/15% for line/region/shape; the authors reproduce VLP’s shape result. On line planning, dense and uniform visual alternatives score 71% and 69%. Table I also improves all reported image/video metrics, including FVD 154 versus 233/258 on Language Table. These are generated-plan evaluations, not physical execution success. Table III separately describes 50 evaluation episodes for its comparison, so its caption and the 100-video planning protocol should not be merged into a single assumed denominator.
| Method | PushT | LIBERO-Long | Language Table |
| E2E Policy | 58.0% | 79.2% | – |
| Language Planning | 62.0% | 75.6% | 56.0% |
| Dense Visual Planning | 70.0% | 84.0% | 71.0% |
| Uniform Subgoal Planning | 72.0% | 82.4% | 69.0% |
| VLA-OS [31] (0.9B, non-pretrained) | 72.0% | 76.0% | – |
| [32] (3.3B, pretrained) | 76.0% | 85.2% | – |
| SEER [33] (0.3B, pretrained) | 68.0% | 87.7% | – |
| [34] (3.3B, pretrained) | 86.0% | 92.4% | – |
| Lingbot-VA [35] (5.3B, pretrained) | 89.0% | 98.5% | – |
| Ours (0.1B policy, non-pretrained) | 86.0% | 91.2% | 86.0% |
V-B Hierarchical Task Execution Results
PushT uses 206 demonstrations and LIBERO-Long 379, with a shared low-level architecture across representation baselines. DreamCan reaches 86.0% and 91.2%, compared with end-to-end 58.0/79.2%, language 62.0/75.6%, dense video 70.0/84.0% and uniform goals 72.0/82.4%. The exact columns qualify the narrative: language is below end-to-end only on LIBERO here, and uniform goals do not beat dense video there. DreamCan exceeds the reported VLA-OS, π0 and SEER results, but LIBERO remains below π0.5’s 92.4% and Lingbot-VA’s 98.5%; PushT ties π0.5 at 86% and trails Lingbot-VA’s 89%. Large-model PushT results are rerun by the authors, whereas LIBERO figures come from the original papers. The 0.1B controller comparison excludes DreamCan’s 5B pretrained planner.
V-C Scaling with Sub-Optimal Dataset
The Square study fixes a controller trained on 100 proficient demonstrations and trains planners with nested subsets of 5 through 200 suboptimal demonstrations. Increasing video data raises DreamCan success from roughly the low fifties to the high sixties in Figure 5; uniform and dense alternatives improve less consistently. This isolates adding planner video data without retraining the controller, but it is one task and one finite dataset rather than evidence of unrestricted large-scale scaling.
V-D Test-Time Search and Language Steering
Whole-plan best-of-N chooses among independent candidates. Anchored search instead commits the best prefix, pins those latents and regenerates the suffix, using the same scripted detection-based progress score used in training. On line planning, anchored search continues improving with more generator calls, reaching approximately 0.9 progress; each point uses 100 test episodes. Dense generation uses the same scorer and matched compute but remains worse. Progress optimized by this scorer is not independently measured execution success. Qualitative language steering changes intended line arrangements, and a vision-language-generated scoring function selects a packing arrangement from several rollouts.
V-E Reachability Value Function Evaluation
LIBERO segment tests reset and replay the simulator to each ground-truth starting subgoal, then check whether task-relevant objects reach the next target within 5 cm. DreamCan averages 0.98 segment success versus 0.88 for uniform goals; resetting means this is not an uninterrupted long-horizon success rate. Selection ablations give spacing-only 81.8%, reachability+spacing 74.2%, utility+spacing 78.0%, reachability+utility 83.8%, and all three 91.2%. This supports the full recipe under these settings, not universal necessity of every term. Square analyses compare learned arrival time with dataset shortest paths rather than elapsed demonstration time. In one dropped-and-regrasped example, reachability recovers to 0.80 when the state returns within 1.4 cm of its starting configuration, versus 0.004 for pure temporal decay.
V-F Real-World Evaluation
A dual-arm YAM setup uses one D435 overhead and two D405 wrist cameras. Training provides 100 box-folding demonstrations, 550 packing demonstrations over 110 object combinations, and 90 line demonstrations over three target shapes. Ninety-five of 205 feasible packing combinations are held out; Table V evaluates twenty episodes per task, not all 95 combinations. DreamCan succeeds in 8/20 folding, 13/20 unseen-combination packing and 11/20 line trials. Uniform goals give 2/20, 4/20 and 5/20; dense planning 0/20, 1/20 and 0/20; end-to-end 1/20, 0/20 and 2/20. These gains coexist with substantial remaining failures. The reported 1.03 seconds per planning step on an RTX 5090 concerns high-level planning, not a measured end-to-end low-level control frequency.
| Method | Box Folding | Box Packing (Unseen) | Make a Line |
|---|---|---|---|
| End-to-End Policy | 5% | 0% | 10% |
| Dense Video Planning | 0% | 5% | 0% |
| Uniform Subgoal Planning | 10% | 20% | 25% |
| Ours | 40% | 65% | 55% |
VI Conclusion
Selecting useful, dataset-supported subgoals improves the tested planner/controller hierarchy and allows extra video data or search to improve the plan. Remaining limits include the reachability/controller compatibility assumption, dependency on pretrained progress/perception models, the 5B planner’s cost, small physical evaluations and substantial physical failure rates. The paper supplies no separate limitations appendix or complete optimizer/search-configuration table. Original inconsistencies in goal counts and evaluation descriptions are retained and called out rather than filled with assumed settings.