Literature
Display
← All papers

Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation

Chongkai Gao · Wenlong Huang · Haonan Chen · Lin Shao · Yilun Du · Jiajun Wu · Ruohan Zhang · Li Fei-Fei

Preprint posted · arXiv (preprint) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

DreamCan turns robot videos into sparse visual plans by selecting frames that combine task progress, dataset-supported reachability and temporal spacing. A fine-tuned 5B video model generates those plans; a separate 0.1B controller executes them. This edition restores the missing opening overview and retains all ten original figures, five tables, six equations and 35 references. Read the planning-only evaluations separately from closed-loop robot success, and the small controller size separately from the much larger pretrained planner.

Fig. 1: Overview of DreamCan. Top left: given a video dataset of a manipulation task, our method can perform unsupervised sparse subgoal image sequence discovery to get subgoal sequences that balance subgoal reachability and task utility; Bottom left: trained on these subgoal sequence datasets, the sparse visual planner can generate a subgoal image sequence given an image observation and language instruction, and a goal-conditioned low-level policy can generate robot actions according to the subgoals. Right: with our subgoal sequences, our parameter-efficient low-level policy can achieve comparable performance with large VLAs or WAMs on LIBERO-Long, and can scale with test-time compute and suboptimal data.

At a glance

The planner learns from action-free videos annotated automatically with useful, reachable-looking subgoals. Dynamic programming chooses training frames, then a goal-conditioned controller follows the generated image sequence with periodic replanning. DreamCan reaches 91.2% reported LIBERO-Long success and 86% PushT success, while benefiting from more suboptimal videos and test-time search. Physical success is lower—40% box folding, 65% unseen-combination packing and 55% line arrangement—but exceeds the tested alternatives. Reachability is an offline-data proxy, not a guarantee that a particular controller can execute every generated plan.

I Introduction

Video goals can express contact geometry and intermediate configurations that are awkward to specify in words. Dense futures may overconstrain execution, while uniformly sampled frames can miss important transitions or preserve detours. Inspired by SayCan’s separation of relevance and affordance, DreamCan scores task utility and transition reachability without a predefined skill vocabulary. Selected video frames supervise the planner, which communicates with an independently trained controller through images alone. Action-free planner learning still depends on video instructions, pretrained visual/progress models and a separately trained action-generating policy; it does not make the complete system annotation- or action-data-free.

II Related Work

Related work covers how to discover sparse task states and how learned world models support control. DreamCan’s main distinction is its supervision for choosing the temporal granularity of explicit image goals.

II-A Keystate Planning for Robot Manipulation Tasks

Manual keyframes and embodiment-specific cues can provide useful subgoals but are costly or unavailable in video-only datasets. Semantic milestones from vision-language models need not be executable. Compared with KeyIn, SuSIE and latent backward planning, DreamCan explicitly combines estimated transition ease with task progress and spacing when choosing training frames.

II-B World Models for Robot Manipulation Tasks

Latent dynamics models support policy optimization or planning, while video and world-action models predict observations or joint observation/action futures. DreamCan uses a video model only for sparse explicit image goals. The planner and controller share neither gradients nor hidden activations, although both rely on compatible goal-image semantics. This modular interface permits separate upgrades; it does not remove the need to match generated goals to controller capability.

III Preliminary and Motivation

The method is a hierarchy from language and current images to visual subgoals, then from observations and those goals to low-level action chunks. The motivation is to make each visual target informative without prescribing every intermediate motion.

III-A Preliminaries

The planner generates a short ordered sequence of goal images from the current observation and instruction. A goal-conditioned policy receives multiple future goals and produces an action chunk; subsequent observations allow periodic replanning. Planner training uses videos, while the controller is trained independently with action supervision. The image sequence is their sole interface, so planner improvement can be evaluated with a fixed controller.

Equations 1–2
g^1,g^2,⋯,g^K=𝒫θ​(ot,ℓ),g^k∈𝒪,\hat{g}_{1},\hat{g}_{2},\cdots,\hat{g}_{K}=\mathcal{P}_{\theta}(o_{t},\ell),\quad\hat{g}_{k}\in\mathcal{O}, (1)
at:t+H−1∼πϕ(at:t+H−1∣ot,g^1:k,ℓ),a_{t:t+H-1}\sim\pi_{\phi}\left(a_{t:t+H-1}\mid o_{t},\hat{g}_{1:k},\ell\right), (2)

III-B Motivation: What Makes a Good Visual Plan?

PushT illustrates the tradeoff. Language may omit the exact contact point or relative geometry; a dense sequence can drift from actual execution; uniform frame selection can place a difficult transition beside an almost redundant one. Useful subgoals should be reachable from preceding states and advance the task. Neither closeness alone nor apparent final-goal utility alone suffices.

Refer to caption
Fig. 2: The problems of language planning, dense video planning, and uniform subgoal planning on the Push-T task.

IV Method

Three steps create the hierarchy: estimate reachability and progress from videos, select sparse training targets by dynamic programming, then fine-tune a pretrained video generator and pair it with a goal-conditioned controller.

Refer to caption
Fig. 3: Subgoal annotation for Box Folding. Left: demonstration frames. Middle: reachability, task utility, their combination, and the dynamic programming cost curves used for selection at the initial frame. Note that reachability and DP curves will change with the starting state. Right: full cost curves JJ during the subgoal selection process. We illustrate four subgoals for clarity; the task uses twelve in our experiments.

IV-A Dataset-Supported Reachability

Reachability estimates discounted first-arrival time using a goal-conditioned value function learned from dataset transitions. It is conditioned on data coverage and the assumption that the eventual controller can perform behavior similar to that recorded.

Formulation

With reward −1 before reaching the target and zero at termination, normalized value is one plus (one minus discount) times the value. Equivalently, it is the expected discount raised to the first-hitting time, with zero contribution when the goal is never reached. A higher score combines faster arrival and more reliable arrival; it is not a fixed-horizon success probability.

Equation 3
Rπ​(si,sj)\displaystyle R^{\pi}(s_{i},s_{j}) =1+(1−γ)​VRπ​(si,sj)\displaystyle=1+(1-\gamma)V_{R}^{\pi}(s_{i},s_{j}) (3)
=𝔼π​[γτi​j]∈[0,1],\displaystyle=\mathbb{E}_{\pi}[\gamma^{\tau_{ij}}]\in[0,1],

Learning from videos

Frozen RADIO features encode observations and goals. A temporal Transformer combines observation history, goal and language to learn action-free goal-conditioned values using an upper expectile of 0.75, discount 0.95 and target-update rate 0.005. The goal-state boundary is zero value. Half the sampled goals come from geometrically offset future frames and half from other trajectories; cross-trajectory goals are not automatically labeled impossible. The upper expectile emphasizes better dataset transitions, following ideas from implicit Q-learning and HIQL. This remains dataset-supported reachability rather than an online measurement of the current controller.

IV-B Task Utility Value Function

Most tasks adapt RoboMeter with LoRA. Spatial tasks supplement progress estimates with vision-language outputs and SAM masks. For successful demonstrations, smoothing, start/completion normalization and a running maximum produce a [0,1] progress score. The running maximum records the best progress reached so far, so it can remain high during a later regression; it is an annotation signal rather than the instantaneous physical success state.

IV-C Subgoal Selection by Dynamic Programming

Selection minimizes summed negative-log reachability and a temporal-gap penalty, minus utility at the final selected frame. Intermediate utility differences telescope, so utility mainly chooses a useful endpoint while reachability places intermediate steps. A minimum gap constrains increasing frame indices. Backward dynamic programming stores the best next index for each remaining budget and runs in O(KL²) after scoring; backtracking recovers K selected frames and appending the demonstration’s terminal frame yields K+1 targets. The original permits the final selected index to equal the terminal index, so the written rule can duplicate that endpoint; no undocumented deduplication is assumed. Figure 3 illustrates four selected goals and states twelve in experiments, while Figure 6’s box-folding row labels a goal g15; that source horizon discrepancy remains visible.

Equations 4–5
mint1:K∑k=1K[wrcR(tk−1,tk)+wsΩk(tk−1,tk)]−wgU(stK),\min_{t_{1:K}}\sum_{k=1}^{K}\bigl[w_{r}c_{R}(t_{k-1},t_{k})+w_{s}\Omega_{k}(t_{k-1},t_{k})\bigr]-w_{g}U(s_{t_{K}}), (4)
Jm​(i)\displaystyle J_{m}(i) =minj⁡[em​(i,j)+Jm−1​(j)],\displaystyle=\min_{j}\bigl[e_{m}(i,j)+J_{m-1}(j)\bigr], (5)
J0​(i)\displaystyle J_{0}(i) =−wg​U​(si),\displaystyle=-w_{g}U(s_{i}),

IV-D Subgoal Generation with a Pretrained Video Model

The high-level generator is pretrained Wan2.2-TI2V-5B fine-tuned to output selected subgoal sequences. Its 5B scale matters when interpreting comparisons advertised in terms of the separate 0.1B controller.

Sparse latent construction

Wan’s causal video autoencoder compresses four frames per temporal slot. Each selected goal image is repeated four times after the initial observation so different goals occupy distinct latent slots instead of being mixed by temporal compression. Equation 6 retains the exact input/latent construction.

Equation 6
[ot,g1,…,g1⏟4,…,gK,…,gK⏟4]→VAE[zot,zg1,…,zgK].\bigl[o_{t},\underbrace{g_{1},\ldots,g_{1}}_{4},\ldots,\underbrace{g_{K},\ldots,g_{K}}_{4}\bigr]\xrightarrow{\mathrm{VAE}}\bigl[z_{o_{t}},z_{g_{1}},\ldots,z_{g_{K}}\bigr]. (6)

Closed-loop planner fine-tuning

LoRA adapts the diffusion Transformer with flow matching while freezing the video autoencoder and text encoder. Training starts from observations sampled throughout a trajectory and conditions on subsequent selected goals, exposing the planner to intermediate states needed for closed-loop replanning. Instruction dropout is 0.5 in domains with limited linguistic variety.

Prefix-conditioned generation for test-time search.

Thirty percent of training examples condition only on the observation. Otherwise a random clean prefix of one through K−4 goal latents is fixed, and denoising loss applies only to the remaining suffix. Prefix positions receive diffusion time zero. At test time, this allows search to commit chosen goals and regenerate the tail. The original alternates between K selected goals and K+1 terminal-appended targets; detailed configuration is needed to resolve the exact sequence length.

IV-E Goal-Conditioned Low-Level Policy

The 0.1B flow-matching Transformer receives current images, proprioception, instruction and several future subgoals. Current and goal images use frozen RADIO features; compressed goal tokens combine with state and language tokens as an action-generator prefix. The controller itself is not pretrained in the reported comparison, but its visual representation and the 5B planner are pretrained. Low policy parameter count therefore does not describe the total deployed system.

V Experiments

Evaluation separates visual-plan validity, end-to-end task success, benefits from extra suboptimal video, search/steering, local subgoal reachability and physical manipulation. These outcomes use different protocols and should not be conflated.

Refer to caption
Fig. 4: Illustration of the subgoals generated from our methods on all simulation tasks in this paper.

V-A Long-horizon Task Planning Results

Language-Table training uses 1,424 line, 1,303 region and 1,378 shape videos, resized to 320×240 with fifteen subgoals; the text specifies 100 test videos for each task. Plan success requires the instructed final arrangement, sustained object integrity and plausible tracked transitions. DreamCan scores 86/100/52% versus VLP’s 56/78/15% for line/region/shape; the authors reproduce VLP’s shape result. On line planning, dense and uniform visual alternatives score 71% and 69%. Table I also improves all reported image/video metrics, including FVD 154 versus 233/258 on Language Table. These are generated-plan evaluations, not physical execution success. Table III separately describes 50 evaluation episodes for its comparison, so its caption and the 100-video planning protocol should not be merged into a single assumed denominator.

TABLE I: Visual plan generation quality comparisons on Language Table (LT) and LIBERO-Long (LL) tasks.
Method PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow FVD ↓\downarrow
LT LL LT LL LT LL LT LL
Dense Visual Planning 27.54 18.52 0.859 0.834 0.115 0.128 233 370
Uniform Subgoal Planning 27.51 18.06 0.855 0.826 0.118 0.143 258 380
Ours 28.95 18.86 0.870 0.838 0.103 0.127 154 363
TABLE II: Long-horizon task planning success rates on three Language Table tasks. We reproduced the canonical shape result by ourselves because the original VLP didn’t do it.
VLP [7] DreamCan (Ours)
make a line 56% 86%
all to region 78% 100%
canonical shape 15% 52%
TABLE III: Comparison of different planning methods on PushT, LIBERO-Long, and Language Table make a line task. Numbers for our methods are the average success rates of 50 evaluation episodes. The VLA and WAM results on PushT were run by us. LIBERO-Long results are from their original papers. Bold numbers indicate the best results or our results.
Method PushT LIBERO-Long Language Table
E2E Policy 58.0% 79.2% –
Language Planning 62.0% 75.6% 56.0%
Dense Visual Planning 70.0% 84.0% 71.0%
Uniform Subgoal Planning 72.0% 82.4% 69.0%
VLA-OS [31] (0.9B, non-pretrained) 72.0% 76.0% –
π0\pi_{0} [32] (3.3B, pretrained) 76.0% 85.2% –
SEER [33] (0.3B, pretrained) 68.0% 87.7% –
π0.5\pi_{0.5} [34] (3.3B, pretrained) 86.0% 92.4% –
Lingbot-VA [35] (5.3B, pretrained) 89.0% 98.5% –
Ours (0.1B policy, non-pretrained) 86.0% 91.2% 86.0%

V-B Hierarchical Task Execution Results

PushT uses 206 demonstrations and LIBERO-Long 379, with a shared low-level architecture across representation baselines. DreamCan reaches 86.0% and 91.2%, compared with end-to-end 58.0/79.2%, language 62.0/75.6%, dense video 70.0/84.0% and uniform goals 72.0/82.4%. The exact columns qualify the narrative: language is below end-to-end only on LIBERO here, and uniform goals do not beat dense video there. DreamCan exceeds the reported VLA-OS, π0 and SEER results, but LIBERO remains below π0.5’s 92.4% and Lingbot-VA’s 98.5%; PushT ties π0.5 at 86% and trails Lingbot-VA’s 89%. Large-model PushT results are rerun by the authors, whereas LIBERO figures come from the original papers. The 0.1B controller comparison excludes DreamCan’s 5B pretrained planner.

V-C Scaling with Sub-Optimal Dataset

The Square study fixes a controller trained on 100 proficient demonstrations and trains planners with nested subsets of 5 through 200 suboptimal demonstrations. Increasing video data raises DreamCan success from roughly the low fifties to the high sixties in Figure 5; uniform and dense alternatives improve less consistently. This isolates adding planner video data without retraining the controller, but it is one task and one finite dataset rather than evidence of unrestricted large-scale scaling.

Refer to caption
Fig. 5: Top: subgoals found by uniform sampling and ours. Bottom: success rates on the Square task with more suboptimal data for different planners.
Refer to caption
Fig. 6: The real-world experiment setup, three real-world tasks, and subgoal sequences.

V-D Test-Time Search and Language Steering

Whole-plan best-of-N chooses among independent candidates. Anchored search instead commits the best prefix, pins those latents and regenerates the suffix, using the same scripted detection-based progress score used in training. On line planning, anchored search continues improving with more generator calls, reaching approximately 0.9 progress; each point uses 100 test episodes. Dense generation uses the same scorer and matched compute but remains worse. Progress optimized by this scorer is not independently measured execution success. Qualitative language steering changes intended line arrangements, and a vision-language-generated scoring function selects a packing arrangement from several rollouts.

Original figure
Fig. 7: Test time search results on the Language Table make a line task. Each point is evaluated over 100 testing episodes.
Original figure
Fig. 8: Per-subgoal reachability on 10 LIBERO-Long tasks. Top: ours. Bottom: uniform subgoal planning.
Refer to caption
Fig. 9: Language steering experiments. Left: given different language instructions, our planner can generate different corresponding planning results. Right: given only the black instruction, our planner can generate different rollouts, while a VLM instructed by the green text can give a task progress value function that selects the correct planning rollout.

V-E Reachability Value Function Evaluation

LIBERO segment tests reset and replay the simulator to each ground-truth starting subgoal, then check whether task-relevant objects reach the next target within 5 cm. DreamCan averages 0.98 segment success versus 0.88 for uniform goals; resetting means this is not an uninterrupted long-horizon success rate. Selection ablations give spacing-only 81.8%, reachability+spacing 74.2%, utility+spacing 78.0%, reachability+utility 83.8%, and all three 91.2%. This supports the full recipe under these settings, not universal necessity of every term. Square analyses compare learned arrival time with dataset shortest paths rather than elapsed demonstration time. In one dropped-and-regrasped example, reachability recovers to 0.80 when the state returns within 1.4 cm of its starting configuration, versus 0.004 for pure temporal decay.

TABLE IV: The ablation study of subgoal discovery terms.
DP Terms S RS US RU Ours
SR 81.8% 74.2% 78.0% 83.8% 91.2%
Refer to caption
Fig. 10: Left: learned reachability follows the shortest path, not elapsed time. Right: our reachability skips a failed grasp.

V-F Real-World Evaluation

A dual-arm YAM setup uses one D435 overhead and two D405 wrist cameras. Training provides 100 box-folding demonstrations, 550 packing demonstrations over 110 object combinations, and 90 line demonstrations over three target shapes. Ninety-five of 205 feasible packing combinations are held out; Table V evaluates twenty episodes per task, not all 95 combinations. DreamCan succeeds in 8/20 folding, 13/20 unseen-combination packing and 11/20 line trials. Uniform goals give 2/20, 4/20 and 5/20; dense planning 0/20, 1/20 and 0/20; end-to-end 1/20, 0/20 and 2/20. These gains coexist with substantial remaining failures. The reported 1.03 seconds per planning step on an RTX 5090 concerns high-level planning, not a measured end-to-end low-level control frequency.

TABLE V: Real-world experiment results. We evaluate each task with 20 episodes.
Method Box Folding Box Packing (Unseen) Make a Line
End-to-End Policy 5% 0% 10%
Dense Video Planning 0% 5% 0%
Uniform Subgoal Planning 10% 20% 25%
Ours 40% 65% 55%

VI Conclusion

Selecting useful, dataset-supported subgoals improves the tested planner/controller hierarchy and allows extra video data or search to improve the plan. Remaining limits include the reachability/controller compatibility assumption, dependency on pretrained progress/perception models, the 5B planner’s cost, small physical evaluations and substantial physical failure rates. The paper supplies no separate limitations appendix or complete optimizer/search-configuration table. Original inconsistencies in goal counts and evaluation descriptions are retained and called out rather than filled with assumed settings.

Concept demonstration

Open image ↗