Literature
Display
← All papers

Hierarchical Planning with Latent World Models

Wancong Zhang · Basile Terver · Artem Zholus · Soham Chitnis · Harsh Sutaria · Mido Assran · Randall Balestriero · Amir Bar · Yann LeCun · Nicolas Ballas

Preprint posted · CoRL 2026 (accepted) ↗

World models6 min text · CondensedOriginal paper ↗
Contents

Overview

The teaser contrasts a high-level latent subgoal plan with the primitive actions used to reach the next subgoal. Its shared representation differs from the later H-JEPA hierarchy.

High Level WM Low Level WMgeneratesubgoalRefer to captionRefer to captionRefer to captionGoal ObsRefer to captionRefer to captionRefer to captionSubgoalHigh Level Plan to GoalLow Level Plan to SubgoalRefer to captionRefer to captionCurrent Obsactions
Original figure
Figure 1: Top: Hierarchical planning in latent space. A high-level planner optimizes macro-actions using a long-horizon world model to reach the goal; the first predicted latent state from the high-level rollout serves as the subgoal for a low-level planner, which optimizes primitive actions with a short-horizon world model. Both levels replan as new observations arrive, in a closed-loop MPC fashion. Solid borders denote ground-truth observations; others are decoder reconstructions shown for interpretability only. Bottom Left: HWM improves long-horizon planning on 3 latent world model architectures across 4 task suites. Bottom Right: Success rate versus normalized planning time. HWM matches or exceeds performance with up to 3×3\times lower planning time.

At a glance

Intuition. To put a cup on a shelf, first plan a useful intermediate grasp and lift, then plan the fine motions that reach it. HWM learns a longer-timescale predictor and compressed macro-actions on top of a shorter-timescale visual world model. Both levels share a state representation, so the high-level model’s predicted next state can become the low-level planner’s goal. At test time, it searches over actions rather than running a separately trained hierarchical policy. The framework improves the studied long-horizon simulated tasks and supports real Franka manipulation from a goal image. Shared latent geometry, finite rollout accuracy and the reachability of proposed subgoals remain limitations; this is distinct from October’s H-JEPA, which learns a separate representation at each level.

1 Introduction

A direct goal-matching optimizer may favor a greedy shortcut even when success requires first moving away from the goal, and long rollouts compound prediction errors. HWM addresses these problems with learned temporal abstraction and top-down subgoal planning. Its high-level action space compresses primitive-action chunks, reducing the search dimension while keeping planning task-agnostic.

z1z_{1}EEs1s_{1}F(2)F^{(2)}lt1l_{t_{1}}z^t2\hat{z}_{t_{2}}F(2)F^{(2)}lt2l_{t_{2}}F(2)F^{(2)}ltHl_{t_{H}}z^tH+1\hat{z}_{t_{H+1}}⋅⋅⋅⋅⋅\cdot\ \ \cdot\ \ \cdot\ \ \cdot\ \ \cdotF(1)F^{(1)}a1a_{1}F(1)F^{(1)}aha_{h}z^h+1\hat{z}_{h+1}…\dotszgoalz_{\text{goal}}EEsgoals_{\text{goal}}ℒlow\mathcal{L}_{\text{low}}ℒhigh\mathcal{L}_{\text{high}}High-Level Planningto Goalarg⁡min{l}⁡ℒhigh\arg\min_{\{l\}}\mathcal{L}_{\text{high}} Low-Level Planningto 1st1^{\text{st}} Subgoalarg⁡min{a}⁡ℒlow\arg\min_{\{a\}}\mathcal{L}_{\text{low}} Model keyF(1)F^{(1)}Low-level world modelF(2)F^{(2)}High-level world model
Figure 2: Hierarchical planning in latent space. A high-level planner optimizes macro actions using a long-horizon latent world model to reach the final goal. The first high level latent prediction serves as a subgoal for low-level planning, where a short-horizon world model optimizes primitive actions to reach this subgoal.

2 Hierarchical Planning with Latent World Models

The agent receives current and goal observations and searches under offline-trained latent dynamics. Receding-horizon execution repeatedly observes feedback. The hierarchy changes how this search is organized; it does not provide a reward-trained controller or assume known physical dynamics.

2.1 Learning Multi-Timescale Latent World Models

The low-level model predicts short transitions. A learned action encoder summarizes longer primitive-action chunks into macro-actions, and a high-level predictor forecasts their endpoints in the same latent state space. Fewer long-scale rollout steps reduce compounding error, while action compression shrinks the search space. The shared representation enables direct subgoal matching but limits level-specific abstraction.

Original figure
Figure 3: The high-level world model captures long-horizon dynamics using macro-actions encoded from chunks of low-level actions. It is causal and trained with a latent prediction loss.

2.2 Top-Down Hierarchical Planning

The high-level optimizer chooses macro-actions toward the final goal. Its next predicted latent waypoint becomes a target for a short-horizon primitive-action optimizer. Only an initial part is executed before replanning, allowing new observations to correct both levels. Predicted waypoints still have to be physically reachable by the low-level planner.

Equations
ℰ2(l^1:H;z1,zg)≜‖zg−F(2)(l^1:H;z1)‖1\mathcal{E}_{2}(\hat{l}_{1:H};z_{1},z_{g})\;\triangleq\;\left\|z_{g}-F^{(2)}(\hat{l}_{1:H};z_{1})\right\|_{1}
ℰ1(a^1:h;z1,z~1)≜‖z~1−F(1)(a^1:h;z1)‖1\mathcal{E}_{1}(\hat{a}_{1:h};z_{1},\tilde{z}_{1})\;\triangleq\;\left\|\tilde{z}_{1}-F^{(1)}(\hat{a}_{1:h};z_{1})\right\|_{1}

3 Empirical Evaluation of Hierarchical Planning Across World Models

The same planning principle is instantiated with V-JEPA2-AC, DINO-WM and PLDM. Real manipulation, longer Push-T episodes and held-out maze layouts test different failure modes; their backbones and data are not identical.

Refer to caption
Figure 4: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Executions on additional tasks are shown in appendix I.

3.1 Franka Arm with Robotiq Gripper

The Franka evaluation includes pick-and-place and drawer tasks with five trials per configuration. A matched flat world-model baseline helps isolate hierarchy; vision-language-action comparators use differing conditioning and sometimes extra camera information. The trial breakdown matters because success rates reflect configuration difficulty as well as stochastic execution.

Table 1: Zero-shot robotic manipulation with Franka, evaluated end-to-end from a single goal image. Each rate is computed over N=5N=5 trials per (start, goal) configuration; per-configuration breakdowns and 95% Clopper–Pearson confidence intervals are reported in Table 16. The flat VJEPA2-AC planner fails on tasks with non-greedy objectives (pick-&-place, multi-stage drawer); HWM solves them without manual decomposition.
Method P&P Cup P&P Box Drawer
Octo 0% 0% 43%
π0\pi_{0}-FAST-DROID 52% 18% –
π0.5\pi_{0.5}-DROID 68% 36% –
VJEPA2-AC 0% 0% 30%
VJEPA2-AC (hierarchy) 70% 60% 70%
Takeaway: HWM enables zero-shot manipulation under non-greedy objectives on a real robot from a single goal image, where flat world-model planners fail.

3.2 Push-T

Push-T start–goal pairs extend beyond the shorter horizons of the original DINO-WM setup. Two-level cross-entropy planning is compared with flat planning and offline goal-conditioned policies. Gains are interpreted together with the planning compute budget rather than as a task-independent hierarchy advantage.

Table 2: Push-T Performance Across Task Horizons. In each trial, a random start–goal pair dd steps apart is sampled from a validation trajectory.
Method d=25d=25 d=50d=50 d=75d=75
GCIQL 40%40\% 25%25\% 7.5%7.5\%
HIQL 55%55\% 30%30\% 20%20\%
HILP 25%25\% 13%13\% 0%0\%
DINO-WM 84%84\% 55%55\% 17%17\%
DINO-WM (hier.) 89% 78% 61%
Takeaway: HWM improves long-horizon control while reducing inference-time compute.
Original figure
Original figure
Figure 5: Success rate vs. test-time compute. We vary the planning budget by sweeping CEM/MPPI samples and iterations (see appendix C). Hierarchical planning achieves higher success while using up to 3×3\times less test-time compute than flat planners. Left: Push-T success rate vs. planning time (d=50d=50). Right: Diverse maze success rate vs. planning time (D∈[9,12]D\!\in\![9,12]).

3.3 Diverse Maze

Diverse Maze uses image observations and held-out layouts with increasingly distant goals. Hierarchical path-integral planning improves the reported long-range results over a flat PLDM planner. Generalization is to the described layout distribution, not arbitrary environments or unseen dynamics.

Table 3: Diverse Maze Performance Across Task Horizons. Agents are evaluated zero-shot on navigation tasks in held-out test maps. For each task, start and goal positions are randomly sampled such that their shortest-path distance is DD.
Method D∈[5,8]D\!\in\![5,8] D∈[9,12]D\!\in\![9,12] D∈[13,16]D\!\in\![13,16]
GCIQL 85% 40% 33%
HIQL 88% 73% 48%
HILP 48% 20% 10%
PLDM 100% 84%84\% 44%44\%
PLDM (hier.) 100% 95% 83%
Takeaway: HWM consistently improves zero-shot planning across diverse latent world models.

3.4 Ablations and Analysis

The short-timescale predictor is more accurate nearby, while the long-timescale predictor becomes more accurate at longer horizons in the tested robot data. Macro-action and capacity ablations show that extra parameters alone do not explain the reported gains. Subgoal usefulness depends on both being a plausible future state and being reachable by lower-level search.

Original figure
Figure 6: Prediction error (L1L_{1}) as a function of prediction horizon. For short horizons (≤\leq 1 s), the low-level world model is more accurate. For longer horizons (≥\geq 1.5 s), a single-step prediction from the high-level world model outperforms autoregressive rollouts of the low-level model, reflecting reduced error accumulation.
Table 4: Learned macro-actions versus alternative high-level action representations. (a) Concatenated primitive actions: planning success rates on medium/hard Diverse Maze tasks. (b) Handcrafted delta-pose actions, which summarize each action sequence by its net end-effector displacement: alignment of the offline plan with expert behavior on Franka (mean ±\pm SE), measured by cosine similarity and ℓ1\ell_{1} distance.
(a) Concatenated actions: Diverse Maze.
Action D∈[9,12]D\!\in\![9,12] D∈[13,16]D\!\in\![13,16]
Concatenated 52% 37%
Macro-actions 95% 83%
(b) Handcrafted actions: Franka.
Action Cos ↑\uparrow ℓ1\ell_{1} ↓\downarrow
Delta pose 0.80±0.020.80\pm 0.02 0.088±0.0050.088\pm 0.005
Macro-actions 0.88±0.03\mathbf{0.88\pm 0.03} 0.080±0.002\mathbf{0.080\pm 0.002}

4 Related Works

The paper separates predictive world-model planning from hierarchical policies that amortize actions through task-oriented learning, and from controllers using known dynamics. Earlier temporal abstraction ideas remain foundational; the contribution is a modular hierarchy over visual latent models trained through prediction.

Table 5: Comparison of hierarchical model-based methods. HWM is the only entry supporting zero-shot MPC directly from pixels, using world models pretrained via next-latent prediction, with predictions from coarser models serving as subgoals for finer-scale MPC via latent matching.
Method Zero-shot Representation Training Hierarchical Interface
HWM (ours) ✓ pixels →\rightarrow latent Next-latent pred. Subgoal matching (latent)
CAVIN [20] ✓ structured state Learned dynamics Subgoal matching (state)
LAT [21] ✓ structured state Learned dynamics Skill-conditioned policy
Safe HMPC [22] ×\times structured state Analytical dynamics Reference tracking
IQL-TD-MPC [42] ×\times state →\rightarrow latent Offline RL + MPC Goal-conditioned policy
Director,Puppeteer [14, 19] ×\times pixels →\rightarrow latent Model-based RL Goal-conditioned policy
THICK [17] ×\times pixels →\rightarrow latent MB-RL + reward Subgoal matching (latent)

5 Conclusion

High-level predictions can provide useful intermediate goals without manual decomposition at deployment. The experiments support this mechanism across several backbones, but every tested method still degrades as tasks become harder or longer.

6 Limitations

A shared representation simplifies transferring subgoals but may retain detail unnecessary at higher levels. High-level waypoints are determined by temporal sampling and macro-action design, and may be difficult for low-level planning. Learned subgoal discovery and feedback from low-level feasibility are proposed improvements.

Appendix Contents

Appendix A Training Details

A.1 Low Level World Model

The low-level objective combines teacher-forced next-latent prediction with an autoregressive rollout term. These two losses train local accuracy and behavior under self-generated context, respectively; rollout training does not remove long-horizon model bias.

Equations 1–3
z^k+1\displaystyle\hat{z}_{k+1} :=Fθ(1)​((at,zt)t≤k)\displaystyle:=F^{(1)}_{\theta}\!\left((a_{t},z_{t})_{t\leq k}\right) (1)
ℒtf​(θ)\displaystyle\mathcal{L}_{\mathrm{tf}}(\theta) :=1T−1​∑k=1T−1‖z^k+1−zk+1‖1.\displaystyle:=\frac{1}{T-1}\sum_{k=1}^{T-1}\left\|\hat{z}_{k+1}-z_{k+1}\right\|_{1}.
ℒroll(θ):=∑j=2T−1∥Fθ(1)(a1:j;z1)−zj+1∥1\displaystyle\mathcal{L}_{\mathrm{roll}}(\theta):=\sum_{j=2}^{T-1}\|F^{(1)}_{\theta}(a_{1:j};z_{1})-z_{j+1}\|_{1} (2)
L⁡(θ):=γt​f∗ℒtf​(θ)+γr​o​l​l∗ℒroll​(θ)\displaystyle L(\theta):=\gamma_{tf}*\mathcal{L}_{\mathrm{tf}}(\theta)+\gamma_{roll}*\mathcal{L}_{\mathrm{roll}}(\theta) (3)

A.2 Franka Arm with Robotiq Gripper

Robot data combines DROID and RoboSet trajectories with visual observations, end-effector proprioception and pose-change actions. Sampling frame rates and action/state conventions are retained because they define the temporal scale of the learned macro-actions.

Table 6: Training hyperparameters for Droid + Roboset. γt​f\gamma_{tf} and γr​o​l​l\gamma_{roll} denote the coefficients for teacher-forcing and rollout losses, respectively, and context TT counts observation frames, including the initial frame. All other hyperparameters follow those used by VJEPA2-AC.
World Model Epochs Batch Size γt​f\gamma_{tf} γr​o​l​l\gamma_{roll} context TT
VJEPA2-AC (Low-Level) 200 256 1.0 1.0 16
VJEPA2-AC (High-Level) 120 768 1.0 0.0 3

A.3 Push-T

Push-T uses the released DINO-WM dataset and frozen DINOv2 features. Actions are chunked across the image stride. The high-level predictor changes temporal abstraction on top of this backbone rather than retraining all visual perception end to end.

Table 7: Training hyperparameters for Push-T. γt​f\gamma_{tf} and γr​o​l​l\gamma_{roll} denote the coefficients for teacher-forcing and rollout losses, respectively, and context TT counts frames including the initial observation. All other hyperparameters follow those used by DINO-WM on Push-T.
World Model Epochs Batch Size γt​f\gamma_{tf} γr​o​l​l\gamma_{roll} context TT
DINO-WM (Low-Level) 100 256 1.0 0.0 4
DINO-WM (High-Level) 500 128 1.0 0.0 5

A.4 Diverse Maze

Random maze trajectories span many layouts and velocities; held-out layouts test navigation generalization. The main text’s10-by-10 layout includes the outer boundary, while the appendix describes the 8-by-8 interior grid. Velocity and coverage remain important to interpretable comparisons.

Table 8: Training hyperparameters for Diverse Maze. α\alpha, β\beta, λ\lambda, and ω\omega are parameters in the PLDM objective used to prevent representation collapse. ℒproprio\mathcal{L}_{\text{proprio}} coef. denotes the coefficient on the proprioceptive prediction loss. γt​f\gamma_{tf} and γr​o​l​l\gamma_{roll} are the coefficients for teacher-forcing and rollout losses, and pred TT is the number of autoregressive prediction steps during training, excluding the initial state. For the high-level world model, we reuse the encoder trained at the low level and keep it frozen, eliminating additional representation learning.
World Model Epochs Batch Size lr α\alpha β\beta λ\lambda ω\omega γt​f\gamma_{tf} γr​o​l​l\gamma_{roll} pred TT ℒproprio\mathcal{L}_{\text{proprio}}
PLDM (Low-Level) 3 128 0.018 29.4 17.9 2.80 4.81 0.0 1.0 14 2.42
PLDM (High-Level) 5 128 0.018 – – – – 0.0 1.0 6 1

Appendix B Planning Details

CEM and MPPI have different low- and high-level budgets. Primitive actions have physical units, whereas macro-action scale comes from training, so their search distributions require different calibration. Full sampling and replanning settings are preserved for reproducing compute–success tradeoffs.

Equations
ht=min⁡(h,B−t),h_{t}=\min\!\left(h,B-t\right),
Ht=max⁡(Hmin,H−⌊t/h⌋),ht=h.H_{t}=\max\!\left(H_{\min},H-\left\lfloor t/h\right\rfloor\right),\qquad h_{t}=h.
Table 9: Single-level CEM planning hyperparameters for Franka and Push-T.
kk BB # elites # iters # samples Var EMA hh
Franka 1 – 20 15 2400 0.75 6
Push-T (d=25d=25) 5 10 30 30 300 0 5
Push-T (d=50d=50) 5 20 10 30 1200 0.9 10
Push-T (d=75d=75) 5 25 20 20 1000 0 15
Table 10: Hierarchical CEM planning hyperparameters for Franka and Push-T.
High-level planner Low-level planner
kk BB # elites # iters # samples Var EMA HH # elites # iters # samples Var EMA hh
Franka 1 – 22 15 3000 0.65 2 12 5 800 0.25 2
Push-T(d=25d=25) 5 10 10 20 900 0 2 10 30 300 0 5
Push-T(d=50d=50) 5 20 10 40 1500 0.9 4 10 20 900 0.8 5
Push-T(d=75d=75) 5 25 10 20 1200 0.9 5 10 20 1200 0.8 5
Table 11: Single-level MPPI planning hyperparameters for Diverse Maze.
kk BB noise σ\sigma # samples λ\lambda hh
Maze (D∈[5,8]D\in[5,8]) 4 500 5 125 0.0025 150
Maze (D∈[9,12]D\in[9,12]) 4 500 5 250 0.0025 200
Maze (D∈[13,16]D\in[13,16]) 4 500 5 250 0.0025 250
Table 12: Hierarchical MPPI planning hyperparameters for Diverse Maze.
High-level planner Low-level planner
kk BB noise σ\sigma # samples λ\lambda HH noise σ\sigma # samples λ\lambda hh
Maze (D∈[5,8]D\in[5,8]) 4 265 10 2000 0.0025 25 5 500 0.0025 10
Maze (D∈[9,12]D\in[9,12]) 4 365 10 2000 0.0025 35 5 500 0.0025 10
Maze (D∈[13,16]D\in[13,16]) 4 485 10 4000 0.0025 47 5 1000 0.0025 10

Appendix C Computational Analysis Details

C.1 Push-T

Compute frontiers are obtained by sweeping candidate counts, iterations and distribution smoothing at several Push-T horizons. This compares optimized configurations across budgets instead of one arbitrarily chosen runtime setting.

Original figure
Figure 7: Success rate as a function of test-time compute for Push-T. Left: Setting d=50d=50. Right: Setting d=75d=75.

C.2 Diverse Maze

Maze frontiers vary flat horizons and both levels’ sampling budgets. Measured wall-clock comparisons depend on the implementation, model shapes and hardware; they are not operation-count guarantees for every planner.

Original figure
Figure 8: Success rate as a function of test-time compute for Diverse Maze. Left: Setting D∈[9,12]D\in[9,12]. Right: Setting D∈[13,16]D\in[13,16].

Appendix D Additional Ablations and Analysis

D.1 Model Size vs. Hierarchy

Larger flat models fail to reproduce the hierarchical gains in the capacity controls. This supports the importance of planning structure in the studied settings, without proving that no differently trained flat model could match it.

Table 13: Capacity-controlled comparison between flat and hierarchical world models. Flat models with comparable or larger parameter counts fail to match hierarchical performance, and sometimes degrade with increased capacity. In contrast, hierarchical models consistently outperform flat baselines on Push-T and Diverse Maze, indicating that gains arise from hierarchical structure rather than model capacity.
(a) Push-T
Method Params d=50d{=}50 d=75d{=}75
DINO-WM (flat) 4444M 55%55\% 17%17\%
DINO-WM (flat) 9898M 35%35\% 15%15\%
DINO-WM (hierarchy) 9494M 78% 61%
(b) Diverse Maze
Method Params D∈[9,12]D\!\in\![9,12] D∈[13,16]D\!\in\![13,16]
PLDM (flat) 5454k 85%85\% 63%63\%
PLDM (flat) 178178k 82%82\% 59%59\%
PLDM (hierarchy) 182182k 95% 83%

D.2 Strides for Fixed Interval Waypoints

A moderate temporal stride helps, but excessive abstraction eventually hurts predictive fidelity and success. The optimal stride therefore balances fewer high-level steps with enough information for reliable subgoals.

Table 14: Sensitivity of hierarchical planning performance to the high-level temporal stride on Diverse Maze.
High-level stride 6 8 10 12 14
Success Rate (D∈[13,16]D\!\in\![13,16]) 50% 55% 77% 78% 75%

D.3 Emergence of semantic subgoals

A macro-action bottleneck must be expressive enough for useful transitions but restrictive enough to avoid complicated, hard-to-reach subgoals. The reported Franka choice reflects this tradeoff. A visually meaningful high-level prediction does not automatically imply low-level executability.

Original figure
Refer to caption
Figure 9: Effect of macro-action dimension on hierarchical planning. Left: Performance as a function of macro-action dimension dd. When the latent space has very low capacity, the high-level planner fails to produce valid plans. As dd increases, success improves; however, beyond a moderate dimensionality, proposed subgoals become harder for the low-level planner to execute, as indicated by a drop in action cosine similarity to expert behavior. Right: Qualitative rollouts illustrating this trade-off. Moderate latent dimensionality biases the planner toward reachable, greedy subgoals, yielding the best overall performance.

Appendix E Manual Subgoals: Isolating Planning from Representation

Manually supplying intermediate grasp images lets the flat planner solve tasks it fails end to end. This control separates planning-objective difficulty from a complete absence of useful dynamics. Such supervised subgoals are a diagnostic, not part of the autonomous HWM deployment protocol.

Table 15: Manual subgoals on Franka. Success rates (%) over 50 trials per object (10 configurations, 5 per configuration).
Planner Subgoal Cup Box
VJEPA2-AC None 0% 0%
HWM Predicted 70% 60%
VJEPA2-AC Manual 80% 80%
HWM Manual 80% 80%
Octo Manual 20% 10%

Appendix F Visualizing World Model Predictions

Separate decoders visualize high-level latent predictions and execution trajectories. Decoder training is not part of the reward-free world-model objective, and a plausible decoded image cannot alone certify a feasible robot motion.

Refer to caption
Figure 10: High-level plans in Diverse Maze. The agent is shown as a green ball and the goal as a yellow star. The pink trajectory is generated by the high-level planner, while the teal trajectory is generated by the flat planner. In these trials, the high-level planner successfully produces a sequence of subgoals that lead to the goal, whereas the flat planner often struggles due to the increased search complexity of long-horizon prediction.

Appendix G Robot Task Definitions

The retained initial and goal images specify pick-and-place and drawer configurations. Box tasks use the same arrangements as cup tasks with a different manipulated object.

Refer to caption
Figure 11: Initial and goal states for all evaluation tasks. (a) Drawer Tasks: 7 subtasks involving opening and closing drawers at various positions. (b) Pick-&-Place Cup Tasks: 10 subtasks requiring the robot to pick up a cup and place it at different target locations. Each panel shows the initial state (left) and goal state (right) for a given task.

Appendix H Per-Configuration Trial Breakdown for Franka

Per-configuration counts and exact confidence intervals accompany aggregate success. Outcomes are largely repeatable within a configuration, so variability across task arrangements is an important source of the aggregate rates.

Table 16: Per-configuration success counts (successes / trials) on Franka, with 95% Clopper–Pearson confidence intervals on aggregate rates. Configurations correspond to the start–goal pairs visualized in appendix G.
(a) Pick-&-Place Cup
Config VJEPA2-AC HWM (ours)
PnP 0 0/5 0/5
PnP 1 0/5 5/5
PnP 2 0/5 5/5
PnP 3 0/5 5/5
PnP 4 0/5 5/5
PnP 5 0/5 4/5
PnP 6 0/5 1/5
PnP 7 0/5 5/5
PnP 8 0/5 0/5
PnP 9 0/5 5/5
Total 0/50 35/50
95% CI [0.0, 7.1] [55.4, 82.1]
(b) Pick-&-Place Box
Config VJEPA2-AC HWM (ours)
PnP 0 0/5 0/5
PnP 1 0/5 1/5
PnP 2 0/5 4/5
PnP 3 0/5 5/5
PnP 4 0/5 5/5
PnP 5 0/5 4/5
PnP 6 0/5 1/5
PnP 7 0/5 5/5
PnP 8 0/5 0/5
PnP 9 0/5 5/5
Total 0/50 30/50
95% CI [0.0, 7.1] [45.2, 73.6]
(c) Drawer (opening and closing)
Config Open 0 Open 1 Open 3 Close 0 Close 1 Close 2 Close 4 Total 95% CI
VJEPA2-AC 5/5 0/5 0/5 5/5 0/5 0/5 0/5 10/35 [14.6, 46.3]
HWM (ours) 5/5 5/5 0/5 5/5 5/5 5/5 0/5 25/35 [53.7, 85.4]

Appendix I Additional Execution Rollouts for Franka

Additional Franka execution sequences retain inferred subgoals and actual motion, allowing comparison between planned intermediate images and execution.

Refer to caption
Figure 12: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka arm. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Rows representing failed episodes are labeled with "Failure". Additional execution videos can be found in https://kevinghst.github.io/HWM/.

Appendix J Execution Rollout Examples for Push-T

Additional Push-T rollouts retain high-level subgoals and the corresponding lower-level action execution. These selected examples supplement the aggregate evaluation.

Refer to caption
Figure 13: Executions on Push-T task. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization.

Appendix K Offline Planning Evaluation for Franka

An offline proxy compares planned action aggregates with expert directions for the evaluated manipulation configurations. It permits faster iteration but uses manually specified subgoal structure and cannot substitute for all closed-loop robot trials.

Appendix L Baselines

L.1 Vision-Language-Action Models

Camera, conditioning and checkpoint choices for the vision-language-action models are documented. Some policy baselines consume wrist images or language goals unavailable to the world-model planner, so the comparison is informative but not a perfectly matched ablation.

L.2 Goal-Conditioned & Zero-Shot RL Methods

Goal-conditioned IQL, hierarchical IQL and HILP use learned policies or value geometry rather than online predictive search. Their official implementations and per-difficulty tuning are retained to separate baseline construction from the hierarchy claim.

Table 17: Hyperparameters. Hyperparameters used for GCIQL, HIQL, and HILP on the Push-T and Diverse Maze tasks.
GCIQL HIQL HILP
lr expectile rand goal pp lr expectile rand goal pp alpha lr expectile
Push-T(d=25d=25) 3​e−53\mathrm{e}{-5} 0.6 0.0 3​e−53\mathrm{e}{-5} 0.9 0.0 3.0 3​e−53\mathrm{e}{-5} 0.6
Push-T(d=50d=50) 3​e−53\mathrm{e}{-5} 0.6 0.3 3​e−53\mathrm{e}{-5} 0.9 0.3 1.0 3​e−53\mathrm{e}{-5} 0.6
Push-T(d=75d=75) 3​e−53\mathrm{e}{-5} 0.9 0.6 3​e−53\mathrm{e}{-5} 0.9 0.3 1.0 3​e−53\mathrm{e}{-5} 0.6
Maze(D∈[5,8]D\in[5,8]) 3​e−53\mathrm{e}{-5} 0.8 0.0 3​e−53\mathrm{e}{-5} 0.6 0.3 1.0 3​e−53\mathrm{e}{-5} 0.9
Maze(D∈[9,12]D\in[9,12]) 3​e−53\mathrm{e}{-5} 0.6 0.3 3​e−53\mathrm{e}{-5} 0.8 0.6 3.0 3​e−53\mathrm{e}{-5} 0.9
Maze(D∈[13,16]D\in[13,16]) 1​e−41\mathrm{e}{-4} 0.6 0.0 3​e−53\mathrm{e}{-5} 0.8 0.6 3.0 3​e−53\mathrm{e}{-5} 0.9
Concept demonstration

Open image ↗