Literature
Display
← All papers

H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning

Wancong Zhang · Basile Terver · Michael Rabbat · Yann LeCun · Randall Balestriero

· arXiv (preprint) ↗

World models11 min text · CondensedOriginal paper ↗
Contents

Overview

The teaser shows top-down planning across learned representation levels. Higher-level subgoals guide lower-level search instead of specifying primitive actions directly.

Refer to caption
Figure 1: Hierarchical abstractions support planning at multiple timescales. Left: visualization of hierarchical planning in Visual AntMaze. Level 3 plans toward the final goal in the most abstract latent space. Its first predicted state becomes a subgoal for the level-2 planner; this decomposition recurses to the level-1 planner, which outputs raw actions. The first and last columns decode each level’s start and goal latents, respectively; the middle columns decode its planned predictions. Higher-level latents retain the ant’s position and maze layout while losing leg-pose detail. See §2.2 for the formal planning procedure. Right: planning success versus planner compute on AntMaze. Curves show Pareto fronts for flat LeWM [53] and two-/three-level H-JEPA; error bars are standard error (SE) over three paired model and planner seeds (§4.1).

At a glance

Intuition. A navigator may need to remember which corridor it is in while ignoring the exact phase of its legs. H-JEPA learns several predictive representations: higher levels look farther ahead and can discard fast detail that becomes hard to predict at that scale. A high-level plan proposes latent subgoals, and lower levels refine them into primitive actions. Unlike April’s HWM, the levels have distinct learned state spaces, not just different clocks in a shared space. Simulated planning and representation probes support this separation of temporal search and goal geometry. Diverse DROID scenes expose a further failure: a model can encode stable background identity instead of the moving robot. Inverse-dynamics supervision helps there, but the DROID results are offline plan-fidelity measurements, not closed-loop deployment on a physical robot.

1 Introduction

A shared fine-grained representation may predict local dynamics well while supplying a poor distant-goal cost. H-JEPA changes both the prediction horizon and the representation at each level. The proposed benefit therefore has two mechanisms: shorter subproblems in time and a higher-level cost that ignores some irrelevant fast detail.

2 Method

2.1 Hierarchical predictive learning

The first level encodes observations and actions. Each higher level encodes lower-level latent windows and pooled action blocks, with a stride controlling how far it advances and a window controlling what it summarizes. Each level has its own predictor and anti-collapse objective. End-to-end training allows upper-level objectives to reshape encoders below them.

(a) Build hierarchy (w2=1w_{2}=1, s2=2s_{2}=2)State encodingAction encoding(b) Train each levelFor each level ℓ=1,2\ell=1,2F(ℓ)F^{(\ell)}o0o_{0}E(1)E^{(1)}z0(1)z^{(1)}_{0}o1o_{1}E(1)E^{(1)}z1(1)z^{(1)}_{1}o2o_{2}E(1)E^{(1)}z2(1)z^{(1)}_{2}o3o_{3}E(1)E^{(1)}z3(1)z^{(1)}_{3}Level 1E(2)E^{(2)}z0(2)z^{(2)}_{0}E(2)E^{(2)}z1(2)z^{(2)}_{1}Level 2a0a_{0}A(1)A^{(1)}a0(1)a^{(1)}_{0}a1a_{1}A(1)A^{(1)}a1(1)a^{(1)}_{1}A(2)A^{(2)}a0(2)a^{(2)}_{0}zt(ℓ)z^{(\ell)}_{t}z^t+1(ℓ)\hat{z}^{(\ell)}_{t+1}ℒpred(ℓ)\mathcal{L}^{(\ell)}_{\mathrm{pred}}zt+1(ℓ)z^{(\ell)}_{t+1}at(ℓ)a^{(\ell)}_{t}ContextPredictionTargetPer-level objectiveℒ(ℓ)=ℒpred(ℓ)+λℓ​SIGReg​(Z(ℓ))\displaystyle{\color[rgb]{0.4648,0.3203,0.6133}\mathcal{L}^{(\ell)}}={\color[rgb]{0.4648,0.3203,0.6133}\mathcal{L}^{(\ell)}_{\mathrm{pred}}}+\lambda_{\ell}\,\mathrm{SIGReg}\!\left({\color[rgb]{0.1563,0.3906,0.582}Z^{(\ell)}}\right)Regularize encoded statesSIGReg⁡(Z(ℓ)),Z(ℓ)={zt(ℓ)}batch, time\mathrm{SIGReg}\!\left({\color[rgb]{0.1563,0.3906,0.582}Z^{(\ell)}}\right),\qquad{\color[rgb]{0.1563,0.3906,0.582}Z^{(\ell)}}=\{{\color[rgb]{0.1563,0.3906,0.582}z^{(\ell)}_{t}}\}_{\text{batch, time}}
Figure 2: H-JEPA training across temporal scales. (a) With w2=1w_{2}{=}1 and s2=2s_{2}{=}2, level 2 encodes every other level-1 state pointwise and aggregates the intervening action embeddings. Faded columns are retained at level 1 but skipped at level 2. (b) Each level predicts the next latent state conditioned on preceding latent states and actions, with SIGReg regularizing the states against collapse. All levels are trained end-to-end, with gradients from upper-level objectives propagating through lower-level encoders.
Equations 1–3
zt(ℓ)=E(ℓ)(zt⋅sℓ:t⋅sℓ+wℓ(ℓ−1)),at(ℓ)=A(ℓ)(at⋅sℓ+wℓ−1:(t+1)⋅sℓ+wℓ−1(ℓ−1)).z^{(\ell)}_{t}=E^{(\ell)}\!\left(z^{(\ell-1)}_{t\cdot s_{\ell}\,:\,t\cdot s_{\ell}+w_{\ell}}\right),\qquad a^{(\ell)}_{t}=A^{(\ell)}\!\left(a^{(\ell-1)}_{t\cdot s_{\ell}+w_{\ell}-1\,:\,(t+1)\cdot s_{\ell}+w_{\ell}-1}\right). (1)
ℒpred(ℓ)=1cℓ​∑τ=1cℓ‖F(ℓ)​(z<τ(ℓ),a<τ(ℓ))−zτ+1(ℓ)‖22.\mathcal{L}^{(\ell)}_{\mathrm{pred}}=\frac{1}{c_{\ell}}\sum_{\tau=1}^{c_{\ell}}\left\|F^{(\ell)}(z^{(\ell)}_{<\tau},a^{(\ell)}_{<\tau})-z^{(\ell)}_{\tau+1}\right\|_{2}^{2}. (2)
ℒ(ℓ)=ℒpred(ℓ)+λℓ​SIGReg​(Z(ℓ)).\mathcal{L}^{(\ell)}=\mathcal{L}^{(\ell)}_{\mathrm{pred}}+\lambda_{\ell}\mathrm{SIGReg}\!\left(Z^{(\ell)}\right). (3)

2.2 Hierarchical planning over learned abstractions

Planning begins at the top with a final-goal cost. Lower-level predicted states are passed through the next encoder before matching the upper level’s subgoals, so states in different latent spaces are compared through the learned hierarchy. The process reaches primitive actions at level one and repeats with new observations during receding-horizon execution.

Hierarchical planningTwo-level example: subgoals K2=1K_{2}=1, stride s2=2s_{2}=2, window w2=1w_{2}=1z0(1)z^{(1)}_{0}o0o_{0}E(1)E^{(1)}E(2)E^{(2)}z0(2)z^{(2)}_{0}Initialg(1)g^{(1)}ogo_{g}E(1)E^{(1)}E(2)E^{(2)}g(2)g^{(2)}GoalLevel 2Level 1F(2)F^{(2)}z^1(2)\hat{z}^{(2)}_{1}⋯\cdotsF(2)F^{(2)}z^H2(2)\hat{z}^{(2)}_{H_{2}}C(2)C^{(2)}a0(2)a^{(2)}_{0}aH2−1(2)a^{(2)}_{H_{2}-1}F(1)F^{(1)}z^1(1)\hat{z}^{(1)}_{1}F(1)F^{(1)}z^2(1)\hat{z}^{(1)}_{2}a0(1)a^{(1)}_{0}a1(1)a^{(1)}_{1}E(2)E^{(2)}C(1)C^{(1)}SubgoalC(1)=‖E(2)​(z^2(1))−z^1(2)‖22{\color[rgb]{0.4648,0.3203,0.6133}C^{(1)}}=\bigl\|E^{(2)}({\color[rgb]{0.1563,0.3906,0.582}\hat{z}^{(1)}_{2}})-{\color[rgb]{0.1563,0.3906,0.582}\hat{z}^{(2)}_{1}}\bigr\|_{2}^{2}Level 1: optimize a0(1),a1(1){\color[rgb]{0.7188,0.3984,0.1328}a^{(1)}_{0},\,a^{(1)}_{1}}first subgoal term shownC(2)=‖z^H2(2)−g(2)‖22{\color[rgb]{0.4648,0.3203,0.6133}C^{(2)}}=\bigl\|{\color[rgb]{0.1563,0.3906,0.582}\hat{z}^{(2)}_{H_{2}}}-{\color[rgb]{0.1563,0.3906,0.582}g^{(2)}}\bigr\|_{2}^{2}Level 2: optimize a(2)0:H2−1{\color[rgb]{0.7188,0.3984,0.1328}a^{(2)}_{0:H_{2}-1}}to minimize goal cost
Figure 3: Planning runs top-down through the hierarchy, and the lower level is steered by subgoals the upper level predicted. Two-level instance (L=2L{=}2) of §2.2. Current and goal observations o0o_{0} and ogo_{g} are encoded bottom-up into initial states z0(1),z0(2)z^{(1)}_{0},z^{(2)}_{0} and goal states g(1),g(2)g^{(1)},g^{(2)}. Level 2 optimizes macro-actions by unrolling F(2)F^{(2)} and matching its final prediction to g(2)g^{(2)}. Its predicted states z^i(2)\hat{z}^{(2)}_{i} become subgoals for level 1, which optimizes primitive actions and lifts its predictions through E(2)E^{(2)} for comparison in level-2 latent space (Eq. (6)).
Equations 4–6
a0:HL−1(L),∗=argmina(L)0:HL−1‖z^HL(L)−g(L)‖22.a^{(L),*}_{0:H_{L}-1}=\arg\min_{a^{(L)}_{0:H_{L}-1}}\left\|\hat{z}^{(L)}_{H_{L}}-g^{(L)}\right\|_{2}^{2}. (4)
z~i(ℓ+1)=E(ℓ+1)(z^i​sℓ+1:i​sℓ+1+wℓ+1(ℓ)),i=1,…,Kℓ+1.\tilde{z}^{(\ell+1)}_{i}=E^{(\ell+1)}\!\left(\hat{z}^{(\ell)}_{is_{\ell+1}\,:\,is_{\ell+1}+w_{\ell+1}}\right),\qquad i=1,\ldots,K_{\ell+1}. (5)
a0:Hℓ−1(ℓ),∗=argmina(ℓ)0:Hℓ−1[‖z~Kℓ+1(ℓ+1)−z^Kℓ+1(ℓ+1),∗‖22+β∑i=1Kℓ+1−1‖z~i(ℓ+1)−z^i(ℓ+1),∗‖22],a^{(\ell),*}_{0:H_{\ell}-1}=\arg\min_{a^{(\ell)}_{0:H_{\ell}-1}}\left[\left\|\tilde{z}^{(\ell+1)}_{K_{\ell+1}}-\hat{z}^{(\ell+1),*}_{K_{\ell+1}}\right\|_{2}^{2}+\beta\sum_{i=1}^{K_{\ell+1}-1}\left\|\tilde{z}^{(\ell+1)}_{i}-\hat{z}^{(\ell+1),*}_{i}\right\|_{2}^{2}\right], (6)

3 Learning Hierarchical Abstractions

Probes compare what each frozen level retains about navigation and manipulation entities. Higher levels selectively lose fast body or distractor information when data separates those dynamics from slower task-relevant motion. That pattern is weaker when all entities evolve on similar timescales. Selective abstraction is therefore a property of the learned objective and training data, not an automatic consequence of adding layers.

Refer to caption
Figure 4: Selective abstraction across hierarchy levels. Left: MLP-probe NMSE (mean ±\pm SE over three training seeds) at successive levels of four-level H-JEPAs on Visual AntMaze and FourRoomDistractors. Body state or distractor position becomes less recoverable at higher levels while agent position remains recoverable. Right: one observation decoded from levels 11–33; decoders serve only for visualization. AntMaze’s leg pose becomes less distinct; FourRoom’s blue distractor fades while the red agent remains visible.
Original figure
Figure 5: Larger temporal-frequency gaps correlate with stronger selective abstraction. The x-axis measures temporal separation in each training dataset: the ratio of the characteristic frequencies (spectral centroids) of the fastest- and slowest-varying state components, shown on a log scale. The y-axis shows the change in per-entity probing NMSE from level 1 to level 2; positive values indicate poorer recoverability at level 2. Across environments, larger frequency gaps coincide with greater increases in fast-entity probing error (filled markers), while slow-entity error remains near its level-1 value (open markers). Environments with gaps below 2×2\times (shaded region) show little selective abstraction.

4 Planning with Hierarchical Abstractions

The evaluation compares hierarchy depth and compute, isolates goal-cost geometry, and then tests diverse real-video scenes. These stages distinguish simulation success from offline imitation-of-expert-path fidelity.

Original figure
Figure 6: Gradient-based planning across compute budgets and hierarchy depths, comparing LeWM [53], H-JEPA, and the hierarchical world model baseline HWM [92]. Error bars in both panels show one SE over three seeds. Top: success versus planner FLOPs per episode, sweeping action-sample counts with other planner settings fixed. FLOPs include forward and backward passes. Bold curves are Pareto fronts. Bottom: success versus hierarchy depth on a separate set of 5050 tasks per environment. Planner settings are selected under a 100100-TFLOP planning budget. See §E for protocols and Fig. 9 for the DROID results.

4.1 Planning performance across compute budgets and depths

Additional levels improve the compute–success frontier on several tasks, but depth is not monotonically helpful. Three-level Push-T performs poorly, plausibly because short episodes leave fewer usable training clips for the larger temporal window. Comparisons with HWM test whether distinct state representations help beyond temporal decomposition alone.

4.2 How hierarchy improves planning

With rollout dynamics fixed, scoring plans in a higher latent space can improve AntMaze navigation, where low-level distances are nearly flat away from the goal. Hierarchical planning also divides a long route into easier subproblems. Cost-only controls show that representation geometry and temporal decomposition contribute separately, with environment-dependent benefits.

Model depth
Cost / planner 2 levels ↑\uparrow 3 levels ↑\uparrow 4 levels ↑\uparrow
Native L1 23.3±3.323.3\pm 3.3 16.7±1.816.7\pm 1.8 10.0±0.010.0\pm 0.0
L2 projection 31.3±2.731.3\pm 2.7 20.7±4.820.7\pm 4.8 20.7±1.820.7\pm 1.8
L3 projection – 22.0±2.322.0\pm 2.3 21.3±4.721.3\pm 4.7
L4 projection – – 26.7±2.926.7\pm 2.9
Hierarchical 39.3±3.739.3\pm 3.7 73.3±3.573.3\pm 3.5 63.3±7.763.3\pm 7.7

Table 1: Changing only the cost on AntMaze. Success (%, mean ±\pm SE, three seeds). Within each column, the first four rows share level-1 dynamics and planner settings; only the cost space changes. Hierarchical planning is a separate reference.
Refer to caption
Figure 7: Planning cost geometry on AntMaze. Distance to anchor ⋆\star in each latent space of a three-level H-JEPA (blue near, red far; one training seed). Higher levels show more graded costs along corridors and a broader low-cost region around the anchor.
Original figure
Figure 8: Temporal decomposition of planning costs. We analyze a two-level H-JEPA with s2=3s_{2}=3. Top: Goal cost in level-1 and level-2 space. For a held-out expert trajectory we encode each frame at both levels, and compute the distance between each timestep’s latent zt(ℓ)z^{(\ell)}_{t} and the goal (final-timestep) latent, ‖zt(ℓ)−g(ℓ)‖22\|z^{(\ell)}_{t}-g^{(\ell)}\|_{2}^{2}. At level 2 we subsample at the training stride. Bottom: Subgoal tracking cost. The level-2 subgoals are the level-2 latents at the subsampled timesteps. Within each subgoal segment we re-anchor the level-1 rollout from the encoded latent at the segment start, unroll it to the next subgoal, project it into level-2 space, and measure its distance to that subgoal. Within each segment, the cost decreases monotonically toward the next subgoal.

4.3 Scaling to diverse scenes: DROID

In diverse scenes, predictable background can dominate a prediction objective while action-relevant robot motion disappears. The paper adds inverse dynamics to make consecutive state representations informative about the action connecting them.

Environments and evaluation.

DROID plans are scored offline against expert end-effector paths using normalized discrete Fréchet distance. Zero corresponds to doing nothing, the expert path to 100%, and worse-than-static plans can score negative. This is a useful trajectory-quality proxy, not a direct manipulation success test.

Equation 7
fidelity=dF​(𝟎,p~)−dF​(p,p~)dF​(𝟎,p~).\text{fidelity}=\frac{d_{F}(\mathbf{0},\tilde{p})-d_{F}(p,\tilde{p})}{d_{F}(\mathbf{0},\tilde{p})}. (7)

Inverse dynamics loss avoids slow-feature collapse.

The inverse-dynamics head reconstructs actions from consecutive latent states. This constrains a joint temporal relationship that marginal Gaussian regularization cannot enforce. At higher levels, target macro-actions are detached so the encoder cannot satisfy reconstruction merely by co-adapting that target.

Equation 8
ℒidm(ℓ)=1T−1​∑t=1T−11da​‖I(ℓ)​(zt(ℓ),zt+1(ℓ))−sg⁡[at(ℓ)]‖22,\mathcal{L}^{(\ell)}_{\mathrm{idm}}=\frac{1}{T-1}\sum_{t=1}^{T-1}\frac{1}{d_{a}}\left\|I^{(\ell)}\!\left(z^{(\ell)}_{t},\,z^{(\ell)}_{t+1}\right)-\operatorname{sg}\!\left[a^{(\ell)}_{t}\right]\right\|_{2}^{2}, (8)

Fréchet fidelity (%) →\rightarrow Original figure(b)                Original figure(c)               

Figure 9: Scene diversity makes the inverse-dynamics loss necessary, and a second level adds performance at a lower planner budget. (a) First frames of three Cube and three DROID clips, showing scene variation across episodes. (b) DROID at 55 fps, Fréchet fidelity (%), showing the slow-feature-collapsed LeWM, LeWM+IDM, then two hierarchies: HWM, whose level 2 is an identity encoder, and H-JEPA, whose level 2 is learned. (c) The one-level and two-level models of (b) swept over planner budgets, Fréchet fidelity against measured planner TFLOPs per episode on a log axis, the bold line being the Pareto front. Two frozen-encoder world models are shown for reference: they are public checkpoints, evaluated at their native 11 fps on the 1515 clips whose window fits at that rate, so their intervals span three planner seeds. All other intervals are SE over three train seeds.

Refer to caption(a)                                         

Hierarchical planning results.

SIGReg restores representation diversity, while inverse dynamics makes that diversity relevant to robot motion. With both present, a second hierarchy level improves the reported offline fidelity at lower planning budgets. The result does not establish successful closed-loop real-robot execution.

5 Related Work

H-JEPA builds on reward-free action-conditioned JEPAs, LeJEPA’s regularizer, and hierarchical planning. Its distinguishing combination is end-to-end prediction across several learned state spaces. It does not introduce temporal hierarchy, latent dynamics or inverse dynamics in isolation.

6 Conclusion

Separate timescales and representations can jointly support efficient planning. The demonstrated real-video extension depends on action supervision and an offline metric. Closed-loop physical-robot evaluation, action-free training, language goals and variable-duration abstractions remain future work.

Acknowledgments

The acknowledgments credit data, infrastructure and research contributions. They do not add an evaluation result or broaden the scope of the experiments.

Appendix A Notation

The full notation table distinguishes hierarchy levels, temporal strides, window sizes, action blocks and planning horizons. These conventions matter because an upper-level step can cover several lower-level transitions.

Table 2: Notation reference. Symbols are introduced in §2 unless the row names another section. Level superscripts are always parenthesized, (ℓ)(\ell); a bare subscript is a time index and a:ba{:}b is a half-open range. Appendix-local symbols are defined where they are used.
Symbol Description
Levels and indices
ℓ\ell, LL Hierarchy level index, ℓ=1,…,L\ell=1,\ldots,L; level 11 is the finest
tt Discrete time index, at the level’s own rate
a:ba{:}b Half-open slice: the b−ab{-}a indices a,…,b−1a,\ldots,b{-}1
Observations, latents and actions
oto_{t} Image observation at time tt
ata_{t} Primitive action block for the transition ot→ot+1o_{t}\to o_{t+1}
zt(ℓ)z^{(\ell)}_{t} Level-ℓ\ell latent state; zt(1)=E(1)​(ot)z^{(1)}_{t}=E^{(1)}(o_{t})
z^(ℓ)1:Hℓ\hat{z}^{(\ell)}_{1:H_{\ell}} Action-conditioned autoregressive rollout at level ℓ\ell
z^(ℓ+1),∗\hat{z}^{(\ell+1),*} Optimized rollout produced by the level above; ∗* marks an optimized quantity
at(ℓ)a^{(\ell)}_{t} Level-ℓ\ell action embedding (a macro-action for ℓ>1\ell>1)
a(ℓ),∗0:Hℓ−1a^{(\ell),*}_{0:H_{\ell}-1} Optimized action sequence returned by the level-ℓ\ell planner
g(ℓ)g^{(\ell)} Goal state at level ℓ\ell, obtained by encoding the goal observation
Z(ℓ)Z^{(\ell)} Minibatch of level-ℓ\ell embeddings, Z(ℓ)∈ℝn×DZ^{(\ell)}\in\mathbb{R}^{n\times D}: nn rows of width DD
NN, TT Sequences and timesteps of a minibatch of embeddings
DD, WW Embedding width, and the number of data-parallel ranks (§K.3)
Learned components
E(ℓ)E^{(\ell)} State encoder at level ℓ\ell; pools a wℓw_{\ell}-step lower-level window
A(ℓ)A^{(\ell)} Action encoder at level ℓ\ell; pools an sℓs_{\ell}-step lower-level window for ℓ>1\ell>1
F(ℓ)F^{(\ell)}, I(ℓ)I^{(\ell)} Latent predictor and inverse-dynamics head at level ℓ\ell
Roll(ℓ)\mathrm{Roll}^{(\ell)} Autoregressive rollout operator induced by F(ℓ)F^{(\ell)}
Training objective
ℒ(ℓ)\mathcal{L}^{(\ell)}, ℒpred(ℓ)\mathcal{L}^{(\ell)}_{\mathrm{pred}}, ℒidm(ℓ)\mathcal{L}^{(\ell)}_{\mathrm{idm}} Total level-ℓ\ell objective and its prediction and inverse-dynamics terms
SIGReg⁡(Z(ℓ))\mathrm{SIGReg}(Z^{(\ell)}) Sketched normality (anti-collapse) regularizer
umu_{m}, hmh_{m} Unit direction um∈𝕊D−1u_{m}\in\mathbb{S}^{D-1} and its projection hm=Z(ℓ)​umh_{m}=Z^{(\ell)}u_{m} (§K)
MM Number of random directions drawn by SIGReg (§K)
𝒯EP\mathcal{T}_{\mathrm{EP}} Epps–Pulley normality statistic (§K)
Planning
HℓH_{\ell} Planning horizon at level ℓ\ell, in level-ℓ\ell steps
C(ℓ)​(z,z′)C^{(\ell)}(z,z^{\prime}) Pairwise cost paid by the level-ℓ\ell planner; squared Euclidean distance ‖z−z′‖22\|z-z^{\prime}\|_{2}^{2} within one latent space
z~i(ℓ+1)\tilde{z}^{(\ell+1)}_{i} Level-ℓ\ell rollout lifted into level-(ℓ+1)(\ell{+}1) space by E(ℓ+1)E^{(\ell+1)}
Kℓ+1K_{\ell+1} Number of subgoal terms in the level-ℓ\ell cost; pinned by the lift, Kℓ+1=⌊Hℓ/sℓ+1⌋≤Hℓ+1K_{\ell+1}=\lfloor H_{\ell}/s_{\ell+1}\rfloor\leq H_{\ell+1}
gi(ℓ+1)g^{(\ell+1)}_{i} The ii-th subgoal handed to level ℓ\ell; the last is anchored, gKℓ+1(ℓ+1)=g(ℓ+1)g^{(\ell+1)}_{K_{\ell+1}}=g^{(\ell+1)}
β\beta Weight of one subgoal term relative to the terminal term, Eq. (6)
tjt_{j}, ξj\xi_{j} Replanning times, and primitive commands committed at phase jj (the commit schedule, §E)
SS, qq, RR Planner candidates per level, scored final steps of a rollout, and level-1 blocks executed per replan (§E)
Architecture and loss hyperparameters
sℓs_{\ell}, wℓw_{\ell} Temporal stride and window size from level ℓ−1\ell{-}1 to level ℓ\ell: each level-ℓ\ell state spans wℓw_{\ell} lower-level states, taken every sℓs_{\ell}
cℓc_{\ell} Context length, in level-ℓ\ell latent states
λℓ\lambda_{\ell}, γℓ\gamma_{\ell} SIGReg and inverse-dynamics weights at level ℓ\ell

Appendix B Environments and datasets

FourRoomDistractors, AntMaze, Push-T, Cube and DROID differ in motion timescales, episode duration, scene diversity and available proprioception. These differences explain why one depth or objective does not behave identically across domains.

Appendix C Architecture and training

Architecture, optimization and dataset settings are separated between single-scene simulation and diverse DROID video. The distinction is central to interpreting the need for inverse dynamics.

C.1 Single-scene environments

The simulated hierarchies are trained jointly from scratch, so upper-level prediction losses can influence lower-level encoders. The flat LeWorldModel baseline uses the corresponding single-level recipe.

Data.

All compared depths use the same source data, but deeper hierarchies require longer within-episode clips. This reduces the effective transition coverage per epoch, especially on short Push-T episodes.

Table 3: Temporal layout of one training sample and the clips it leaves per epoch. The minimum clip length is the shortest episode segment that yields one training sample, in environment steps, including both endpoint observations. All H-JEPA models use stride two and window one at every level above the first. A clip must lie within one episode (the loader reserves 5​n15\,n_{1} environment steps per clip), so the number of clips an epoch yields falls as the minimum clip length grows; parentheses give each count as a percentage of the LeWM count in the same environment. Each clip supplies three teacher-forced transitions per level.
LeWM Two-level H-JEPA Three-level H-JEPA Four-level H-JEPA
Level-1 frames per clip 44 77 1313 2525
Minimum clip length (env. steps) 1616 3131 6161 121121
Training clips per epoch
FourRoom 952,708952{,}708 915,298915{,}298 (96%96\%) 840,478840{,}478 (88%88\%) 690,838690{,}838 (73%73\%)
AntMaze 9,575,0009{,}575{,}000 9,200,0009{,}200{,}000 (96%96\%) 8,450,0008{,}450{,}000 (88%88\%) 6,950,0006{,}950{,}000 (73%73\%)
Push-T 1,981,7211{,}981{,}721 1,701,4461{,}701{,}446 (86%86\%) 1,143,6231{,}143{,}623 (58%58\%) 285,931285{,}931 (14%14\%)
Cube 1,783,6001{,}783{,}600 1,636,6001{,}636{,}600 (92%92\%) 1,342,6001{,}342{,}600 (75%75\%) 754,600754{,}600 (42%42\%)

Training dynamics.

Early anti-collapse regularization spreads concentrated embeddings, so prediction loss can initially rise before declining. A near-zero prediction loss is not automatically desirable because a constant representation is trivial to predict. Both loss components and downstream probes are retained.

Original figure
Original figure
Figure 10: Per-level training losses of the four-level H-JEPA models and of the DROID models. Top three rows: unweighted prediction loss and SIGReg statistic of each level of the four-level models over training, and their total training objective, the sum over levels of the prediction loss plus the weighted SIGReg term. Every level of a model uses the same SIGReg weight (0.180.18 on FourRoom, 0.090.09 elsewhere), so the weighted curves differ from the unweighted ones by a constant factor per environment. The seed band of the total objective is narrower than its line. Bottom row: unweighted prediction, SIGReg and inverse-dynamics losses of the DROID flat LeWM+IDM (orange, dashed) and of both levels of the two-level H-JEPA (solid) over their 100100 training epochs, and the total training objective, the sum of all terms weighted as in Table 5 (level 1: SIGReg 0.080.08, IDM 100100; level 2: SIGReg 0.160.16, IDM 5050), summed over both levels for H-JEPA. All curves are the mean over three training seeds with a min/max band, on a log scale.

Architecture.

A projected vision-transformer token supplies the first representation, with added proprioceptive features where specified. Upper levels use compact encoders and action-conditioned causal predictors. Their distinct latent spaces are learned rather than manually assigned physical variables.

Training.

The recipe uses prediction and SIGReg losses at every level, finite random projections, AdamW and three training seeds. Equal epoch budgets do not imply equal usable data when temporal clip requirements differ.

Table 4: Per-environment architecture and training settings. Upper-level predictor dimensions and H-JEPA epoch counts apply to every depth. H-JEPA models are initialized independently of the LeWM baselines.
FourRoom AntMaze Push-T Cube
Frame size / patch 224224 / 1414 6464 / 44 224224 / 1414 224224 / 1414
Command / proprio dim. 22 / n/a 88 / 2727 22 / n/a 55 / n/a
Latent dim. dd 192192 384384 192192 192192
Upper predictor layers / heads 44 / 88 55 / 1212 66 / 1616 66 / 1616
Upper predictor width / FF width 192192 / 10241024 224224 / 15361536 192192 / 20482048 192192 / 20482048
SIGReg λ\lambda, LeWM / H-JEPA 0.720.72 / 0.180.18 0.090.09 / 0.090.09 0.090.09 / 0.090.09 0.090.09 / 0.090.09
H-JEPA epochs 77 55 99 88

Implementation and hardware.

The implementation uses the stable-worldmodel and stable-pretraining libraries on the specified GPU hardware. Hardware-dependent timing should be separated from the reported planning-compute measure.

C.2 DROID

DROID training uses pixels and actions without proprioception and modifies data sampling and level weighting. Upper-level action embeddings also receive regularization to avoid collapsing macro-actions. These changes are part of the diverse-scene recipe, not an unchanged simulation model applied directly to real video.

Table 5: Training hyperparameters on DROID. Predictors are causal transformers, given as width/depth. The grouping row gives the shape of the tensor SIGReg tests: (T,N,D)(T,N,D) tests normality within each timestep, over the NN sequences of the batch. The architecture and loss settings do not depend on the training schedule, whether end-to-end or stagewise (§C.2); the optimization block does, and reports the end-to-end headline runs. The peak learning rate is the same at both levels, AdamW runs at β=(0.9,0.999)\beta=(0.9,0.999), and the dataloader length caps the iterations per epoch, which gives 292292 at this batch size.
DROID
Global
Stop-gradient on latent targets None
Level-1 to level-2 stride 33
Level 1
Encoder ViT-S/16, CLS
Predictor 384384/1212
Predictor context c1c_{1} 77 states
SIGReg grouping (T,N,D)(T,N,D)
SIGReg weight λ1\lambda_{1} 0.080.08
IDM weight γ1\gamma_{1} 100100
Level 2
Encoder latent MLP 768768
Predictor 384384/1212
Predictor context c2c_{2} 22 states
SIGReg grouping (T,N,D)(T,N,D)
SIGReg weight λ2\lambda_{2} 0.160.16
IDM weight γ2\gamma_{2} 5050
Action SIGReg weight 0.0050.005
Optimization
Optimizer AdamW
Total batch size 256256
Peak learning rate 5×10−45\times 10^{-4}
Warm-up, fraction of steps 0.10.1, linear
Decay cosine to 00
Weight decay 10−410^{-4}
Gradient clipping, per level 1.01.0
Precision bfloat16
Epochs 100100
Iterations per epoch 292292
Training seeds 11, 10001000, 1000010000

Appendix D Probing and decoding

World models are frozen before fitting probes and visualization decoders. These supervised diagnostic tools inspect learned information without supplying their targets to the world-model objective.

Probes.

Separate nonlinear probes predict each entity from each hierarchy level, reporting variance-normalized error. Low probe error measures recoverability under that probe family, not an identified causal factorization.

Decoder.

A lightweight decoder reconstructs images from a single latent for visualization. It does not turn the predictive training objective into pixel reconstruction.

Training.

Probe and decoder optimization is specified separately. Entity groups are classified by temporal frequency, so the analysis can distinguish retaining slow state from discarding faster detail rather than averaging all variables together.

Table 6: Probing data and probed entities per environment. Transition counts are for fitting (train) and scoring (eval). Visual AntMaze is scored on one evaluation set with equal numbers of explore and stitch episodes. On FourRoomDistractors and Visual AntMaze, whose state separates into slow and fast entities (Fig. 12), the entities are labeled accordingly. Entity dimensions in parentheses.
Environment transitions (train / eval) Probed entities
FourRoomDistractors 160160k / 4040k
slow: ego x​yxy (2)
fast: distractor x​yxy (2)
Visual AntMaze 500500k / 5050k
slow: global x​yxy (2)
fast: body state (27)
Push-T 250250k / 2.52.5k
agent x​yxy (2), block x​yxy (2),
block orientation (2), agent velocity (2)
OGBench Cube 250250k / 4040k
cube position (3), cube orientation (6),
effector position (3), effector yaw (2), gripper (2),
arm joint positions (6), arm joint velocities (6)

Appendix E Planning evaluation

Evaluation tasks.

Held-out task pairs require navigation or manipulation over specified distances and horizons. Goals are images and, on AntMaze, accompanying proprioception. These controlled goal specifications differ from following free-form instructions.

Refer to caption
Figure 11: Successful hierarchical planner executions. Selected H-JEPA executions on one evaluation task per environment, using three-level models for FourRoom and AntMaze and two-level models for Cube and Push-T. These checkpoints also appear in Fig. 6 (bottom); the recordings use their original evaluation settings rather than the compute-selected sample counts. Each row shows eight observations sampled uniformly in environment steps, followed by the goal observation supplied to the planner. Rows end at first recorded success, except Cube, which shows the full 3030-step execution to include the completed lift. Labels give elapsed environment steps.

Shared solver.

Each active level differentiates through candidate latent rollouts. Upper-level action search is calibrated to macro-actions observed during training because those latent actions lack native physical bounds. This anchoring limits unrealistic candidates but does not prove complete feasibility.

Replanning and horizons.

Models execute a fixed initial action chunk before replanning, with level-specific horizons that decrease during an episode. This keeps the comparison tied to the stated controller rather than assuming an open-loop plan is executed unchanged.

Reported settings.

Candidate budgets are selected on compute-sweep tasks and then evaluated on a separate task set. Depth comparisons vary candidate allocation while retaining the chosen learning rates and horizon settings.

Table 7: Planning settings for the simulated depth comparison. SS: candidate sequences; η\eta: AdamW learning rate; HH: configured horizon in predicted steps of that level, used by the schedule described above. Tuples list levels from lowest to highest, except the H-JEPA learning-rate row, which lists level one and the rate shared by all upper levels. Episode budgets are in environment steps. Cube’s horizon-one upper solvers are skipped, so their listed candidate counts are unused.
FourRoom AntMaze Push-T Cube
Episode budget 210210 350350 9090 3030
η\eta, LeWM 0.10.1 0.30.3 0.050.05 0.050.05
η\eta, H-JEPA (L1, upper) (0.08,0.1)(0.08,0.1) (0.06,0.1)(0.06,0.1) (0.1,0.075)(0.1,0.075) (0.1,0.075)(0.1,0.075)
HH, LeWM 2323 2323 1515 44
HH, two-level H-JEPA (2,12)(2,12) (4,9)(4,9) (2,8)(2,8) (2,2)(2,2)
HH, three-level H-JEPA (2,2,6)(2,2,6) (4,2,5)(4,2,5) (2,2,4)(2,2,4) (2,2,1)(2,2,1)
HH, four-level H-JEPA (2,2,2,3)(2,2,2,3) (3,2,2,3)(3,2,2,3) (2,2,2,2)(2,2,2,2) (2,2,1,1)(2,2,1,1)
SS, LeWM 200200 100100 800800 800800
SS, two-level H-JEPA (100,800)(100,800) (25,200)(25,200) (50,400)(50,400) (400,100)(400,100)
SS, three-level H-JEPA (50,200,200)(50,200,200) (100,100,400)(100,100,400) (25,50,200)(25,50,200) (25,800,800)(25,800,800)
SS, four-level H-JEPA (50,200,200,400)(50,200,200,400) (50,200,25,50)(50,200,25,50) (200,800,800,800)(200,800,800,800) (200,50,800,800)(200,50,800,800)
Table 8: Planning settings on DROID. SS: candidate sequences; η\eta: AdamW learning rate; HH: configured horizon in predicted steps of that level. Tuples list levels from lowest to highest. Every level runs AdamW for at most 9090 iterations with early stopping when the best cost stalls; the cross-level action cost is 00 and the subgoal weight β\beta of Eq. (6) is 0.50.5, for both two-level arms. Planning is open-loop: the whole level-1 plan is executed, so the commit width equals the level-1 horizon. Each model keeps one η\eta and these settings at every planner budget: the compute-Pareto of Fig. 9 varies only the number of candidates, chosen independently per level for the two-level arms. The listed SS is the operating point of each bar in Fig. 9.
DROID, 55 fps
Episode budget 3636
η\eta, LeWM+IDM 0.010.01
η\eta, HWM (L1, L2) (0.01,0.3)(0.01,0.3)
η\eta, H-JEPA (L1, L2) (0.01,0.3)(0.01,0.3)
HH, LeWM+IDM 3636
HH, HWM (36,12)(36,12)
HH, H-JEPA (36,12)(36,12)
SS, LeWM+IDM 3232
SS, HWM (16,16)(16,16)
SS, H-JEPA (16,16)(16,16)

Appendix F Temporal-frequency separation and selective abstraction

F.1 Frequency separation within a trajectory

An example AntMaze trajectory displays slow global motion and faster body dynamics at the sampling rate used for the spectral analysis. It illustrates the timescale statistic without replacing the aggregate multi-seed probe results.

Original figure(a)
Refer to caption(b)
Figure 12: Temporal-frequency separation and selective abstraction. (a) One AntMaze episode: normalized global position (red) and 1313 body-state dimensions (blue) over 8181 level-1 frames (left), and cumulative absolute normalized increments averaged across dimensions (right). Inverse spectral centroids are about 3535 frames for position and 44 for body state (8.3×8.3\times gap); body state accumulates roughly ten times more variation. Including the omitted velocity dimensions leaves the frequency gap unchanged. (b) Per-entity spectral centroids in cycles per level-1 frame (top, log scale) and paired level-1 and level-2 probe macro NMSE (bottom). The left group shows FourRoom, AntMaze and Humanoid; the right shows Push-T, OGBench Cube and DROID. Probe heads are matched across levels within each environment and differ across environments. Error bars are one SE over three training seeds. All entities in an environment share the same level-2 configuration.

F.2 Per-entity probes across environments

Per-entity spectral centroids are paired with probe errors across environments. Larger temporal gaps coincide with selective loss of fast information at the next level; tasks without clear separation retain more entities together.

Appendix G Data specialization across levels

Stagewise training lets each hierarchy level use a different dataset, testing whether temporal data structure influences abstraction. This changes the training strategy as well as the data assigned to each level.

Refer to caption
Refer to caption
Refer to caption
Figure 13: Planning with different training datasets on Visual AntMaze. Planning success rate (GD online, mean over three seeds), using explore, stitch, or their union for training. Left: flat level-1 planning, with one model trained on each of the three datasets. Middle: two-level hierarchical planning after stagewise training, covering all nine combinations of level-1 (rows) and level-2 (columns) datasets. Right: two-level hierarchical planning after end-to-end training, with both levels trained on the same dataset in each run, giving only diagonal entries. Grey cells were not run. The end-to-end runs are FLOP-matched to the stagewise ones.

Experimental setup.

AntMaze exploratory, stitched and combined trajectories are compared under stagewise and joint training. The controlled combinations distinguish a richer lower-level transition set from more coherent higher-level temporal patterns.

Matched data: a second level helps only when the data is temporally structured.

A second level helps more with temporally structured stitched trajectories than purely exploratory data in these experiments. Hierarchy cannot be assumed to create useful long-term abstraction from any arbitrary transition collection.

Appendix H Planning-cost ablations across environments

Cost-only ablations keep low-level dynamics fixed while changing the latent space used to score final states. This isolates representation geometry from splitting the temporal search.

FourRoom: gains from projected costs.

At least one upper-level cost improves FourRoom navigation, though the deepest cost is not always best. Overlapping error bars qualify small differences between projected costs.

Cube and Push-T: no clear gain from projected costs.

Cube and Push-T show no clear general benefit from upper-level cost projection, and some deep projections hurt. Their weak selective abstraction limits transferring the AntMaze cost-geometry explanation unchanged.

Table 9: Planning-cost ablations across environments. Success (%, mean ±\pm SE, three seeds), using the same protocol as Table 1. Within each H-JEPA row, only the cost space varies across the four level-1 planning columns; LeWM and hierarchical H-JEPA are references. For FourRoom, bold marks the highest mean among cost spaces. FourRoom gains from projected costs, while Cube and Push-T show no clear benefit.
Level-1 planning: vary cost space Hierarchical planner
Environment Depth Native L1 ↑\uparrow L2 proj. ↑\uparrow L3 proj. ↑\uparrow L4 proj. ↑\uparrow H-JEPA ↑\uparrow
FourRoom Distractors LeWM (flat) 40.7±6.840.7\pm 6.8
2 levels 9.3±0.79.3\pm 0.7 38.7±6.638.7\pm 6.6 81.3±6.781.3\pm 6.7
3 levels 43.3±8.743.3\pm 8.7 65.3±5.565.3\pm 5.5 71.3±4.171.3\pm 4.1 96.0±1.196.0\pm 1.1
4 levels 68.7±4.768.7\pm 4.7 72.0±3.172.0\pm 3.1 66.0±3.166.0\pm 3.1 70.7±2.770.7\pm 2.7 80.0±4.080.0\pm 4.0
OGBench Cube LeWM (flat) 32.0±6.132.0\pm 6.1
2 levels 24.0±1.224.0\pm 1.2 13.3±0.713.3\pm 0.7 47.3±2.747.3\pm 2.7
3 levels 24.7±1.324.7\pm 1.3 28.0±4.228.0\pm 4.2 14.0±3.514.0\pm 3.5 60.0±5.060.0\pm 5.0
4 levels 26.7±2.726.7\pm 2.7 25.3±2.425.3\pm 2.4 12.0±2.012.0\pm 2.0 12.0±2.312.0\pm 2.3 55.3±4.455.3\pm 4.4
Push-T LeWM (flat) 40.0±2.340.0\pm 2.3
2 levels 36.7±1.836.7\pm 1.8 38.0±3.138.0\pm 3.1 45.3±1.345.3\pm 1.3
3 levels 30.7±1.830.7\pm 1.8 28.0±4.028.0\pm 4.0 24.7±4.124.7\pm 4.1 17.3±3.517.3\pm 3.5
4 levels 1.3±0.71.3\pm 0.7 1.3±0.71.3\pm 0.7 0.7±0.70.7\pm 0.7 1.3±1.31.3\pm 1.3 0.7±0.70.7\pm 0.7

Appendix I Prediction-cost monotonicity

Rank correlation compares latent-goal distance with progress along trajectories. It is a diagnostic of cost monotonicity in the chosen data, not a guarantee that every gradient direction leads to a feasible route.

Table 10: Prediction-cost monotonicity (−Spearman-\mathrm{Spearman} of ‖z(ℓ)−g(ℓ)‖22\|z^{(\ell)}-g^{(\ell)}\|_{2}^{2} against time; higher is better; mean over three training seeds, held-out episodes). L2 columns score each model’s level-2 encoding against the final goal; subgoal scores a level-1 rollout, re-anchored to the true state at each subgoal (bounded horizon), against the level-2 subgoals (per-segment).
HWM H-JEPA
Environment Level-1 ↑\uparrow Level-2 ↑\uparrow subgoal ↑\uparrow Level-2 ↑\uparrow subgoal ↑\uparrow
Visual AntMaze 0.48 0.52 0.94 0.68 0.94
Push-T 0.75 0.77 0.85 0.77 0.89
OGBench Cube 0.65 0.74 0.79 1.00 0.99
FourRoom Distractors 0.56 0.62 0.92 0.85 1.00

Appendix J DROID: corpus, planning evaluation and offline metric

The appendix details real-video data selection and calibrates the offline trajectory metric against simulated task outcomes. Both are needed to bound the interpretation of DROID results.

J.1 Planning-evaluation protocol

Clips.

Evaluation clips are chosen to require non-greedy manipulation rather than trivial endpoint motion. Their selection and held-out separation are specified because a small curated set cannot support a broad deployment claim.

Frame rates.

Frame-rate choices define action blocks and the time span of each prediction. Results at one sampling rate should not be equated with the same model at another physical timescale.

Leak control.

The protocol separates evaluation data and training clips as described. This controls the reported test rather than establishing absence of every possible pretraining overlap.

Task and action space.

Planning acts in the end-effector delta-pose space and compares with recorded expert paths. The action convention determines what the offline fidelity score measures.

J.2 The Fréchet fidelity offline metric

Fréchet fidelity normalizes path discrepancy by a do-nothing baseline. Calibration shows that an expert-like path can be necessary for success while still failing to complete a task, and can rank partial progress more smoothly than binary outcomes.

Against a binary outcome it is a necessary condition.

Following much of the expert path does not ensure final completion. A high offline score can therefore be a necessary but insufficient indicator under the tested tasks.

Against a continuous outcome it ranks progress.

Against a continuous progress measure, the metric can order partial improvements. Its calibration is empirical and tied to the selected simulation and trajectory conventions.

Original figure
Figure 14: On OGBench Cube, where the outcome is observable, Fréchet fidelity separates the episodes that can succeed from those that cannot, but not the ones that do. One frozen checkpoint, 99 configuration groups ×\times 33 planner seeds ×\times 5050 episodes. (a) One point per planner configuration; in the shaded band the flat and hierarchical planners have equal offline fidelity and opposite grasp outcomes. (b) Every episode: none succeeds below a fidelity threshold, and most above it still fail.
Original figure
Figure 15: Against a continuous outcome the same offline metric ranks degree of progress. RoboCasa CloseDrawer, one frozen checkpoint, 9999 planner configurations ×\times 3232 held-out episodes, budget 200200 steps; the configurations are varied to spread the outcomes, not tuned for task performance. xx is Fréchet fidelity, computed exactly as on DROID; yy is graded drawer-closure success, a continuous outcome DROID cannot report. (a) One point per planner configuration, median fidelity against mean graded success. (b) Every episode, with the mean of each fidelity decile joined. The claim rests on the within-configuration rank correlation, which holds episode difficulty and planner settings fixed; the pooled and between-configuration values are inflated by a between-configuration mean shift.

J.3 Decoded plans

Decoded plans expose motion and errors hidden by one scalar fidelity score. They visualize imagined trajectories under planned actions, not frames obtained from executing those actions on a physical robot.

Refer to caption
Figure 16: What the DROID planner imagines, on every clip of the evaluation set. Two-level stagewise model, held-out, training seed 1; all 1616 clips ordered best to worst by Fréchet fidelity, down the left column then the right, +75%+75\% down to +0%+0\% at the zero-action floor. Each clip is two strips: above, the logged expert clip; below, the decoded plan, which is the level-1 rollout under the planner’s committed commands with the level-2 macro-plan interleaved into it at every third column (temporal stride 33), ringed in blue. Columns are timesteps. The leftmost is the context frame the planner is given (green); the rest are imagined (red), and blur along the rollout, which is the decoder’s fidelity limit and not a planning failure.

J.4 The re-encoded training corpus

A re-encoded DROID copy reduces decoding and storage overhead while retaining frames and timestamps. The documented preprocessing is part of reproducing the training data; it should not be confused with a new model contribution.

J.5 Future work

Joint multi-camera encoding, wrist views and action-free training remain open extensions. Removing action labels would also remove the inverse-dynamics signal used to prevent the demonstrated diverse-scene failure.

Appendix K Anti-collapse: SIGReg, inverse dynamics and detachment

SIGReg tests the marginal distribution of representations; inverse dynamics tests temporal action information. Target detachment changes gradient flow. Their roles are separated because a high-rank distribution alone need not represent controllable motion.

Equation 9
SIGReg⁡(Z(ℓ))=1M​∑m=1M𝒯EP​(hm),\mathrm{SIGReg}\!\left(Z^{(\ell)}\right)=\frac{1}{M}\sum_{m=1}^{M}\mathcal{T}_{\mathrm{EP}}(h_{m}), (9)

K.1 What SIGReg, IDM and detachment each contribute

The coefficient grid shows that SIGReg can maintain rank while planning remains poor without sufficient inverse-dynamics weight. Detachment is not a general replacement for either information diversity or action relevance.

Original figure
Figure 17: SIGReg restores representation rank; the inverse-dynamics term is what turns rank into planning fidelity. Full grid around the level-1 DROID configuration of table 5: SIGReg ∈{0,0.08}\in\{0,0.08\} ×\times IDM weight ∈{0,50,100,200}\in\{0,50,100,200\} ×\times prediction-target detachment, 1616 cells, one training seed each, read at epoch 100100. Five SIGReg-off cells (hatched) were stopped once collapsed and are read at their last epoch. Left: level-1 effective rank [25] of the CLS embedding, max 384384. Right: Fréchet fidelity on the 16-clip set; the grey band holds sub-floor cells, drawn as fixed-depth stubs annotated with their true value. Each planning bar is the mean over planner seeds 11, 22 and 33 of one trained model, and its error bar is the standard error (SE) over those three planner seeds.

K.2 Cross-validating the SIGReg and inverse-dynamics coefficients

The two loss coefficients are tuned jointly, and the next level is tuned after freezing the previous stage in this analysis. One coefficient’s effect cannot be interpreted independently of the other’s scale.

Refer to caption
Figure 18: The SIGReg and inverse-dynamics coefficients are chosen jointly, once per level. DROID at 55 fps; Fréchet fidelity (%, higher is better), mean and standard error over three training seeds, each averaged over three planner seeds. Left, the level-1 grid on single-level models; right, the level-2 grid on stagewise models with level 1 frozen. Both use the gradient-based planner of our headline results, and share one color scale. The heavy outline marks the coefficients we keep; at level 1 they lie within one standard error of the best cell.

K.3 SIGReg under data parallelism

The normality statistic is not a sum of per-example terms. The implementation aggregates empirical characteristic functions across devices before evaluating the statistic, maintaining consistency for a fixed global batch when the data is sharded differently.

Table 11: Our distributed SIGReg is exactly invariant to how a fixed total batch is sharded; a single-GPU implementation is not. One fixed Gaussian matrix (n=256n=256, D=32D=32, M=64M=64 directions) sharded contiguously across WW ranks in float64. The reference is the SIGReg released with LeWM [53], formula-identical to ours but with its prefactor set by the local batch and no collective; its code documents itself as single-GPU. The data follow the target, so its 14%14\% drift is launch topology, not signal.
𝒯EP\mathcal{T}_{\mathrm{EP}}
Shards ‖∇‖\|\nabla\| (ours) ours single-GPU reference
1×2561\times 256 0.3491541780120.349154178012 1.3726191785431.372619178543 1.3726191.372619
2×1282\times 128 0.3491541780120.349154178012 1.3726191785431.372619178543 1.1807701.180770
4×644\times 64 0.3491541780120.349154178012 1.3726191785431.372619178543 1.1862331.186233

Appendix L Slow-feature collapse in joint-embedding predictive objectives

The theory distinguishes avoiding a constant vector from avoiding a representation that is constant within each episode. Background-only embeddings can be diverse across episodes and perfectly predictable while useless for controlling the moving agent.

L.1 Three kinds of content in a sequence of observations

The analysis separates episode-constant background, action-driven foreground and other changing content. It defines slow-feature collapse through what the encoder retains within an episode, rather than through global covariance alone.

Equation 10
χ⁡(E)=𝔼b​[tr⁡Vart​(zt∣b)]tr⁡Varb,t⁡(zt),\chi(E)=\frac{\mathbb{E}_{b}\!\left[\operatorname{tr}\operatorname{Var}_{t}(z_{t}\mid b)\right]}{\operatorname{tr}\operatorname{Var}_{b,t}(z_{t})}, (10)

L.2 Where the latent variance goes

A variance decomposition compares within- and between-episode variation. Diverse scenes can consume most latent capacity in scene identity even though a same-episode difference cancels a constant offset. Cancellation alone does not restore the omitted robot information.

Original figure
Figure 19: Scene diversity sends latent variance between episodes. Within-episode (the slow-feature degree χ\chi of Definition 1) and between-episode shares of the latent variance. Bars are ±\pm1 standard error (SE) over three training seeds.

L.3 No regularizer on the marginal law can exclude it

With an atomless readable background and a regularizer depending only on the single-embedding marginal law, the construction can match the regularizer’s target while encoding only background. This is an existence result under explicit assumptions, not a statement that every trained model must reach that solution.

L.4 The objective selects by predictability, and the collapse is an optimum

The optimal predictor’s residual risk is bounded by unpredictable latent variation. Thus a perfectly predictable background representation can be globally attractive under the modeled objective. The characterization clarifies when preserving foreground introduces irreducible prediction cost.

Experiments of Sobal et al. [72] .

Earlier dot experiments are interpreted using the background/controlled/random taxonomy. These are supporting examples of the proposed mechanism, not independent proof that every nuisance variable behaves the same way.

L.5 Pooling time into the sample axis: a real but small barrier

Pooling time into the batch makes repeated within-episode embeddings visible to a finite-sample normality test. The resulting barrier can be real but small, so a nonzero SIGReg penalty alone does not reliably prevent background-only representations.

Equation 11
Δ=λ⁡(T−1)​κ\Delta=\lambda\,(T-1)\,\kappa (11)
Table 12: Training without the inverse-dynamics term: the SIGReg ladder (top) and time repulsion at its largest coefficient (middle). Level-1 models at 55 fps, γ=0\gamma=0, three train seeds, 100100 epochs. These runs feed SIGReg the flattened (N​T,D)(NT,D) tensor, one normality test over all N​TNT embeddings of the minibatch, which places time in the sample axis, so they carry the Δ\Delta of Proposition 2; the barrier is what the ladder probes. “Onset” is the first epoch at which Fréchet fidelity (Eq. (7)) drops clearly below the zero-action floor, the next column the epoch at which the predictor’s action sensitivity first falls below 0.050.05 (seed spread ±2\pm 2 epochs). Best fidelity is the highest before turning over; final fidelity and sensitivity are read over epochs 9898–100100; χ\chi is Eq. (10) on the final checkpoint (one seed for the repulsion runs, three for the IDM-on row). Ladder rows carry no terminal value (at most two seeds reach epoch 100100).
λ\lambda λsim\lambda_{\mathrm{sim}} Onset ↑\uparrow Sens. <0.05<0.05 ↑\uparrow Best fidelity ↑\uparrow Final fidelity ↑\uparrow Final sens. χ\chi
9.77×10−69.77\times 10^{-6} 00 77 1313 −0.64-0.64 (2.74) – – –
1.95×10−51.95\times 10^{-5} 00 2121 2020 3.353.35 (1.40) – – –
3.91×10−53.91\times 10^{-5} 00 2727 2626 0.630.63 (5.49) – – –
7.81×10−57.81\times 10^{-5} 00 3434 3939 5.865.86 (5.10) – – –
1.56×10−41.56\times 10^{-4} 00 4848 5757 8.388.38 (2.29) −81.03-81.03 0.0140.014 0.0110.011
1.56×10−41.56\times 10^{-4} −0.05-0.05 – none 6.466.46 2.44\mathbf{2.44} (5.80) 0.3010.301 (0.001) 0.170.17
1.56×10−41.56\times 10^{-4} −0.5-0.5 – none −8.05-8.05 −15.18-15.18 (17.39) 0.2100.210 (0.024) 0.840.84
1.56×10−41.56\times 10^{-4} −5.0-5.0 – none −3.26-3.26 −5.96-5.96 (2.35) 0.5290.529 (0.083) 0.920.92
IDM-on, γ=50\gamma=50 00 none none 30.66\mathbf{30.66} (1.05) 30.66\mathbf{30.66} (1.05) 0.2800.280 (0.016) 0.120.12

L.6 Time repulsion: a barrier that needs no action labels

Rewarding temporal change uses pairwise information and can oppose episode-constant collapse without action labels. It can also reward irrelevant fast distractors, so it is not interchangeable with a control-aware inverse-dynamics objective.

L.7 The inverse-dynamics term is a barrier of a different order

If representations reveal only background, action reconstruction retains a dataset-dependent error floor. This provides a qualitatively different barrier from marginal regularization, assuming actions are not already determined by background identity.

Equation 12
ℒidm≥1da​1T−1​∑t=1T−1(tr⁡Var⁡(at)−tr⁡Var⁡(𝔼⁡[at∣𝒞])),\mathcal{L}_{\mathrm{idm}}\;\geq\;\frac{1}{d_{a}}\,\frac{1}{T-1}\sum_{t=1}^{T-1}\left(\operatorname{tr}\operatorname{Var}(a_{t})-\operatorname{tr}\operatorname{Var}\!\left(\mathbb{E}[a_{t}\mid\mathcal{C}]\right)\right), (12)

Appendix M Extended related work

M.1 World models and planning

The extended survey places predictive latent models among generative simulators and model-based policy-learning methods. These approaches differ in rewards, reconstruction, pretraining and deployment-time search.

M.2 Preventing collapse in non-contrastive SSL

Contrastive negatives, architectural asymmetry, covariance regularization and distribution matching address constant collapse differently. The paper’s slow-feature analysis adds a separate issue: useful temporal content can disappear even when global distributional diversity survives.

M.3 Hierarchical World Models

Earlier hierarchical video and latent models use different timescales and sometimes reconstruction. H-JEPA combines a learned representation hierarchy with action-conditioned prediction and explicit planning, rather than claiming hierarchy itself is new.

M.4 Hierarchical control

Hierarchical policies and classical hierarchical model-predictive controllers organize decisions across timescales. H-JEPA learns predictive abstractions from offline observations and actions, then searches through them; that distinction limits comparisons with reward-trained policies and known-dynamics controllers.

Table 13: Model-based hierarchical control methods along four axes that define H-JEPA. Pixels: operates from high-dimensional visual observations rather than a low-dimensional state. Task-agnostic: reward-free training with no task signal; the task objective (a goal or a planning cost) is defined only at test time. Reconstruction Free: learns representations and dynamics with no reconstruction objective, predicting latent targets rather than decoding observations. Hierarchy of learned latent spaces: a stack of distinct latent spaces, each a coarser abstraction of the one below that discards detail (vs. a single/shared space, parallel spaces, or a hierarchy of decisions only). ✓ yes, ✗ no, ∼\sim partial. H-JEPA is the only method satisfying all four; HWM, our baseline, differs on exactly one axis.
Method
Pixels (In)
Task-Agnostic
World Models
Reconstruction
Free
Hierarchy of Learned
Latent Spaces
Classical hierarchical MPC [23, 51, 45] ✗ ✓ ✗ ✗
Director [32] ✓ ✗ ✗ ✗
Puppeteer [36] ✓ ✗ ✓ ✗
SPlaTES [28] ✗ ✗ ✓ ∼\sim
Schiewer et al. [69] ✗ ✗ ✗ ✓
THICK [27] ✓ ∼\sim ✗ ✗
GCP [65] ✓ ✓ ✗ ✗
HWM [92] ✓ ✓ ✓ ✗
H-JEPA (ours) ✓ ✓ ✓ ✓
Concept demonstration

Open image ↗