Literature
Display
← All papers

Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models

Vlad Sobal · Wancong Zhang · Kyunghyun Cho · Randall Balestriero · Tim G. J. Rudner · Yann LeCun

· Neural Information Processing Systems (NeurIPS 2025) ↗

World models11 min text · CondensedOriginal paper ↗
Contents

Overview

The authors provide a project page and public code. The study compares ways to learn reusable behavior from recorded trajectories without reward annotations.

At a glance

Intuition: a collection of short, imperfect trips may teach how movement works even when it never contains a complete route to the destination. PLDM learns an encoder and action-conditioned latent dynamics, then searches for actions at deployment. This study compares that approach with five reward-free offline learning methods under controlled changes in trajectory length, quality, quantity and maze layout. Planning generalizes particularly well to unseen layouts and a state-avoidance task, but does not win every test: HILP and GCIQL are much stronger when no training trajectory crosses the doorway. The evidence comes from deterministic simulated navigation, including state-based ant control. It does not establish the same ranking for manipulation, partial observability or unrestricted robotics.

1 Introduction

The question is which learning procedure makes best use of reward-free offline trajectories when the eventual task or layout changes. The paper deliberately controls data properties rather than only reporting one favorable benchmark. Its main contribution is a comparative stress test and a simple planning baseline. Generalization claims apply to the tested navigation families and data regimes.

Original figure
Figure 1: Overview of our analysis. We test six methods for learning from offline reward-free trajectories on 23 different datasets across several navigation environments. We evaluate for six generalization properties required to scale to large offline datasets of suboptimal trajectories. We find that planning with a latent dynamics model (PLDM) demonstrates the highest level of generalization. For a full comparison, see Table 1. Right: diagram of PLDM. Circles represent variables, rectangles – loss components, half-ovals – trained models.

2 Related Work

Goal-conditioned methods learn to reach specified states; zero-shot methods learn reusable skills or representations that can express additional tasks. Optimal-control methods instead search through a learned or known transition model. The paper uses these distinctions to separate what is learned offline from what is optimized when a task is supplied.

Refer to caption
Original figure
Original figure
Figure 2: Left: We train offline goal-conditioned agents on trajectories collected in a subset of maze layouts (left), and evaluate on held out layouts, observing trajectories shown on the right. Only PLDM solves the task (see Figure 8 for more). Right: Success rates of tested methods on held-out layouts, as a function of the number of training layouts. Rightmost plot shows success rates of models trained on data from five layouts, evaluated on held-out layouts ranging from those similar to training layouts to out-of-distribution ones. We use map layout edit distance from the training layouts as a measure of distribution shift. PLDM demonstrates the best generalization performance. Results are averaged over 3 seeds, shaded area denotes standard error. See Figure 1 for more details on PLDM.

3 The Landscape of Available Methods

The comparison includes four goal-conditioned methods, a direction-conditioned zero-shot method and explicit latent-model planning. Each receives offline transitions without recorded rewards.

3.1 Problem Setting

The formal setting assumes deterministic dynamics and fully specified state observations for its model. A task is supplied at evaluation, usually as a goal state. Reward-free data means no reward annotations are provided during collection; goal-conditioned algorithms can still construct their own training objectives from state–goal pairs.

3.2 Reward-free Offline Reinforcement Learning

GCIQL learns a goal-conditioned value and policy, HIQL adds a subgoal hierarchy, CRL learns contrastive state–goal compatibility and GCBC imitates actions conditioned on future goals. HILP learns a distance-like representation plus a policy for moving along latent directions. These are distinct ways to reuse data; Table 1 is a qualitative summary of the following experiments, not an independent benchmark.

Table 1: Road-map of our generalization stress-testing experiments. We test 4 offline goal-conditioned methods - HIQL, GCIQL, CRL, GCBC; a zero-shot RL method HILP, and a learned latent dynamics planning method PLDM. ★★★ denotes good performance in the specified experiment, ★★✩ denotes average performance, and ★✩✩ denotes poor performance. We see that HILP and PLDM are the best-performing methods, with PLDM standing out as the only method that reaches competitive performance in all settings.
Property (Experiment section) HILP HIQL GCIQL CRL GCBC PLDM
Transfer to new environment layouts (4.8) ★✩✩ ★✩✩ ★✩✩ ★✩✩ ★✩✩ ★★★
Transfer to a new task (4.6) ★★✩ ★✩✩ ★✩✩ ★✩✩ ★✩✩ ★★★
Data efficiency (4.3) ★✩✩ ★★✩ ★★★ ★★✩ ★★✩ ★★★
Best-case performance (4.2) ★★★ ★★★ ★★★ ★★★ ★★✩ ★★★
Can learn from random policy trajectories (4.5) ★★★ ★✩✩ ★★★ ★✩✩ ★✩✩ ★★✩
Can stitch suboptimal trajectories (4.4) ★★★ ★✩✩ ★★★ ★✩✩ ★✩✩ ★★✩

3.3 Planning with a Latent Dynamics Model

An encoder maps observations to features and an action-conditioned predictor rolls them forward. Training matches predicted features to encoded future observations. Variance and covariance regularization discourage a constant or redundant representation, while an inverse-dynamics objective can require transitions to retain action information. Multiple predictors optionally estimate disagreement. No pixel reconstruction is required.

Equations 1–3
Encoder:\displaystyle\mathrm{Encoder:} z0^=z0=hθ​(s0)\displaystyle\quad\hat{z_{0}}=z_{0}=h_{\theta}(s_{0}) (1)
Predictors:\displaystyle\mathrm{Predictors:} z^tk=fθk​(z^t−1k,at−1),∀k∈{1,…,K}\displaystyle\quad\hat{z}_{t}^{k}=f_{\theta}^{k}(\hat{z}_{t-1}^{k},a_{t-1}),\forall k\in\{1,\ldots,K\} (2)
ℒsim=∑k=1K∑t=0H1N​∑b=0N‖Z^t,bk−Zt,b‖22\displaystyle\mathcal{L}_{\mathrm{sim}}=\sum_{k=1}^{K}\sum_{t=0}^{H}\frac{1}{N}\sum_{b=0}^{N}\|\hat{Z}^{k}_{t,b}-Z_{t,b}\|^{2}_{2} (3)

Goal-conditioned planning with PLDM.

At deployment, MPPI samples and reweights candidate actions using latent distance to the goal, plus an optional predictor-disagreement penalty. The latter discourages uncertain plans but is not a calibrated safety guarantee. Replanning incorporates new observations. Changing the planning cost can express a new task without retraining the dynamics. This flexibility costs more inference computation than a single policy call.

Equations 4–7
∀k∈{1,…,K}:z^0k=z0k=hθ​(s0),z^tk=fθk​(z^t−1k,at−1)\displaystyle\forall k\in\{1,\ldots,K\}:\ \hat{z}_{0}^{k}=z_{0}^{k}=h_{\theta}(s_{0}),\ \hat{z}_{t}^{k}=f_{\theta}^{k}(\hat{z}_{t-1}^{k},a_{t-1}) (4)
Cgoal​(𝐚,s0,sg)=1K​∑k=1K∑t=0H‖hθ​(sg)−fθk​(z^tk,at)‖\displaystyle C_{\text{goal}}(\mathbf{a},s_{0},s_{g})=\frac{1}{K}\sum_{k=1}^{K}\sum_{t=0}^{H}\|h_{\theta}(s_{g})-f_{\theta}^{k}(\hat{z}_{t}^{k},a_{t})\| (5)
Cuncertainty​(𝐚,s0,sg)=∑t=0Hγt​∑j=1dVar⁡({fθk​(stk,at)j}k=1K)\displaystyle C_{\text{uncertainty}}(\mathbf{a},s_{0},s_{g})=\sum_{t=0}^{H}\gamma^{t}\sum_{j=1}^{d}\mathrm{Var}(\{f_{\theta_{k}}(s_{t}^{k},a_{t})_{j}\}_{k=1}^{K}) (6)
𝐚∗=arg⁡min𝐚​{Cgoal​(𝐚,s0,sg)+β​Cuncertainty​(𝐚,s0,sg)}\displaystyle\mathbf{a}^{*}=\arg\min_{\mathbf{a}}\{C_{\text{goal}}(\mathbf{a},s_{0},s_{g})+\beta C_{\text{uncertainty}}(\mathbf{a},s_{0},s_{g})\} (7)
Refer to caption
Refer to caption
Figure 3: Left: The Two-Rooms environment. The agent starts at a random location and is tasked with reaching the goal at another randomly sampled location in the other room using 200 steps or less. Observations are 64×6464\times 64 pixels images. Right: Examples of trajectories in the offline data. Red: each step’s direction is sampled from Von Mises distribution. Blue: each step’s direction is sampled uniformly.
Method Good-quality data No door-passing trajectories
CRL 189.3   ± 10.7 114.7   ± 14.1
GCBC 186.0   ± 12.0 118.4   ± 11.2
GCIQL 198.0   ± 10.9 199.6   ± 10.4
HILP 100.0   ± 10.0 100.0   ± 10.0
HIQL 196.4   ± 11.3 126.3   ± 15.6
PLDM 197.8   ± 10.7 134.4   ± 12.7
Table 2: Performance of tested methods on good-quality data and on data with no trajectories passing through the door. Values are average success rates (±standard error)(\pm\text{standard error}) across 3 seeds.

4 Every Method Can Excel but Few Generalize

The experiments vary one data property at a time where possible, then compare navigation across changed layouts and a higher-dimensional ant controller. The outcome is a map of conditional strengths rather than one universally best method.

4.1 Two-Rooms Environment

Two-Rooms is a simplified top-down world: actions directly displace a point agent, and the image separates the agent from walls into channels. Persistent random directions generate broad trajectories; independently random directions produce local oscillation. This controlled contrast is useful scientifically but easier than learning real visual locomotion.

4.2 What Methods Excel In-Distribution with a Large High-Quality Dataset?

With abundant trajectories that cover the doorway and long routes, every method performs well and several approach perfect success. The harder tests therefore concern coverage, data scarcity and transfer rather than this favorable in-distribution setting alone.

Takeaway: All methods can perform well when data is plentiful and high-quality.

4.3 What Method is the Most Sample-Efficient?

PLDM and GCIQL retain strong performance with relatively few transitions. HILP needs more data in this environment, though it becomes very strong when enough is available. Sample efficiency here measures transitions required for these simulated tasks, not total training or inference cost.

Original figure
Figure 4: Testing the selected methods’ performance under different dataset constraints. Values and shaded regions are means and standard error over 3 seeds, respectively. Left: To test the importance of the dataset quality, we mix the random policy trajectories with good quality trajectories (see Figure 3). As the amount of good quality data goes to 0, methods begin to fail, with PLDM, GCIQL, and HILP being the most robust ones. Center: We measure methods’ performance when trained with different sequence lengths. We find that many goal-conditioned methods fail when train trajectories are short, which causes far-away goals to become out-of-distribution for the resulting policy. Right: We measure methods’ performance with datasets of varying sizes. We see that PLDM and GCIQL are the most sample efficient, and manage to get almost 80% success rate even with a few thousand transitions. See Appendix H for the analysis of statistical significance.
Takeaway: PLDM and GCIQL are more sample-efficient than other methods.

4.4 What Methods Can Stitch Suboptimal Trajectories?

Two stitching tests must be separated. Short episodes can still teach local transitions that a planner composes into a longer route. Removing every doorway-crossing trajectory removes a crucial transition: PLDM drops to 34.4% success, while HILP and GCIQL reach 100%. Learning a model does not automatically reconstruct missing data coverage.

Takeaway: When solving the task requires ‘stitching’, HILP and GCIQL work great. The performance of PLDM drops, but is better than that of most offline model-free GCRL methods.

4.5 What Methods Can Learn From Trajectories of a Random Policy?

Independent random actions produce short-range wandering. HILP, GCIQL and PLDM cope better than the other tested goal-conditioned policies. A local transition can still supervise a dynamics model even when its trajectory contains no distant goal. The explanation involving goal-distribution mismatch is plausible, not an isolated causal proof.

Takeaway: When the dataset quality is low, HILP, GCIQL and PLDM perform better than other offline GCRL methods.

4.6 What Methods Can Generalize to a New Task?

To evade a chaser, PLDM reverses the sign of its goal-distance cost; HILP reverses its skill direction. Neither is retrained. PLDM maintains greater separation in this fixed-dynamics task. This demonstrates one useful cost change, not automatic generalization to any new reward or changed physics.

Takeaway: Assuming the environment dynamics remain fixed, PLDM can generalize to tasks other than goal-reaching simply by changing the planning objective.
Refer to caption
(a)
Original figure
(b)
Refer to caption
(c)
Figure 5: Zero-shot generalization to the chasing task. (a) In the chase environment, the blue agent is tasked with avoiding the red chaser. The chaser follows the shortest path to the agent. The observations of the agent remain unchanged: we pass the chaser state as the goal state. The agent has to avoid the specified state instead of reaching it. (b) Left: Performance of the tested methods on the chasing task across different chaser speeds, with faster chaser making the task harder. Baselines include agents that take no action (‘Zero’) and random actions (‘Random’). (b) Right: Average distance between the agent and chaser agent throughout the episode when chaser speed is 1.01.0. (c) Visualization of ant-umaze environment. The 4-legged ant is tasked with reaching a randomly sampled goal within a u-shaped room.

4.7 Extending to a Higher-Dimensional Control Environment.

Ant-U-Maze increases control complexity to an eight-dimensional action and 29-dimensional state. The dataset comes from a noisy directional expert policy, rather than raw random joint actions. PLDM, HIQL and HILP stitch short trajectories successfully; GCIQL’s relative advantage from Two-Rooms does not persist. The result is state-based simulation, not camera-only locomotion.

Original figure
Figure 6: Success rates in Ant-U-Maze for agents trained on trajectories of varying lengths.
Takeaway: Planning with a latent dynamics model maintains good performance in a standard quadruped maze task, indicating promising generalization to higher control complexity.

4.8 What Methods Can Generalize to Unseen Environment Layouts?

Diverse PointMaze changes obstacle layouts and adds acceleration and velocity. The agent receives a maze image plus velocity. Training layouts are disjoint from test layouts, and edit distance measures the specified layout shift. PLDM transfers much better than the compared policies. Single-layout results show that failure under transfer is different from inability to solve the training task.

Refer to caption
Figure 7: Left: Plans generated by PLDM at test time. Right: Actual agent trajectories for the tested methods. PLDM is the only method that reliably succeeds on held-out mazes.
Takeaway: The model-based approach enables better generalization to unseen obstacle layouts than model-free methods.

5 Conclusion

The practical lesson is to match the method to the data and expected deployment shift. PLDM handles several shifts well; structured skill representations and goal-conditioned value learning can be stronger for missing-transition stitching; plentiful high-quality data makes many policy methods effective. These are empirical guidelines within this suite.

1. PLDM exhibits robustness to data quality, a high level of data efficiency, best-of-class generalization to new layouts, and excels at adapting to tasks beyond goal-reaching; 2. Learning a well-structured latent-space (e.g. using HILP) enables trajectory stitching and robustness to data quality, although it is more data-hungry than other methods; 3. Model-free GCRL methods are a great choice when the data is plentiful and good quality.

Future work.

The open directions are more complex manipulation, partial observability, accumulated long-rollout errors and reducing planning cost. These are needed before extending the study’s conclusions to general-purpose embodied systems.

Acknowledgments

The source credits computing facilities, grants and institutional support for the study.

Appendix

Table of Contents

Appendix A Reproducibility

Public code includes environments, data generation, training and evaluation. AppendixJ retains the experiment-dependent hyperparameters; reproducing a result requires its corresponding data regime and tuning choices.

Code is available at https://github.com/vladisai/PLDM

Appendix B Visualization of Plans and Trajectories for Diverse PointMaze

The trajectory panels compare unseen layouts at increasing minimum edit distance from the training set. They make the layout-shift definition concrete and illustrate PLDM’s stronger transfer. Selected successful or failed paths complement the aggregate scores rather than replacing them.

Refer to caption
Figure 8: Top: The training layouts used in the 5 layout setting. Middle: Trajectories of different agents navigating an unseen maze layout towards goal at test time. As the layouts become increasingly out-of-distribution, only PLDN consistently succeeds. Layouts can be represented as a 4x4 array, with each value being either a wall or empty space. The distribution shift is quantified by the minimum edit distance between a given test layout and the closest training layout. The top row corresponds to an in-distribution layout with a minimum edit distance of 0, and with each subsequent row, the minimum edit distance increases by 1 incrementally.

Appendix C Environments and Datasets

C.1 Two-Rooms Environment

The Two-Rooms simulator has no inertia: an action moves the agent immediately, and wall intersection clips that movement. Learning this transition map is substantially simpler than inferring velocities and contacts in realistic control.

C.2 Diverse PointMaze

Diverse PointMaze renders a randomly generated connected four-by-four layout and supplies the agent’s velocity. The dynamics include inertia; actions repeat for four simulator steps. These extra inputs and temporal conventions matter when comparing it with Two-Rooms.

C.2.1 Dataset Generation

Each layout-count condition keeps the total transition count fixed at one million. Increasing training layouts therefore reduces episodes per layout. This tests diversity at a controlled overall data budget, with randomly initialized position and velocity.

# Transitions # layouts # episodes per layout episode length
1000000 5 2000 100
1000000 10 1000 100
1000000 20 500 100
1000000 40 250 100
Table 3: Details for Diverse PointMaze datasets

C.2.2 Evaluation

The same held-out layouts and start–goal trials are used for all agents. One test varies training diversity; another groups test layouts by their minimum grid edit distance to a training layout. That metric measures this particular structural shift, not every possible visual or physical distribution shift.

C.2.3 Results for Single Maze Setting

All methods perform very well on a single fixed maze. The table establishes a useful control: their difficulties on unseen layouts cannot be explained simply by universal inability to navigate.

Table 4: Results averaged over 3 seeds ± std
Method Success rate
PLDM 0.990 ± 0.001
CRL 0.980 ± 0.001
GCBC 0.970 ± 0.024
GCIQL 1.000 ± 0.000
HIQL 1.000 ± 0.000
HILP 1.000 ± 0.000

C.3 Ant U-Maze

The ant benchmark checks trajectory composition with more complex actuation using simulator state observations. It extends control dimensionality without testing visual perception or real-world contact uncertainty.

C.3.1 Dataset Generation

A pretrained directional expert, noisy actions and periodically resampled directions produce five million exploratory transitions. This dataset contains useful locomotion skills even though its downstream goal is not annotated.

C.3.2 Evaluation

The ant starts at one end of a U-shaped room and must reach the diagonally opposite end. Each method is tested on ten trials. This is a specific route requiring a turn, not a broad suite of unfamiliar terrain.

Appendix D Models

Baselines use released OGBench or HILP implementations. Velocity is appended directly for Diverse PointMaze instead of being inferred by stacking images. This aligns the available state information across the compared methods.

D.1 PLDM

D.1.1 Objective for collapse prevention

The anti-collapse loss combines feature spread, low cross-feature covariance, optional temporal smoothness and inverse action reconstruction. The prose describes temporal variance, but the displayed variance formula indexes variance across the batch at each timestep; the original equations are retained, so this distinction should be checked against code before reproduction. Smoothness and action reconstruction trade off persistence against retaining changes.

Equations
ℒvar=1H​D∑t=0H∑j=0Dmax(0,γ−Var(Zt,:,j)+ϵ)\displaystyle\mathcal{L}_{\mathrm{var}}=\frac{1}{HD}\sum^{H}_{t=0}\sum^{D}_{j=0}\mathrm{max}(0,\gamma-\sqrt{\mathrm{Var}(Z_{t,:,j})+\epsilon})
C⁡(Zt)=1N−1​(Zt−Zt¯)⊤​(Zt−Zt¯),Z¯=1N​∑b=1NZt,b\displaystyle C(Z_{t})=\frac{1}{N-1}(Z_{t}-\bar{Z_{t}})^{\top}(Z_{t}-\bar{Z_{t}}),\ \bar{Z}=\frac{1}{N}\sum^{N}_{b=1}Z_{t,b}
ℒcov=1H​∑t=0H1D​∑i≠j[C⁡(Zt)]i,j2\displaystyle\mathcal{L}_{\mathrm{cov}}=\frac{1}{H}\sum^{H}_{t=0}\frac{1}{D}\sum_{i\neq j}[C(Z_{t})]^{2}_{i,j}
ℒIDM=∑t=0H1N​∑b=0N‖at,b−MLP⁡(Z(t,b),Z(t+1,b))‖22\displaystyle\mathcal{L}_{\mathrm{IDM}}=\sum^{H}_{t=0}\frac{1}{N}\sum^{N}_{b=0}\|a_{t,b}-\mathrm{MLP}(Z_{(t,b)},Z_{(t+1,b)})\|^{2}_{2}
ℒtime−sim=∑t=0H−11N​∑b=0N‖Zt,b−Zt+1,b‖22\displaystyle\mathcal{L}_{\mathrm{time-sim}}=\sum_{t=0}^{H-1}\frac{1}{N}\sum_{b=0}^{N}\|Z_{t,b}-Z_{t+1,b}\|^{2}_{2}
ℒJEPA=ℒsim+α​ℒvar+β​ℒcov+δ​ℒtime−sim+ω​ℒIDM\displaystyle\mathcal{L_{\mathrm{JEPA}}}=\mathcal{L}_{\mathrm{sim}}+\alpha\mathcal{L}_{\mathrm{var}}+\beta\mathcal{L}_{\mathrm{cov}}+\delta\mathcal{L}_{\mathrm{time-sim}}+\omega\mathcal{L}_{\mathrm{IDM}}

D.1.2 Ablations of Objective Components

Removing variance or covariance terms severely harms both tested environments. Removing inverse dynamics hurts Diverse Maze but does not change Two-Rooms in this setting, where its coefficient is already zero. Regularizer importance is therefore conditional on the environment and configured objective.

Ablation Success rate (Two-Rooms) Success rate (Diverse Maze)
– 98.0 ± 1.5 98.7 ± 2.8
var coeff (α\alpha) 13.4 ± 9.2 11.4 ± 6.5
cov coeff (β\beta) 29.2 ± 4.4 7.8 ± 4.1
time sim coeff (δ\delta) 71.0 ± 3.0 95.6 ± 3.2
IDM coeff (ω\omega) 98.0 ± 1.5 75.5 ± 8.2

D.1.3 Model Details for Two-Rooms

Two-Rooms uses a small visual encoder and a recurrent predictor whose initial hidden state is the encoded observation. Subsequent action inputs advance that state. Layer normalization is applied to features and predictions. The architecture is small enough for many controlled experiments.

D.1.4 Model Details for Diverse PointMaze Environment

Diverse PointMaze preserves a spatial feature map. Expanded velocity and action planes are concatenated to that map before convolutional prediction. It replans every four interactions and uses a single predictor in the reported hyperparameters. This differs from the recurrent and ensemble configurations elsewhere.

D.1.5 Model Details for Ant-U-Maze

For the ant, an MLP encodes global position and concatenates raw proprioception. A five-model MLP ensemble predicts the next representation. Anti-collapse regularization applies to the learned position features, not raw coordinates that cannot collapse through an encoder.

Appendix E Effects of Uncertainty Regularization via Ensembles

The plots ablate ensemble uncertainty regularization. Its benefit depends on the task and data regime, so disagreement should be treated as a configurable empirical aid. The method also has a single-predictor configuration and is not inherently dependent on an ensemble.

Original figure
Figure 9: Top Row: Two-Rooms environment. Bottom Row: AntMaze environment.

Appendix F Analyzing planning time of PLDM

Replanning every step is slower than policy execution. In Two-Rooms, replanning every four steps reduces average episode time from 16.0 to 4.8 seconds while retaining 95% of the every-step success rate; every 32 steps retains only 62%. Less frequent feedback creates a clear speed–reliability tradeoff.

Table 5: Time of evaluation on one episode in Two-Rooms environment. Time is averaged across 25 episodes. PLDM success rates are normalized against the setting that replans every step. PLDM can match the latencies of GCIQL and HIQL by replanning less frequently with neligible cost to performance.
Method Replan Every Time per episode (seconds) Normalized Success Rate
PLDM 1 16.0 ± 0.13 1.00
PLDM 4 4.8 ± 0.09 0.95
PLDM 16 2.6 ± 0.07 0.90
PLDM 32 2.2 ± 0.07 0.62
GCIQL – 3.6 ± 0.10 –
HIQL – 4.0 ± 0.08 –

F.1 Evaluating PLDM Performance With Adjusted Inference Compute Budget

Additional plots compare the reduced-frequency planner under a more similar inference budget. Much of its performance remains, though some is lost. Matching total runtime does not make the internal computation or action-feedback frequency identical.

Original figure
Figure 10: Comparing PLDM’s performance under fixed inference time compute budget on two-rooms. We see that across two-rooms experiments, PLDM performs only slightly worse when replanning every 4 steps compared to replanning every step.
Original figure
Figure 11: Evaluating PLDM’s performance under fixed inference time compute budget on diverse maze. Similarly to two-rooms, we only see a minor degradation in performnace

Appendix G Extended Baselines: Input Reconstruction Learning and TD-MPC2

These controls replace the representation objective or adapt existing world models to the reward-free task. They test particular configurations and should be interpreted with the adaptations in view.

Dreamer

DreamerV3 loses reward training and learned policy execution, then uses latent-distance planning. The authors explicitly acknowledge that this differs from its intended design. Poor performance here is evidence about this reward-free adaptation, not the full Dreamer system.

Reconstruction

A closer reconstruction control retains PLDM’s encoder and predictor architecture while replacing its regularization with pixel reconstruction. It isolates that objective change more directly than comparing entirely different model families.

TD-MPC2

Reward removal causes TD-MPC2 to collapse in this setup. Adding inverse dynamics helps but remains weak; that modified variant has only one tested seed. The table supports the need to prevent collapse under this altered objective, not a general comparison against normally trained TD-MPC2.

Table 6: Comparing reconstruction-based latent-dynamics learning and TDMPC-2 to other baselines. We test the methods on good-quality data in the two-rooms environment. We see that reconstruction-based methods perform significantly worse than PLDM and other baselines. TD-MPC2 fails to learn altogether due to collapse, and performs poorly even with the added IDM objective.
Method Success rate
CRL 189.3   ± 11.2
GCBC 186.0   ± 14.5
GCIQL 198.0   ± 10.9
HILP 100.0   ± 10.0
HIQL 196.4   ± 13.0
PLDM 197.4   ± 11.3
DreamerV3 124.0   ± 16.9
Reconstruction 126.2   ± 13.9
TD-MPC2 110.0   ± 10.0
TD-MPC2 + IDM 135.0   ± 10.0

Appendix H Analyzing Statistical Significance of Results

Welch tests compare methods using pooled settings and selected ten-seed runs. Several comparisons with HILP and GCIQL are not significant, consistent with their competitive performance. Pooling different data conditions provides one aggregate analysis and should not be read as proving superiority in each individual condition.

Table 7: Statistical significance of results (pooled). ✓ means that Welch’s t-test showed that PLDM is better than the corresponding method when pooling results across seeds and dataset parameters.
Sequence length
Fraction of
non-random data
Dataset size
Number of
training maps
In-distribution →
Out-of-distribution
CRL ✓(p=1.68e-03) ✓(p=8.46e-07) ✓(p=1.13e-02) ✓(p=3.43e-10) ✓(p=4.02e-03)
GCBC ✓(p=4.25e-04) ✓(p=1.46e-08) ✓(p=7.26e-03) ✓(p=3.71e-20) ✓(p=1.94e-05)
GCIQL ✗ ✗ ✗ ✓(p=1.29e-12) ✓(p=1.22e-03)
HILP ✗ ✗ ✓(p=2.11e-02) ✓(p=6.54e-06) ✓(p=4.10e-05)
HIQL ✓(p=2.21e-02) ✓(p=9.26e-05) ✓(p=2.39e-02) ✓(p=2.84e-06) ✓(p=9.01e-03)
Table 8: Statistical significance of results for selected datasets. ✓ means that Welch’s t-test showed that PLDM is better than the corresponding method results across 10 seeds.
Sequence length 17
Fraction of
non-random data 0%
Dataset size 20312
CRL ✓(p=3.82e-11) ✓(p=1.13e-07) ✓(p=6.76e-11)
GCBC ✓(p=1.54e-12) ✓(p=5.75e-10) ✓(p=1.88e-14)
GCIQL ✗ ✗ ✗
HILP ✗ ✗ ✓(p=2.49e-19)
HIQL ✓(p=8.89e-05) ✓(p=2.83e-07) ✓(p=2.65e-11)

Appendix I Analyzing HILP’s Out-of-Distribution Generalization

Visualized HILP distances align with routes on training layouts but become noisy on held-out layouts. This supports representation transfer as a failure mode in the examined models. It is a diagnostic example rather than proof that the representation must always fail under layout changes.

Refer to caption
(a) HILP distances on in-distribution maze layouts
Refer to caption
(b) HILP distances on out-of-distribution maze layouts
Figure 12: HILP learns representations of states such that distance in representation space between a pair of states is equal to the number of steps needed to go between steps. These plots visualize the distance from the state denoted with a red dot to other states across the maze. (a) We see that on an in-distribution maze, the distances increase smoothly and mostly reflect the number of steps needed to go between states. (b) We see that on an out-of distribution maze, the representation space distances no longer make sense.

Appendix J Hyperparameters

J.1 CRL, GCBC, GCIQL, HIQL, HILP

Baseline defaults come from released repositories, with the reported task-specific adjustments. Equivalent architectures do not imply identical optimal learning rates or goal-sampling distributions.

J.1.1 Two-Rooms

Grid searches cover good-quality and short-trajectory conditions, then selected settings are applied across tests. Different regimes receive some different parameters; the comparison is not one universal untuned configuration.

HILP.

HILP uses a fixed learning rate and expectiles in these Two-Rooms experiments.

GCIQL.

GCIQL changes behavior-cloning weight and random-goal sampling for particular data regimes. These settings influence conservative policy learning and the goals seen during training.

HIQL.

HIQL changes learning rate and random-goal probability in the short-trajectory conditions while retaining its two-level policy design.

J.1.2 Diverse PointMaze

The retained tables give learning rates, expectiles and goal-sampling parameters for each layout-count condition. Reproduction should use the row matching the tested training diversity.

Table 9: Dataset-specific hyperparameters of CRL, GCBC, and HILP for the Diverse PointMaze environment. For HILP, we set the same value for expectile and skill expectile.
Dataset CRL GCBC HILP
LR LR LR Expectile
5 layouts 0.0003 0.0003 0.0001 0.9
10 layouts 0.0003 0.0001 0.0001 0.9
20 layouts 0.0003 0.0001 0.0001 0.9
40 layouts 0.0003 0.0001 0.0001 0.9
Table 10: Dataset-specific hyperparameters of GCIQL and HIQL for the Diverse PointMaze environment.
Dataset GCIQL HIQL
LR Expectile
BC
Coeff.
Prob
Rand Goal
LR Expectile
AWR
Temp.
Prob
Rand Goal
5 layouts 0.0003 0.6 0.3 0.15 0.00003 0.6 1.0 0.6
10 layouts 0.0003 0.6 0.3 0.15 0.0001 0.7 3.0 0.0
20 layouts 0.0003 0.6 0.3 0.15 0.00003 0.6 1.0 0.6
40 layouts 0.0003 0.6 0.3 0.15 0.0001 0.7 3.0 0.0

J.1.3 Ant U-Maze

Ant settings vary with episode length and method. The tables retain those tuning choices rather than suggesting that every method used one shared hyperparameter set.

Table 11: Ant-Umaze: dataset-specific hyperparameters for CRL, GCBC, and HILP, parsed from run names.
Seq Len CRL GCBC HILP
LR
BC
Coeff.
Prob
Rand Goal
LR LR
Expectile
(skill / actor)
25 3e-5 0.1 0.3 3e-3 3e-4 0.6 / 0.8
50 3e-5 0.1 0.3 3e-3 3e-4 0.6 / 0.8
100 3e-4 0.1 0.0 3e-3 3e-4 0.6 / 0.8
250 3e-4 0.1 0.0 3e-3 3e-4 0.6 / 0.8
500 3e-4 0.1 0.0 3e-3 3e-4 0.6 / 0.8
Table 12: Ant-Umaze: dataset-specific hyperparameters for GCIQL and HIQL parsed from run names.
Seq Len GCIQL HIQL
LR Expectile
BC
Coeff.
Prob
Rand Goal
LR Expectile
AWR
Temp.
Prob
Rand Goal
25 3e-5 0.9 0.15 0.15 3e-4 0.6 3.0 0.6
50 3e-5 0.9 0.15 0.15 3e-4 0.6 3.0 0.6
100 3e-5 0.9 0.15 0.15 3e-4 0.6 3.0 0.6
250 3e-4 0.9 0.3 0.0 3e-4 0.6 3.0 0.6
500 3e-4 0.9 0.3 0.0 3e-4 0.6 3.0 0.6

J.2 PLDM

J.2.1 Two-Rooms

Two-Rooms uses a five-predictor ensemble and a horizon of 16 model steps, but the regularization weights and learning rate vary by dataset. Inverse dynamics is disabled in this configuration. The table repeats one non-random-data row; the printed record is retained.

Table 13: Dataset-agnostic hyperparameters for Two-Rooms
Hyperparameter Value
Batch Size 64
Predictor Horizon (HH) 16
Optimizer Adam
Scheduler Cosine
Ensemble size KK 5
ω\omega 0
MPPI noise σ\sigma 5
MPPI #\# samples 500
MPPI λ\lambda 0.005
Planner CuncertaintyC_{\text{uncertainty}} coeff β\beta 0.0001
Planner CuncertaintyC_{\text{uncertainty}} coeff γ\gamma 0.9
Table 14: Dataset specific hyperparameters for Two-Rooms
Dataset LR α\alpha β\beta δ\delta
Sequence length = 91 0.0007 4.0 6.9 0.75
Sequence length = 65 0.0003 5.0 6.9 0.75
Sequence length = 33 0.0014 3.5 6.9 0.75
Sequence length = 17 0.0028 3.0 6.9 0.75
Dataset size = 634 0.0030 2.2 13.0 0.50
Dataset size = 1269 0.0010 2.2 13.0 0.50
Dataset size = 5078 0.0005 2.2 13.0 0.90
Dataset size = 20312 0.0030 2.2 13.0 0.50
Dataset size = 81250 0.0010 2.2 13.0 0.50
Dataset size = 325k 0.0010 4.0 6.9 0.75
Dataset size = 1500k 0.0010 4.0 6.9 0.75
Non-random % = 0.001 0.0007 3.9 6.9 0.74
Non-random % = 0.01 0.0007 3.9 6.5 0.19
Non-random % = 0.02 0.0007 3.9 6.5 0.72
Non-random % = 0.04 0.0007 3.9 6.5 0.65
Non-random % = 0.08 0.0007 3.9 6.5 0.24
Non-random % = 0.08 0.0007 3.9 6.5 0.24
Wall crossing % = 0 0.0007 4.0 6.9 0.75

J.2.2 Diverse PointMaze

Diverse PointMaze trains for five epochs with one predictor. The source repeats the five-layout row and omits an explicitly labeled ten-layout row in its dataset-specific table. The edition preserves that ambiguity rather than inventing missing settings.

Table 15: Dataset-agnostic hyperparameters for Diverse PointMaze
Hyperparameter Value
Epochs 5
Batch Size 128
Predictor Horizon (HH) 16
Optimizer Adam
Scheduler Cosine
Ensemble size KK 1
MPPI noise σ\sigma 5
MPPI #\# samples 500
MPPI λ\lambda 0.0025
Table 16: Dataset specific hyperparameters for Diverse PointMaze
Dataset LR α\alpha β\beta δ\delta ω\omega
# map layouts = 5 0.04 35.0 12.0 0.1 5.4
# map layouts = 5 0.04 35.0 12.0 0.1 5.4
# map layouts = 20 0.05 54.5 15.5 0.1 5.2
# map layouts = 40 0.05 54.5 15.5 0.1 5.2

J.2.3 Ant-U-Maze

Ant uses a five-predictor ensemble and retains inverse dynamics, with learning rate changing by trajectory length. These settings differ materially from the simpler image-navigation configurations.

Table 17: Dataset-agnostic hyperparameters for Ant-U-Maze
Hyperparameter Value
Epochs 5
Batch Size 64
Predictor Horizon (HH) 16
Optimizer Adam
Scheduler Cosine
Ensemble size KK 5
α\alpha 26.2
β\beta 0.5
δ\delta 8.1
ω\omega 0.58
MPPI noise σ\sigma 5
MPPI #\# samples 500
MPPI λ\lambda 0.0025
Planner CuncertaintyC_{\text{uncertainty}} coeff β\beta 1
Planner CuncertaintyC_{\text{uncertainty}} coeff γ\gamma 0.9
Table 18: Dataset-specific hyperparameters for Ant-U-Maze
Dataset LR
Sequence length = 25 0.006
Sequence length = 50 0.004
Sequence length = 100 0.003
Sequence length = 250 0.001
Sequence length = 500 0.001

J.3 Further related work

Further related work connects conservative offline learning, large robot datasets and self-supervised representations. These links motivate the study without enlarging its tested task scope.

Appendix K Discussion of Computational Costs

A run uses one GPU and up to a day on a V100, while the complete research process is estimated at 500–2000 GPU-days. Per-run affordability and aggregate experimental cost are different quantities.

Appendix L Limitations

The principal limits are navigation-only experiments, deterministic/fully observed settings and increased planning cost. The compute-matched tests partly address latency, but manipulation, partial observability and broad real-world transfer remain untested.

Concept demonstration

Open image ↗