Literature
Display
← All papers

LUMOS: Language-Conditioned Imitation Learning with World Models

Iman Nematollahi · Branton DeMoss · Akshay L Chandra · Nick Hawes · Wolfram Burgard · Ingmar Posner

· ICRA 2025 ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

At a glance

LUMOS trains a language-conditioned robot policy inside a learned world model. The model first learns dynamics from offline play; an actor–critic then practices latent rollouts, receiving rewards for matching demonstration trajectories. Hindsight goal relabeling, latent plans, and contrastive language alignment allow sparsely annotated play to support language commands.

At matched 64×64 image resolution, LUMOS completes an average 2.34 instructions in CALVIN chains versus HULC’s 2.27. Physical experiments report 2.05 versus 1.90 sequential tasks. “Zero-shot” here means no online policy fine-tuning after offline training on real robot data, not transfer without real-world experience. Project videos, dataset, and code accompany the paper.

I Introduction

Behavior cloning learns actions on demonstrated states, but small deployment mistakes can lead to unfamiliar observations and compound over time. Practicing a policy under its own state distribution can address this mismatch; doing so physically is expensive.

LUMOS practices in an action-conditioned latent simulator learned from play data. It extends DITTO trajectory matching with language conditioning and hindsight latent planning. The proposed benefit is recovery over multiple imagined steps rather than independent action prediction.

This substitutes learned dynamics for a hand-built simulator, but does not eliminate modeling error or guarantee recovery on real states outside the dataset. The experiments show a proof of concept on specified tabletop tasks, not unrestricted long-horizon language execution.

Refer to caption
Fig. 1: LUMOS learns a general-purpose language-conditioned visuomotor policy within the latent space of a learned world model. By optimizing an intrinsic reward to match expert performance, it recovers from its own mistakes across multiple time steps and reduces covariate shift. As a result, LUMOS can handle complex, long-horizon robot tasks from abstract language instructions in real-world scenarios, without online learning or fine-tuning.

II Related Work

MCIL learns language-steerable skills from sparsely labeled play. HULC adds latent planning and language–vision contrastive alignment; other systems add affordances, explicit skill decompositions, or diffusion policies. LUMOS studies a different source of improvement: on-policy learning inside a fixed offline world model.

Hindsight planning gives a training-time recognizer the entire demonstration, then teaches a proposal network to predict a useful plan from the current state and goal. At inference only the proposal is available. This compresses multi-step behavior without assuming one unique trajectory for every start/goal pair.

Dreamer-style recurrent models provide compact predictive states. DITTO rewards similarity to expert latent trajectories and trains an actor–critic through imagined dynamics. LUMOS combines these components; its baseline experiment does not compare against every newer diffusion, affordance, or hybrid control system.

Refer to caption
Fig. 2: LUMOS learns a language-guided general-purpose policy within the latent space of a world model. (1) The world model, comprising an image encoder, a Recurrent State-Space Model (RSSM) for dynamics, and an image decoder, transforms play dataset experience into a predictive model that enables behavior learning in the latent state space. (2) The goal-conditioned policy samples latent trajectories and uses either a language annotation or the final latent state as the goal, with plan recognition and proposal networks being trained to identify and organize behaviors in a latent plan space. The action decoder is intrinsically rewarded by matching the expert’s latent trajectory. (3) During inference, the policy acts based on the latent state inferred by the world model from the current observation and is guided by a user’s language command.

III Problem Formulation

The setting is goal-conditioned imitation under partial observability: the robot sees camera images, takes continuous pose commands plus a gripper command, and has no usable task-reward feedback for offline policy training.

Fixed play trajectories are sampled into windows and retrospectively labeled with their final reached state as a goal. A small subset also has free-form language descriptions. These two goal modalities train a common policy; the language labels describe behavior already present in the data rather than generating new demonstrations.

IV LUMOS

Training has two stages. Fit the world model to play observations and actions, then freeze it while optimizing the policy through imagined latent trajectories. At execution, current camera observations update the model state and a language instruction supplies the goal.

IV-A World Model Learning

Separate convolutional encoders and decoders process static and gripper-camera images. Their features feed a DreamerV2 recurrent state-space model with deterministic recurrent memory and stochastic categorical latent variables.

The posterior sees the current observation; the prior predicts the next latent state from history and actions without that observation. Training combines reconstruction and KL regularization, using straight-through gradients for categorical samples. The retained module definitions and objective specify this structure.

Imagined rollouts use prior predictions after an initial state estimate. They are cheap to generate, but their quality remains bounded by learned dynamics and dataset coverage. “Unlimited” model rollouts do not provide unlimited reliable real-world experience.

Equations 1–2
 RSSM {Recurrent state:ht=fϕ​(s^t−1,at−1)Representation model:zt∼qϕ​(zt∣ht,xt)Dynamics predictor:z^t∼pϕ​(z^t∣ht)Image decoder:x^t∼pϕ​(x^t∣s^t)\hskip-5.69046pt\text{ RSSM }\begin{cases}\text{Recurrent state:}&h_{t}=f_{\phi}(\hat{s}_{t-1},a_{t-1})\\ \text{Representation model:}&z_{t}\sim q_{\phi}(z_{t}\mid h_{t},x_{t})\\ \text{Dynamics predictor:}&\hat{z}_{t}\sim p_{\phi}(\hat{z}_{t}\mid h_{t})\\ \text{Image decoder:}&\hat{x}_{t}\sim p_{\phi}(\hat{x}_{t}\mid\hat{s}_{t})\end{cases} (1)
minϕ𝔼qϕ(z1:T∣a1:T,x1:T)[∑t=1T−logpϕ(xt∣s^t)+βDKL(qϕ(zt∣s^t)∥pϕ(z^t∣ht))].\begin{split}\min_{\phi}\mathbb{E}_{q_{\phi}(z_{1:T}\mid a_{1:T},x_{1:T})}\Bigg[\sum_{t=1}^{T}-\log p_{\phi}(x_{t}\mid\hat{s}_{t}&)\\ +\beta D_{\text{KL}}(q_{\phi}(z_{t}\mid\hat{s}_{t})\;\|\;p_{\phi}(\hat{z}_{t}\mid h_{t}))&\Bigg].\end{split} (2)

IV-B Behavior Learning

The policy encodes the world-model trajectory into perceptual features. A transformer plan recognizer sees the complete training window and produces categorical latent plans; a proposal network learns to predict their distribution from the initial state and goal. At test time it supplies the plan without future observations.

For language-labeled windows, a symmetric contrastive objective aligns latent-trajectory and sentence embeddings, treating corresponding pairs as positives and other batch pairs as negatives. Unlabeled windows use reached-state goals and omit the language contrastive term.

The intrinsic reward is the expert/agent latent inner product divided by the squared larger latent norm. It favors aligned, similarly scaled states at corresponding times. Equation 3 contains no separate action-distance term, despite the prose’s state–action matching description. This surrogate encourages imitation within the model; it is not an observed task-success reward.

A feed-forward actor outputs a tanh-transformed Gaussian from current features, goal, and sampled plan. A critic predicts discounted intrinsic returns. The critic fits bootstrapped lambda returns with stopped target gradients, while the actor differentiates through frozen model dynamics to increase return and also minimize plan-KL and language-alignment losses.

This creates on-policy experience in the learned model, not new online robot data. It can reduce action-error accumulation relative to the tested cloning ablation, but model errors can still produce apparently rewarding imagined behavior that fails physically.

Equations 3–4
rti​n​t​(stE,stπ)=stE⋅stπmax⁡(‖stE‖,‖stπ‖)2r^{int}_{t}(s^{E}_{t},s^{\pi}_{t})=\frac{s^{E}_{t}\cdot s^{\pi}_{t}}{\max(\|s^{E}_{t}\|,\|s^{\pi}_{t}\|)^{2}} (3)
Actor:at∼πθ(at∣st,g),\displaystyle\text{Actor:}\hskip 10.00002pta_{t}\sim\pi_{\theta}(a_{t}\mid s_{t},g),\hskip 10.00002pt (4)
Critic:vψ(st,g)≈𝔼πθ,pϕ[∑t=0Hγtri​n​tt].\displaystyle\text{Critic:}\hskip 10.00002ptv_{\psi}(s_{t},g)\approx\mathbb{E}_{\pi_{\theta},p_{\phi}}\left[\sum_{t=0}^{H}\gamma^{t}r^{int}_{t}\right].
Method No. Instructions in a Row (1000 chains)
1 2 3 4 5 Avg. Len.
GCBC [9] 56.7% (3.4) 20.9% (1.9) 7.3% (2.9) 1.6% (0.5) 0.04% (0.0) 0.63 (0.4)
MCIL [11] 70.3% (3.1) 40.2% (2.9) 21.6% (4.1) 11.1% (1.9) 5.4% (1.0) 1.48 (0.3)
HULC [10] 77.6% (1.1) 58.05% (0.3) 41.6% (1.3) 29.8% (0.9) 20.0% (1.5) 2.27 (0.05)
LUMOS (ours) 80.7% (1.0) 59.3% (1.4) 42.6% (1.8) 30.7% (1.2) 21.1% (0.8) 2.34 (0.05)
No DITTO 71.8% (1.3) 45.6% (1.5) 27.8% (1.9) 14.2% (1.8) 8.7% (0.7) 1.68 (0.02)
No latent plan 76.5% (2.5) 50.5% (0.9) 28.7% (1.3) 15.9% (0.8) 11.0% (0.7) 1.81 (0.01)
No alignment 73.3% (1.5) 52.6% (0.9) 39.8% (2.0) 27.3% (1.3) 17.2% (0.9) 2.05 (0.06)
TABLE I: Performance of all models on environment D of the CALVIN Challenge with an ablation study of key components evaluated over three seeds. All models receive 64×6464\times 64 resolution RGB images from both a static and gripper camera. Standard deviation in parentheses.
Equations 5–6
ℒ⁡(ψ)​=˙​𝔼πθ,pϕ​[∑t=1H−112​(vψ​(s^t)−s​g​(Vtλ))2]\mathcal{L}(\psi)\,\dot{=}\,\mathbb{E}_{\pi_{\theta},p_{\phi}}\left[\sum_{t=1}^{H-1}\tfrac{1}{2}(v_{\psi}(\hat{s}_{t})-sg(V^{\lambda}_{t}))^{2}\right] (5)
ℒ⁡(θ)​=˙​𝔼πθ,pϕ​[∑τ=tt+H(−Vλ​(sτ)+α1​ℒK​L+α2​ℒcontrast)]\hskip-5.69046pt\mathcal{L}(\theta)\,\dot{=}\,\mathbb{E}_{\pi_{\theta},p_{\phi}}\left[\sum_{\tau=t}^{t+H}\left(-V_{\lambda}(s_{\tau})+\alpha_{1}\mathcal{L}_{KL}+\alpha_{2}\mathcal{L}_{\text{contrast}}\right)\right] (6)

V Experimental Results

Evaluation separates CALVIN instruction chaining, component ablations, and transfer of offline-trained policies to a real tabletop robot.

V-A Experiments in Simulation

The simulation study trains and evaluates in CALVIN environment D, using six hours of play, 34 subtasks, and about 1% language annotation. Each of three seeds is evaluated on 1,000 chains of up to five instructions. This is not the harder cross-environment ABC→D generalization protocol.

All methods receive two 64×64 camera streams. LUMOS has 80.7% first-task success, 21.1% five-task completion, and average chain length 2.34±0.05. HULC has 77.6%, 20.0%, and 2.27±0.05. Parentheses in the original table are standard deviations across three seeds, not confidence intervals. The aggregate improvement is modest and should not be presented as a broad benchmark-state-of-the-art claim.

Replacing intrinsic-reward practice with mean-squared-error cloning lowers mean chain length to 1.68; removing latent plans gives 1.81, and removing language alignment gives 2.05. These controlled ablations support the combined training design, without independently measuring distribution shift.

The source table contains one numerical inconsistency: GCBC’s reported completion probabilities sum to about 0.865 expected tasks, whereas its printed average length is 0.63. Both original entries remain unchanged; no corrected baseline value is inferred.

Figure 3 is a qualitative prediction check: five observed frames initialize a held-out trajectory, then known actions drive 195 predicted steps. The model trains on 50-step windows. Visual plausibility in this example is not a quantitative guarantee of long-horizon state or task accuracy.

Refer to caption
Fig. 3: World Model Long-Horizon Predictions. Using the first five images of a hold-out trajectory as context, the world model predicts the next 195 steps using its latent dynamics, given only the actions. Trained on sequences of horizon 50, our model makes precise long-term predictions, aiding efficient behavior learning in a compact latent space. Only gripper camera reconstructions are visualized.
Refer to caption
Fig. 4: Real-world Tasks. Examples from left to right are: placing the carrot onto the pan, lifting the carrot from the table, dropping the carrot into the bowl, storing the carrot in the cabinet, placing the pan into the cabinet, setting the pan onto the stove, adding the eggplant to the pan, picking up the pan from the table, taking the eggplant from the cabinet, and putting the eggplant into the bowl.

V-B Experiments in Real-World

The real setup uses a Panda arm with a stove, bowl, cabinet, pan, and toy carrot/eggplant. Three hours of teleoperated play train the world model and policy; roughly 2,800 sampled windows receive language descriptions. The paper calls this less than 1% of data but does not define the overlapping-window denominator, so it should not be read as a percentage of unique recorded frames.

Each task is tested in 20 rollouts from neutral starting configurations. Vegetable tasks are evaluated separately for carrot and eggplant and then merged for display; choosing the wrong vegetable is a failure. The table’s 67.68% LUMOS and 63.39% HULC averages are means across the 14 displayed grouped rows, not necessarily the same weighting as all individual tasks.

LUMOS is better on many tasks but not all: lifting the pan from the table is 65% versus HULC’s 75%, and placing a vegetable in the pan is 62.5% versus 65%. Both methods reach only 35% when lifting the pan from the cabinet. Across 20 five-instruction chains, mean length is 2.05 versus 1.90.

The model-predicted table average, 75.89%, exceeds physical performance, exposing a model-to-real gap; individual simulated outcomes are not uniformly better. The paper supplies no real-world repeated-seed uncertainty. Neutral starts reduce task/start correlations but do not prove language is the only usable cue.

Task / Method LUMOS (ours) HULC [10] World Model
Lift the vegetable from the table 67.5% 62.5% 72.5%
Lift the vegetable from the pan 75% 72.5% 77.5%
Lift the vegetable from the bowl 72.5% 67.5% 75%
Lift the vegetable from the cabinet 55% 50% 70%
Lift the pan from the table 65% 75% 70%
Lift the pan from the stove 80% 70% 75%
Lift the pan from the cabinet 35% 35% 60%
Place the vegetable onto the table 90% 85% 92.5%
Place the vegetable onto the pan 62.5% 65% 77.5%
Place the vegetable into the bowl 85% 75% 80%
Place the vegetable into the cabinet 70% 60% 77.5%
Place the pan onto the table 70% 65% 80%
Place the pan onto the stove 65% 55% 85%
Place the pan into the cabinet 55% 50% 70%
Average over tasks 67.68% 63.39% 75.89%
Average no. of sequential tasks 2.05 1.90 -
TABLE II: The average success rate of language-conditioned general-purpose policies in the real world setup.

VI Conclusion and Limitations

LUMOS shows that a policy can learn language-guided tabletop skills through a fixed world model and then execute without online fine-tuning. Its strongest evidence is the within-architecture ablation, while comparisons with HULC show smaller aggregate gains.

The main limitations are uncertain model fidelity, restricted data/environment coverage, and expensive higher-resolution world modeling. Offline real data remain necessary. Increasing rollout length alone cannot solve missing dynamics or guarantee robust physical recovery.

VII Appendix

The appendix specifies world-model and policy architectures, optimization settings, baseline modifications, robot actions, and real-data collection. All parameter tables and mathematical details are retained.

VII-A Hyperparameters and Training Details of the World Model

The world model has about 40 million parameters. Its 1,024-dimensional recurrent state is concatenated with a flattened 32-by-32 categorical sample, giving 2,048 latent dimensions. Training uses batch size 50 and 50-step sequences, reconstruction/KL scale 0.3, learning rate 3×10⁻⁴, gradient clipping 100, and the remaining settings in Table III.

Training samples windows from long play episodes and randomly resets recurrent state with probability 0.01, encouraging use of current observations as well as history.

Equation 7’s printed KL-balancing weights conflict with its explanation: delta=0.8 weights the posterior-update term by 0.8 and prior-update term by 0.2, whereas the prose says the prior should update faster. The original equation and table are retained; the implemented convention is not established by this text.

Equation 7
DKL(q∥p)=δDKL(q∥sg(p))⏟posterior regularizer+(1−δ)DKL(sg(q)∥p)⏟prior regularizerD_{\text{KL}}(q\;\|\;p)=\delta\underbrace{D_{\text{KL}}(q\;\|\;sg(p))}_{\text{posterior regularizer}}+(1-\delta)\underbrace{D_{\text{KL}}(sg(q)\;\|\;p)}_{\text{prior regularizer}} (7)
Name Symbol Value
Batch size BB 50
Sequence length LL 50
Deterministic latent state dimensions — 1024
Discrete latent state dimensions — 32
Discrete latent state classes — 32
Latent dimensions kk 2048
KL loss scale β\beta 0.3
KL balancing δ\delta 0.8
RSSM reset probability ζ\zeta 0.010.01
World model learning rate — 3⋅10−43\cdot 10^{-4}
Gradient clipping — 100
Adam epsilon ϵ\epsilon 10−510^{-5}
Weight decay (decoupled) — 5⋅10−25\cdot 10^{-2}
TABLE III: hyperparameters for learning the world model.

VII-B Hyperparameters and Training Details of the Agent

The perceptual encoder maps 2,048-dimensional latent states through two fully connected layers, with hidden size 1,024 and output size 128. The text encoder is paraphrase-MiniLM-L3-v2, producing 384-dimensional sentence vectors; goal encoders map image/latent or language inputs to 32 dimensions.

Plan recognition uses two transformer blocks, eight heads, and hidden size 2,048. Plan proposal has four 2,048-unit fully connected layers. Plans use 32 categorical variables with 32 classes each. Their KL loss uses the same balancing convention whose printed direction is ambiguous in Appendix A.

Contrastive alignment normalizes latent/text features, scales their pairwise similarities by an exponentiated learned temperature, and averages cross-entropies in both retrieval directions. Only matching diagonal pairs are positive; semantically equivalent instructions elsewhere in the batch can still act as negatives under this rule.

The actor has eight 256-unit layers, about one million parameters, versus roughly 15 million in the described HULC action decoder. This is an action-decoder comparison, not total-system model size: LUMOS additionally uses the roughly 40-million-parameter world model. The critic has eight 1,024-unit layers.

Agent windows are sampled at lengths 20–32 and padded to 32; batch size is 512. Table IV specifies plan-KL scale 0.1, contrastive scale 3.0, discount 0.995, lambda 0.95, actor/critic learning rates 2×10⁻⁴/3×10⁻⁴, and slow-critic updates every 100 steps.

Lambda returns mix immediate rewards with recursive bootstrapped values. At lambda=1 the printed finite-horizon recurrence still ends in a learned value estimate, so the source’s “unbiased Monte Carlo” description requires additional terminal-value assumptions. The formula remains intact.

Name Symbol Value
Batch size BB 512
Sequence length TT 32
Minimum window length — 20
Maximum window length — 32
Feature dimensions dd 128
Latent goal dimensions gg 32
Discrete latent plan dimensions — 32
Discrete latent plan classes — 32
KL loss scale α1\alpha_{1} 0.1
Contrastive loss scale α2\alpha_{2} 3.0
KL balancing δ\delta 0.8
Discount factor γ\gamma 0.9950.995
T​D​(λ)TD(\lambda) parameter λ\lambda 0.950.95
Optimizer — Adam
Actor learning rate — 2⋅10−42\cdot 10^{-4}
Critic learning rate — 3⋅10−43\cdot 10^{-4}
Slow critic update interval — 100100
TABLE IV: hyperparameters for training the actor and critic of LUMOS.
Equations 8
logit​(xi,yj)=cos_sim​(xi,yj)⋅exp⁡(τ),∀(i,j)∈{1,2,…,M},\text{logit}(x_{i},y_{j})=\text{cos\_sim}(x_{i},y_{j})\cdot\exp(\tau),\hskip 8.50012pt\forall(i,j)\in\{1,2,\ldots,M\},
ℒcontrast\displaystyle\mathcal{L}_{\text{contrast}} =12​(∑i=1MCrossEntropy​(logits_per_latent​(i),i)\displaystyle=\frac{1}{2}\Bigg(\sum_{i=1}^{M}\text{CrossEntropy}(\textit{logits\_per\_latent}(i),i)
+∑i=1MCrossEntropy(logits_per_text(i),i))\displaystyle\hskip 8.50012pt+\sum_{i=1}^{M}\text{CrossEntropy}(\textit{logits\_per\_text}(i),i)\Bigg) (8)
Vtλ=rt+γ⁡((1−λ)​vψ​(s^t+1)+λ​Vt+1λ),Vt+Hλ=vψ​(s^t+H).V^{\lambda}_{t}=r_{t}+\gamma\left((1-\lambda)v_{\psi}(\hat{s}_{t+1})+\lambda V^{\lambda}_{t+1}\right),\hskip 8.50012ptV^{\lambda}_{t+H}=v_{\psi}(\hat{s}_{t+H}).
Refer to caption
Fig. 5: Visualization of the complete real-world robot setup. This figure highlights the static and gripper cameras, as well as the VR controller and tracking system.

VII-C Modifications to Baselines for Fair Comparisons

To match the 64×64 inputs, baseline convolution kernels/strides change from 8/4, 4/2, 3/1 to 4/2, 2/2, 2/2. Baseline images receive random shifts of 0–3 pixels with bilinear interpolation; LUMOS does not use this image augmentation.

These changes equalize input resolution but differ from original higher-resolution baseline configurations. The comparison supports the reported compute-conscious setup, not the best attainable performance of each baseline at its original resolution.

VII-D 7-DoF Action Framework

Actions contain three relative translations, three base-frame Euler-angle changes, and one gripper value. Pose values lie in [−1,1]; the actor also treats the gripper output continuously before thresholding.

The appendix gives incompatible gripper conventions: it first says −1 opens and +1 closes, then says positive outputs open and nonpositive outputs close. Both statements are preserved here as a source ambiguity; a reproducer must verify the actual controller mapping before execution. The paper also does not give physical translation/rotation scaling in this paragraph.

Equation
[δ​x,δ​y,δ​z,δ​ϕ,δ​θ,δ​ψ,gripperAction]\left[\delta x,\delta y,\delta z,\delta\phi,\delta\theta,\delta\psi,\textit{gripperAction}\right]

VII-E Data Collection Details

Three hours of play were recorded using HTC VIVE Pro teleoperation. The dataset includes proprioception, absolute commanded actions, and 200×200 static Azure Kinect and gripper-mounted FRAMOS D435e RGB observations. Training derives relative actions by differencing consecutive absolute commands and resizes images to 64×64.

Recorded data are downsampled from 30 Hz to 15 Hz. Figure 2 labels inference at 30 Hz, while the appendix does not explain rate conversion between training actions and execution. These data-collection details qualify the zero-online-learning claim; the system still learns from substantial recorded physical interaction.

Refer to caption
Fig. 6: Visualization of the real-world environment observations across two time steps. At each time step, the RGB image from the static camera is presented on the left, and the RGB image from the gripper camera is shown on the right.
Concept demonstration

Open image ↗