Literature
Display
← All papers

Factored Latent Action World Models

Zizhao Wang · Chang Shi · Jiaheng Hu · Kevin Rohling · Roberto Martín-Martín · Amy Zhang · Peter Stone

· ICML 2026 ↗

World models10 min text · CondensedOriginal paper ↗
Contents

At a glance

FLAM learns a video world model with separate latent actions for scene factors rather than one action for the entire image. Jointly learned slot representations, inverse dynamics, and forward dynamics improve multi-entity reconstruction and prediction on four simulated datasets and driving video. Downstream experiments use a small action-labeled subset to decode latent actions and pseudo-label larger demonstrations. Prediction benchmarks infer latent actions from the true future video, so their scores measure reconstruction under inferred controls, not unconstrained forecasting from an initial image alone.

1 Introduction

When several entities move independently, one scene-wide action must encode their combinations. FLAM assigns each learned factor its own action while sharing the dynamics networks and action prior across factors. This encourages reuse of individual motion patterns. Factors need not be individual visual objects: correlated entities can share a factor, and interactions remain possible through conditioning on the current scene’s slots.

Original figure
Figure 1: In multi-entity scenarios, (left) such as an intersection with three road users: (middle) a vanilla latent action model encodes the scene change with a single latent action of dimension dd, which makes learning challenging as this latent action space needs to model all |𝒜|K|\mathcal{A}|^{K} joint action combinations. (right) In contrast, FLAM decomposes the state into KK factors, each with its own latent action of dimension dK\frac{d}{K}. Additionally, we assume all latent actions share the same space (i.e., with the same prior / codebook), which reduces the learning problem to modeling the |𝒜||\mathcal{A}| actions per factor rather than their |𝒜|K|\mathcal{A}|^{K} joint combinations.

2 Related Work

Observation-only methods such as LAPO and Genie infer a latent transition control and train a predictor conditioned on it. Factored decision models motivate splitting complex state/action spaces, while Slot Attention provides a learnable decomposition of image features.

FLAM jointly trains its factorizer and dynamics on future-feature prediction. This differs from freezing a separately learned object reconstruction representation before dynamics training, as in the compared object-centric baselines. Its intended grouping is driven by transition structure rather than visual appearance alone.

3 Preliminaries

An inverse model sees current and next observations and infers a latent action; a forward model sees the current observation and that action and predicts the next observation. Restricting the action channel prevents the inverse model from simply copying the entire future. These latents capture useful transition information but need not coincide uniquely with real commands: visually indistinguishable outcomes can yield indistinguishable inferred actions.

Original figure
Figure 2: Latent action model.

4 Factored Latent Action Model (FLAM)

Training first learns a dataset-specific image tokenizer and decoder, then freezes them while learning factorized dynamics in feature space. The factorizer, inverse model, forward model, and feature aggregator train jointly in the second stage. Afterwards, inferred latents can steer generated sequences or support action pseudo-labeling for behavior cloning.

4.1 Pretrained Encoder

A convolutional encoder maps pixels to patch features, finite scalar quantization constrains their representation, and a decoder reconstructs the image. Reconstruction loss trains this tokenizer separately for each dataset. Subsequent world-model learning predicts its frozen features rather than raw pixels; decoder reconstruction quality therefore provides a useful reference for attainable visual fidelity.

Equations 1
z\displaystyle z =Encoder​(o),\displaystyle=\texttt{Encoder}(o),
zq\displaystyle z_{q} =FSQ​(z),\displaystyle=\texttt{FSQ}(z),
o^\displaystyle\hat{o} =Decoder​(zq).\displaystyle=\texttt{Decoder}(z_{q}).
ℒVQ-VAE​(o)=‖o−o^‖2.\mathcal{L}_{\texttt{VQ-VAE}}(o)=||o-\hat{o}||^{2}. (1)
Original figure
Figure 3: Two training stages of FLAM. (a) A VQ-VAE is pretrained to extract features for latent action model learning. (b) FLAM infers latent actions and makes predictions for each factor independently, with all modules trained jointly to minimize the prediction error.

4.2 Factored Latent Action Model (FLAM)

Slot Attention decomposes each frame’s features. Causal temporal attention lets each slot consult its own history, stabilizing its binding across time. For a particular factor, the shared inverse model reads all current slots plus that factor’s next slot and predicts a Gaussian latent-action posterior. A Kullback–Leibler penalty toward a unit-normal prior limits transition information.

The shared forward model predicts a factor’s next state from all current slots and that factor’s action. Thus action channels are separated while current-state interactions remain available. An aggregator attends from current image features to predicted slots, preserving static detail without forcing slots to reconstruct everything. Feature prediction error plus the summed posterior penalties trains all four modules. Shared weights make the same action space reusable; they do not establish statistical independence or physical identifiability of the learned factors.

Equations 2–4
for each slot attention iteration
st=Slot-Attn​(query=st,key=zt),\displaystyle s_{t}=\texttt{Slot-Attn}(\texttt{query}=s_{t},\texttt{key}=z_{t}), (2)
sti=Self-Attn(query=sti,key=s1:ti).\displaystyle s^{i}_{t}=\texttt{Self-Attn}(\texttt{query}=s^{i}_{t},\texttt{key}=s^{i}_{1:t}). (3)
ati=IDM(st1:K,st+1i),a^{i}_{t}=\texttt{IDM}(s^{1:K}_{t},s^{i}_{t+1}), (4)
s~ti\displaystyle\tilde{s}^{i}_{t} =Self-Attn(query=sti,key=st1:K),\displaystyle=\texttt{Self-Attn}(\texttt{query}=s^{i}_{t},\texttt{key}=s^{1:K}_{t}),
uti\displaystyle u^{i}_{t} =Cross-Attn​(query=st+1i,key=[s~ti,st+1i]).\displaystyle=\texttt{Cross-Attn}(\texttt{query}=s^{i}_{t+1},\texttt{key}=[\tilde{s}^{i}_{t},s^{i}_{t+1}]).
ati∼q⁡(ati)=𝒩⁡(μ⁡(uti),σ⁡(uti)),\displaystyle a^{i}_{t}\sim q(a^{i}_{t})=\mathcal{N}\left(\mu(u^{i}_{t}),\sigma(u^{i}_{t})\right),
Algorithm 1 FLAM
Complete original FLAM Algorithm 1, preserving equations and loop indentation.
Equations 5–6
s^t+1i=FDM(st1:K,ati).\hat{s}^{i}_{t+1}=\texttt{FDM}(s^{1:K}_{t},a^{i}_{t}). (5)
s~ti\displaystyle\tilde{s}^{i}_{t} =Self-Attn(query=sti,key=st1:K),\displaystyle=\texttt{Self-Attn}(\texttt{query}=s^{i}_{t},\texttt{key}=s^{1:K}_{t}),
s^t+1i\displaystyle\hat{s}^{i}_{t+1} =Cross-Attn​(query=sti,key=[s~ti,ati]).\displaystyle=\texttt{Cross-Attn}(\texttt{query}=s^{i}_{t},\texttt{key}=[\tilde{s}^{i}_{t},a^{i}_{t}]).
z^t+1\displaystyle\hat{z}_{t+1} =Cross-Attn(query=zt,key=s^t+11:K).\displaystyle=\texttt{Cross-Attn}(\texttt{query}=z_{t},\texttt{key}=\hat{s}^{1:K}_{t+1}).
ℒLAM​(zt,zt+1)=\displaystyle\mathcal{L}_{\texttt{LAM}}(z_{t},z_{t+1})= ∥zt+1−z^t+1∥22+\displaystyle\lVert z_{t+1}-\hat{z}_{t+1}\rVert^{2}_{2}+
β⋅∑i=1KDK​L[q(ati)||p(ati)],\displaystyle\beta\cdot\sum_{i=1}^{K}D_{KL}[q(a^{i}_{t})||p(a^{i}_{t})], (6)

4.3 Learned Latent Actions Utilization

For generation, a user selects an entity and substitutes sampled or chosen latent controls while other entities retain their inferred original actions. For policy learning, a small action-labeled demonstration subset trains a decoder from inferred latents to real actions. The decoder then pseudo-labels a larger video-only set, and a behavior-cloning policy learns from those labels. The downstream policy therefore still uses action supervision for calibration.

Equation 7
ℒπ=𝔼(o,a)∼DE​∥π⁡(o)−f⁡(a)∥.\mathcal{L}_{\pi}=\mathbb{E}_{(o,a)\sim D_{E}}\lVert\pi(o)-f(a)\rVert. (7)

5 Experiments

Experiments use MultiGrid, three Procgen games, and front-camera nuPlan video. Each dataset has its own pretrained encoder; compared methods share that encoder and, where applicable, component architectures. FLAM and PlaySlot divide total latent-action dimensionality across factors to match the scene-wide baselines’ total dimension. Ground-truth-action World Model is evaluated only in simulation. The study demonstrates dataset-specific models, not one universal pretrained world model.

5.1 World Model Accuracy

Ground-truth evaluation videos supply the inverse model’s latent controls, followed by ten autoregressive prediction steps. Across five datasets, FLAM averages 34.9 PSNR, 0.890 SSIM, 0.051 LPIPS, and 419.6 FVD. It is best or tied on most individual comparisons, although AdaWorld’s Bigfish FVD is slightly lower, 63.3 versus 63.6. On nuPlan, FLAM’s PSNR is 19.7 versus 18.1 for AdaWorld and 16.0 for Genie; frozen reconstruction reaches 21.7.

Beating the action-conditioned World Model does not imply that inferred controls are more informative than complete physical actions: Procgen supplies only the player action, while FLAM’s posterior can encode other entities’ realized future changes; MultiGrid’s action baseline uses a monolithic joint token. Generation examples show factor-specific edits with other inferred controls fixed. They establish controllability in the examples rather than unrestricted intervention validity in the real world.

Table 1: Prediction performance of all methods on simulation and real-world datasets. Recon reports the reconstruction quality of the pretrained VQ-VAE, serving as a reference upper bound for prediction performance.
Dataset Metric FLAM (ours) AdaWorld Genie World Model PlaySlot SlotFormer Recon
Average PSNR (↑\uparrow) 34.9 24.7 28.1 N/A 29.1 19.9 37.1
SSIM (↑\uparrow) 0.890 0.862 0.859 N/A 0.847 0.814 0.909
LPIPS (↓\downarrow) 0.051 0.096 0.091 N/A 0.120 0.223 0.035
FVD (↓\downarrow) 419.6 749.5 717.5 N/A 858.0 1597.7 215.1
MultiGrid PSNR (↑\uparrow) 56.5 24.0 33.9 56.1 56.5 24.9 56.5
SSIM (↑\uparrow) 0.944 0.909 0.933 0.943 0.944 0.912 0.944
LPIPS (↓\downarrow) 3e-4 0.08 0.014 6e-4 3e-4 0.056 3e-4
FVD (↓\downarrow) 2.9 913.6 177.5 5.5 2.9 451.6 2.5
Bigfish PSNR (↑\uparrow) 30.8 30.3 28.5 23.9 26.0 19.6 34.4
SSIM (↑\uparrow) 0.983 0.982 0.974 0.949 0.962 0.934 0.992
LPIPS (↓\downarrow) 0.011 0.012 0.021 0.049 0.038 0.221 0.004
FVD (↓\downarrow) 63.6 63.3 106.5 168.9 227.4 1455 16.8
Leaper PSNR (↑\uparrow) 38.1 23.3 35.0 24.7 27.5 21.4 40.3
SSIM (↑\uparrow) 0.976 0.924 0.973 0.924 0.945 0.913 0.977
LPIPS (↓\downarrow) 0.002 0.072 0.005 0.044 0.031 0.164 0.001
FVD (↓\downarrow) 12.1 348.1 27.3 127.3 361.5 1151 6.27
Starpilot PSNR (↑\uparrow) 29.3 27.8 27.0 25.8 18.3 19.8 32.6
SSIM (↑\uparrow) 0.971 0.965 0.962 0.959 0.886 0.907 0.980
LPIPS (↓\downarrow) 0.015 0.023 0.028 0.040 0.187 0.166 0.006
FVD (↓\downarrow) 73.4 113.4 141.0 194.9 792.0 720.9 24.7
nuPlan PSNR (↑\uparrow) 19.7 18.1 16.0 N/A 17.2 14.0 21.7
SSIM (↑\uparrow) 0.577 0.529 0.453 N/A 0.498 0.405 0.650
LPIPS (↓\downarrow) 0.229 0.292 0.389 N/A 0.344 0.508 0.163
FVD (↓\downarrow) 1946 2309 3135 N/A 2906 4210 1025

5.2 FLAM State Representation

With four MultiGrid agents and three slots, two agents forced to take identical actions can share a factor. With four independently acting agents and four slots, FLAM’s adapted disentanglement/completeness/informativeness scores are 0.91/0.91/0.93, compared with PlaySlot’s 0.81/0.81/0.86. These scores use position probes with per-frame optimal matching, so they do not independently prove globally fixed semantic slot identity. Reducing slots to two lowers PSNR to 53.4; four, eight, and sixteen slots all reach 56.5 in this four-agent setting.

Table 2: Factor-agent correspondence in the MultiGrid environments, evaluated by DCI (disentanglement, completeness, informativeness) metrics.
FLAM(ours) PlaySlot SlotFormer Random
D (↑\uparrow) 0.91 0.81 0.85 0.32
C (↑\uparrow) 0.91 0.81 0.85 0.30
I (↑\uparrow) 0.93 0.86 0.90 0.10
Table 3: Prediction performance of FLAM on the MultiGrid dataset with 4 agents and different number of factors KK.
KK 2 4 8 16
PSNR (↑\uparrow) 53.4 56.5 56.5 56.5
SSIM (↑\uparrow) 0.943 0.944 0.944 0.944
LPIPS (↓\downarrow) 4e-4 3e-4 3e-4 3e-4
FVD (↓\downarrow) 3.62 2.87 2.80 2.73

5.3 Latent Action Policy Learning

An expert supplies one million demonstration frames per environment. A decoder trained on one thousand or ten thousand labeled frames pseudo-labels the rest. With ten thousand labels, episodic returns rise from 3.6 to 8.8 in Bigfish and from 6.4 to 9.1 in Starpilot versus labeled-only cloning. With one thousand labels, gains are small and nonuniform: Bigfish rises from 1.0 to 1.8, but Starpilot falls from 1.9 to 1.8. Limited decoder accuracy can make extra pseudo-labels unhelpful.

Table 4: Behavior cloning policy performance, evaluated by mean episodic return.
Dataset FLAM vanilla BC random
Bigfish (1k) 1.8 1.0 0.9
Bigfish (10k) 8.8 3.6
Starpilot (1k) 1.8 1.9 1.1
Starpilot (10k) 9.1 6.4

5.4 Ablation Studies

Temporal attention is important for stable factor assignments: removing it reduces MultiGrid PSNR from 56.5 to 51.6 and informativeness from 0.93 to 0.20. Allowing every predicted slot to use every latent action largely preserves prediction quality but weakens disentanglement and position information. The KL sweep is stable from 10⁻⁵ through 10⁻³; at 10⁻², PSNR collapses to 24.9 and FVD rises to 479.9. Robustness therefore has a clear tested limit.

6 Conclusion

Factorwise latent controls and joint dynamics-driven representation learning improve the tested multi-entity video models. Remaining limitations include dataset-specific tokenizers, restricted decoder fidelity, and no universal model. Evaluation uses posterior actions inferred from future video, and quantitative tables do not provide run-to-run uncertainty. More expressive diffusion/flow decoders and shared pretrained tokenizers are proposed extensions; physical closed-loop robot or autonomous-driving control is not demonstrated.

Impact Statement

The impact statement describes the work as advancing machine learning and does not identify a specific societal consequence for additional discussion.

Acknowledgments

The acknowledgments list the participating University of Texas laboratories, public and industry research funding, and a disclosure that Peter Stone is compensated as Sony AI’s Chief Scientist. The authors state that the university reviewed the arrangement under its research-objectivity policy.

Appendix A Dataset Details

Unless otherwise specified, data is split 80/10/10 by frame count for training, validation, and test. Inputs are normalized to zero–one. The appendix does not fully specify scene-level separation for every video-model dataset; policy demonstrations receive a separate disjoint-level protocol.

MultiGrid (Simulation).

MultiGrid uses eight-by-eight worlds rendered at 128-by-128 pixels, with two to sixteen simultaneously moving agents. The default experiment has four agents and four factors.

Procgen (Simulation).

The three Procgen games disable rendered backgrounds and use 224-by-224 images. This simplifies visual variation compared with the original full-background game distribution.

nuPlan (Real-world).

nuPlan contributes real driving video from front-facing cameras, resized to 224-by-224 pixels. Sixteen factors represent vehicles, pedestrians, and other scene changes. This evaluates video modeling rather than a driving policy.

Appendix B Implementation Details

Simulation uses an IMPALA-style encoder/decoder and nuPlan uses MAGVIT-v2 components. Finite scalar quantizers have 1,024 and 16,384 codes, respectively. Patch-feature dimension is 128; attention width is 256 with eight heads and two simulation or three real-video layers. The default latent regularizer is 2×10⁻⁴. Both training phases use AdamW at 10⁻⁴ learning rate, with batches of sixty-four for the tokenizer and thirty-two for dynamics.

Table 5: Experiment hyperparameters. Simulation datasets are Multigrid and Procgen, while the real-world dataset is nuPlan.
Simulation Real-world
Encoder architecture IMPALA MAGVIT-v2
Decoder architecture IMPALA MAGVIT-v2
Image size 128 (Multigrid), 224 (Procgen) 224
dzd_{z} 128 128
Tokenizer quantizer FSQ FSQ
Tokenizer codebook size 1024 16384
Tokenizer codebook levels [4,4,4,4,4] [4,4,4,4,4,4,4]
Optimizer AdamW AdamW
Learning rate 1e-4 1e-4
Batch size 64 64
Attention model dimension 256 256
Number of attention heads 8 8
Number of attention layers 2 3
Number of factors KK 4 (Multigrid), 16 (Procgen) 16
KL regularization coefficient β\beta 2e-4 2e-4
Optimizer AdamW AdamW
Learning rate 1e-4 1e-4
Batch size 32 32

Computational Cost

Patch–slot attention scales as O(NK), and interactions among K slots as O(K²) per timestep, versus O(N²) for dense patch attention. This comparison explains a potential saving when slots are much fewer than patches; it is not a measured end-to-end runtime, memory, or scaling-law guarantee.

Appendix C Experiments

The appendix supplies qualitative prediction sequences, control edits, representation probes, policy-data details, and architecture/regularizer ablations. Every rollout panel, slot-binding mask, and marked prediction error is retained.

C.1 World Model Accuracy

World-model evaluation separates posterior-conditioned prediction accuracy from qualitative latent-control interventions. Both use the learned factorized model but support different claims.

Prediction Accuracy

All datasets use ten-step autoregressive rollouts driven by actions inferred from the true video. Figures compare generated frames and slot-binding masks. Error annotations show remaining occlusion, projectile, and appearance failures even in FLAM, so accurate examples should not be read as perfect dynamics.

Scaling with the number of entities

The MultiGrid scaling study varies entity count while matching total latent capacity against AdaWorld and matching Genie’s discrete capacity to joint combinations. For eight agents, FLAM uses eight thirty-two-dimensional actions and AdaWorld one 256-dimensional action. FLAM degrades less as the tested scene becomes more crowded; the evidence is an empirical grid-world comparison.

Original figure
Original figure
Original figure
Original figure
Figure 4: Prediction performance variation along with increasing number of entities in the scene.

Controllable video generation

Control examples replace one entity’s latent actions while retaining the others’ inferred controls. A two-dimensional UMAP plot shows action-associated clusters. Forward motion blocked by an obstacle can overlap with staying still, because the visible transition is the same. This illustrates the distinction between inferred visual effects and uniquely recovering the command that caused them.

Refer to caption
Figure 5: UMAP projection of the learned latent actions on the MultiGrid dataset. Each point corresponds to a latent action inferred by the IDM for one factor on a single transition from the observation-only dataset. Points are colored by the ground-truth actions taken by the corresponding agent at that transition (action labels are used only for visualization and are not used for training).
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 6: FLAM allows for changing / editing the motion of one entity without affecting the others. Here we show controllable video generation on MultiGrid through specifying latent actions for one of the movable agents, while other agents follow their original inferred latent actions.

C.2 FLAM State Representation

Representation analysis measures whether factors separate independently acting agents and group deliberately correlated ones. The quantitative probe uses positions, while visualizations show binding masks and behavior over time.

Correlated Entities

Two of four agents execute identical commands at every step, and three slots are available. Example rollouts show the correlated pair sharing a slot despite varying positions, supporting a role for common dynamics in grouping.

Refer to caption
Refer to caption
Figure 7: Two example rollouts where two entities (green and red ones) share the same actions. FLAM consistently maps the two correlated agents to the same factor.

Independent Entities

A linear classifier predicts agent grid position from each slot. Hungarian matching minimizes position-label negative log likelihood to pair slots and agents during probe training. Normalized entropies of soft assignment probabilities define the adapted disentanglement and completeness scores; informativeness is matched position classification accuracy. Evaluation averages over slots/agents and held-out frames. The metrics are inspired by Eastwood and Williams, with this task-specific matching construction.

Equations 8
τ=argminτ∈SK∑i=1K−logf(yτ⁡(i)|si),\tau=\arg\min_{\tau\in S_{K}}\sum_{i=1}^{K}-\log f(y^{\tau(i)}|s^{i}),
Pdisi​j\displaystyle P^{ij}_{\text{dis}} =f⁡(yj|si)∑k=1Kf⁡(yk|si),\displaystyle=\frac{f(y^{j}|s^{i})}{\sum_{k=1}^{K}f(y^{k}|s^{i})},
Pcompi​j\displaystyle P^{ij}_{\text{comp}} =f⁡(yj|si)∑k=1Kf⁡(yj|sk).\displaystyle=\frac{f(y^{j}|s^{i})}{\sum_{k=1}^{K}f(y^{j}|s^{k})}.
D\displaystyle D =1+∑j=1KPdisi​j​logK​Pdisi​j,\displaystyle=1+\sum_{j=1}^{K}P^{ij}_{\text{dis}}\log_{K}P^{ij}_{\text{dis}},
C\displaystyle C =1+∑i=1KPcompi​j​logK​Pcompi​j.\displaystyle=1+\sum_{i=1}^{K}P^{ij}_{\text{comp}}\log_{K}P^{ij}_{\text{comp}}.
I=1K∑i=1K𝟙[argmaxcpi(c)=yτ⁡(i)].I=\frac{1}{K}\sum_{i=1}^{K}\mathbbm{1}\left[\arg\max_{c}p_{i}(c)=y^{\tau(i)}\right]. (8)

Number of factors K

Sweeping two, four, eight, and sixteen factors on four independent agents shows degradation when slots are insufficient and little prediction change once enough are available. Choosing enough slots for a varying population is a suggested practice; robustness beyond the tested counts and visual settings is not established.

Refer to caption
(a) K=2K=2
Refer to caption
(b) K=4K=4
Refer to caption
(c) K=8K=8
Refer to caption
(d) K=16K=16
Figure 8: Ablation of FLAM on 4-agent MultiGrid dataset, with K∈{2,4,8,16}K\in\{2,4,8,16\}. When KK is less than the number agents, FLAM assigns two agents to each factor to maximize latent action utilization. When KK is equal to or greater than the number of agents, FLAM consistently assigns each entity to a distinct factor, even with excessive KK, demonstrating FLAM’s robustness to over-specifying KK.

C.3 Latent Action Policy Learning

Experts train with Phasic Policy Gradient for twenty-five million environment steps. Demonstrations for policy training, validation, and testing use disjoint Procgen levels, although the expert itself trains on all levels. FLAM-assisted and labeled-only cloning share the policy architecture and training setup; the former additionally uses inferred labels on the large demonstration set.

C.4 Ablation Studies

Ablations separately change temporal slot binding, access to other factors’ latent controls, and the strength of the latent bottleneck. Prediction and representation scores can move differently.

Factorizer architecture

Replacing factorizer temporal attention with previous-slot initialization makes bindings less consistent. FVD worsens from 2.9 to 36.6, while disentanglement/completeness/informativeness fall from 0.91/0.91/0.93 to 0.33/0.23/0.20. The retained rollout visualization shows the corresponding late-horizon failures.

Latent action model architecture

Global-Coupled FLAM gives each inverse prediction all next-frame slots and each forward prediction all latent actions. PSNR remains 56.3, near 56.5 for FLAM, but disentanglement falls to 0.79 and informativeness to 0.58. Completeness increases to 1.00, so the table does not show every representation score worsening. Accurate image prediction alone does not ensure separate entity-control channels.

Table 6: Prediction performance of FLAM and its ablative variants on the MultiGrid dataset.
Metric FLAM (ours) FLAM w/o Factorizer Temporal Attention Global-Coupled FLAM
PSNR (↑\uparrow) 56.5 51.6 56.3
SSIM (↑\uparrow) 0.944 0.941 0.944
LPIPS (↓\downarrow) 3e-4 3e-3 4e-4
FVD (↓\downarrow) 2.9 36.6 3.72
Table 7: Factor-agent correspondence of FLAM and its ablative variants on the MultiGrid dataset, evaluated by DCI (disentanglement, completeness, informativeness) metrics.
FLAM(ours) FLAM w/o Factorizer Temporal Attention Global-Coupled FLAM Random
D (↑\uparrow) 0.91 0.33 0.79 0.32
C (↑\uparrow) 0.91 0.23 1.00 0.30
I (↑\uparrow) 0.93 0.20 0.58 0.10
Refer to caption
(a) Original FLAM
Refer to caption
(b) FLAM without factorizer temporal attention
Refer to caption
(c) Global-Coupled FLAM
Figure 9: Architecture ablations of FLAM on the 4-agent MultiGrid dataset. Removing temporal attention from the factorizer leads to temporally inconsistent factor assignments and degraded long-horizon predictions (notably in the last few timestamps). In contrast, the Global-Coupled variant can produce accurate predictions but tends to assign multiple entities into the same factor, resulting in poorer disentanglement than the original FLAM implementation.

KL regularization coefficient β

The KL penalty trades information capacity against prediction. Coefficients from 10⁻⁵ to 10⁻³ retain approximately 56.3–56.5 PSNR. At 10⁻², visual prediction degrades sharply. The default 2×10⁻⁴ lies within the successful tested range.

Table 8: Prediction performance of FLAM on the MultiGrid dataset with different KL regularization coefficient β\beta values.
Metric β=1​e−5\beta=1e-5 β=1​e−4\beta=1e-4 β=1​e−3\beta=1e-3 β=1​e−2\beta=1e-2
PSNR (↑\uparrow) 56.5 56.5 56.3 24.9
SSIM (↑\uparrow) 0.944 0.944 0.944 0.911
LPIPS (↓\downarrow) 3e-4 3e-4 4e-4 0.057
FVD (↓\downarrow) 2.84 3.40 3.78 479.9
Refer to caption
Figure 10: Prediction rollouts of all methods on the MultiGrid dataset, with inaccuracies highlighted by red boxes (except for SlotFormer, where the errors are visually apparent). When applicable, we also show the slot-patch binding masks from slot attention, displayed on top of the ground-truth observations. FLAM maintains consistent factor-entity bindings over time and generates accurate predictions. In contrast, AdaWorld and SlotFormer fail to generate accurate predictions from the start, and Genie makes mistakes at a few steps. Meanwhile, PlaySlot and SlotFormer rely on object-centric representations learned from reconstruction which incorrectly bind the red and purple entities to the same slot.
Refer to caption
Figure 11: Prediction rollouts of all methods on the Bigfish dataset, with inaccuracies highlighted by red circles (except for SlotFormer, where the errors are visually apparent). When applicable, we also show the slot-patch binding masks from slot attention, displayed on top of the ground-truth observations. FLAM maintains consistent factor-entity bindings over time, e.g., the brown slot consistently attends to the green fish controlled by the player. In terms of prediction quality, FLAM and Genie only make errors on the occluded fish. AdaWorld to track the speeds of the circled yellow and red fish, which move more slowly than in the ground truth. The World Model baseline also struggles to capture the motion of non-controllable fish, since only the player’s action is available. PlaySlot and SlotFormer rely on reconstruction-based object-centric representations that do not yield consistent slot assignments. As a result, PlaySlot fails to preserve consistent shapes and colors for multiple fish.
Refer to caption
Figure 12: Prediction rollouts of all methods on the Leaper dataset, with inaccuracies highlighted by red boxes (except for PlaySlot and SlotFormer, where the errors are visually apparent). When applicable, we also show the slot-patch binding masks from slot attention, displayed on top of the ground-truth observations. FLAM generates accurate predictions, in particular capturing how the frog jumps between logs. Genie and World Model achieve the second best prediction quality, but fails to model the frog’s jumps in some timestamps. In contrast, AdaWorld and PlaySlot generate inaccurate predictions from the start, either removing the frog and cars from the scene or hallucinating non-existent.
Refer to caption
Figure 13: Prediction rollouts of all methods on the Starpilot dataset, with inaccuracies highlighted by red circles (except for PlaySlot and SlotFormer, where the errors are visually apparent). When applicable, we also show the slot-patch binding masks from slot attention, displayed on top of the ground-truth observations. FLAM maintains a consistent binding for the player in the bottom-left corner (shown in cyan) and generates accurate predictions, except for the enemy bullet heading toward the player. Other entities (e.g., bullets, enemies, and meteors) move at constant speed, so even with less consistent bindings, FLAM can still achieve reasonable predictions. In contrast, AdaWorld, Genie, and the World Model baseline generate inaccurate predictions from the start, frequently shifting enemy appearances and missing the enemy bullet toward the player.
Refer to caption
Figure 14: Prediction rollouts of all methods on the nuPlan dataset, with inaccuracies highlighted by red boxes (except for SlotFormer, where the errors are visually apparent). Models are conditioned on a fixed history window of size w=5w=5. When applicable, we also show the slot-patch binding masks from slot attention, displayed on top of the ground-truth observations. Compared to the baselines, FLAM generates relatively accurate predictions for the grey car moving from the center to the right. In contrast, AdaWorld and Genie fail to maintain a consistent appearance for this moving car. Meanwhile, PlaySlot and SlotFormer fail to model the dynamics and largely ignore the cars in the scene.
Concept demonstration

Open image ↗