Literature
Display
← All papers

Discovering Hybrid World Representations with Co-Evolving Foundation Models

Jiajun Wu · Yunzhi Zhang · Hong-Xing Yu · Joy Hsu · Jiayuan Mao

· AAAI 2026 ↗

World Models4 min text · CondensedOriginal paper ↗
Contents

At a glance

This perspective proposes discovering structured world models from foundation-model priors and refining both through interaction. A hybrid representation combines explicit symbols or programs with learned continuous features. The paper defines a research agenda and potential learning mechanisms; it does not report a new trained system or quantitative experiment.

Introduction

Foundation models contain broad statistical knowledge, but extracting executable states, operators, and constraints remains difficult. The proposed direction combines the interpretability of symbolic systems with the coverage of learned representations, aiming for models that can be inspected, edited, and improved through action.

The authors explicitly describe the article as a perspective rather than a comprehensive survey. Its illustrative references are biased toward their own work.

Structured world models

The formulation assumes compositional abstractions and a Markovian physical world. A structured world model contains states—entities, attributes, and scene context—and probabilistic, action-conditioned transition functions. It may also include ontologies, relations, constraints, and physical laws.

Perceivers and generators connect these states to observations but are outside the world model under this definition. They can be foundation models, modality-specific encoders, physics engines, renderers, or other generative systems. Figure 1 preserves the proposed interaction loop.

Structured world model with hybrid representations, perceiver, generator, and an interaction loop
Figure 1: An emerging research direction is to discover structured, Markovian world models with hybrid representations from foundation models, which involves the coevolution of the representations and models via interaction.

Hybrid world representations

Hybrid structure appears at three levels: representation form, the attributes being modeled, and complementary observation or representation modalities.

Forms

Explicit language and code make knowledge interpretable and accessible to foundation-model reasoning; continuous latent features supply capacity for geometry, texture, and physics. Scene Language [37] and neuro-symbolic planning illustrate combinations of these forms.

Domains

A common representation can jointly describe distinct attributes such as appearance and physical behavior. The cited work illustrates action-conditioned prediction across different materials and object properties; the perspective does not establish that one representation already covers every domain.

Modalities

Representations can remain hybrid even within one form and domain when their components have complementary strengths, as in HybridPose [28]. The authors argue that these combinations may improve generalization and interaction efficiency through useful inductive biases.

Discovery paradigms

Three proposed approaches are foundation-model-guided learning, continual learning through interaction, and integration of multiple experts. They can be combined into multi-stage discovery pipelines.

Foundation-model-guided learning

A pretrained model can supervise structure discovery through prompting, differentiable objectives, or reinforcement learning. The supervision may be generated code, plans, scene descriptions, likelihoods, or rewards.

Prompting

Inference-only prompting turns observations and instructions into structured outputs such as semantic programs, policies, motion plans, scene layouts, or combinations of existing models. The cited examples demonstrate different components of this idea, rather than a single unified method.

Gradient-based optimization

When a generator maps a candidate representation into images or other observations differentiably, a foundation-model-based objective can refine the representation through gradients. If matching real observations are absent, generated observations or distillation-style sampling can supply a learning signal.

Reinforcement learning

Reinforcement learning provides an alternative when the foundation model, generator, or intermediate modules cannot be differentiated. Rewards can judge the resulting behavior or output without requiring gradients through every component.

Continual learning through interaction

After initializing representations and transitions from pretrained models, the system repeatedly perceives, acts, and abstracts. Learning has two coupled goals: explain observed outcomes and improve a decision policy’s task reward.

The proposed co-evolution is bidirectional. Foundation models help interpret interaction results and revise structured models; newly acquired knowledge can also update the foundation models. This is an intended learning loop, not an experimentally validated convergence claim.

Ensemble learning with multiple experts

Multiple pretrained experts can contribute complementary knowledge. Prompt-based systems may call them sequentially or hierarchically; reinforcement-learning systems can combine rewards; gradient-based integration requires suitably differentiable interfaces.

A proposed tool-orchestration approach trains a planner to select and compose expert calls. Program traces receive reward and are improved with policy gradients; executors may adapt to the resulting query distribution. Joint improvement is presented as a research direction with technical integration challenges.

Advantages of the formulation

The claimed benefits follow from the representation design and supporting examples. The paper does not provide new benchmarks, sample-complexity bounds, or proofs that the proposed combination achieves them.

Markov decision structure

Separating states, actions, and transitions aligns the model with a Markov decision process and supports action-conditioned prediction. A manipulation state might mix continuous poses with symbolic spatial predicates, with probabilistic action outcomes.

Compositional abstractions

Object-level abstractions reuse knowledge across instances, while composition combines familiar pieces in new situations. Together with the Markov assumption, this may yield a compact representation that is easier to learn. The necessary abstraction and state-sufficiency assumptions remain central.

Domain knowledge

Explicit predicates and program fragments can encode constraints or conservation laws. Learned simulators and renderers supply softer biases, and expert modules can act as callable components. This supports both rule-based and gradient-based updates.

Human interfaces

Language, code, and execution traces provide possible interfaces for inspection, editing, hypothesis testing, and debugging. These interfaces are a proposed advantage of making part of the representation explicit.

Scientific discovery

Discovering latent states and transitions may expose candidate generative rules useful for scientific investigation. The paper frames this as a path toward causal understanding; the Markovian organization alone does not verify that learned rules are causally correct.

Conclusion

Structured hybrid world models offer a proposed bridge between broad foundation-model priors and interpretable, action-conditioned reasoning. The central open problem is how to discover and validate useful structure while allowing both the representation and its supporting models to improve through interaction.

Acknowledgments

The authors acknowledge US research-agency and Stanford support. They report using foundation-model assistance, through interaction with hybrid representations, to create the figure.

Concept demonstration

Open image ↗