Literature
Display
← All papers

A Path Towards Autonomous Machine Intelligence

Yann LeCun

· Position paper (OpenReview) ↗

World models10 min text · CondensedOriginal paper ↗
Contents

At a glance

Imagine learning how a kitchen works by watching, then planning how to prepare a meal before moving a robot. LeCun proposes an architecture for that kind of agent. A world model predicts useful aspects of possible futures; a cost model scores them; an actor searches for actions; memory and a configurator supply context and goals. JEPA learns by predicting representations instead of reconstructing every detail of an observation. A hierarchy would support longer, more abstract plans. This is a research proposal that identifies important missing pieces, especially training the hierarchy and configurator. It does not report a completed autonomous system.

1 Prologue

The paper assembles existing ideas into a proposed research program. It explicitly does not claim that every component is new. The central question is how self-supervised world models, planning and intrinsic objectives could fit together in one learnable architecture.

2 Introduction

The three goals are to learn mainly through observation, reason through methods compatible with gradient-based learning, and represent plans at several levels of detail and time. The proposal connects these goals through a learned world model and hierarchical prediction.

2.1 Learning World Models

A world model predicts what may happen and fills in information that perception does not reveal. Such knowledge could let an agent learn a new task with fewer costly real-world trials: imagine a possible action, predict its consequence, and avoid an obviously poor choice before executing it.

2.2 Humans and Animals learn Hierarchies of Models

The developmental examples motivate a hierarchy: simpler perceptual regularities may support objects, physical relationships and more abstract concepts. A configurable shared world-model engine is proposed to reuse this knowledge across tasks. The infant timeline and brain interpretation are motivations, not an implementation or proof that a machine must learn in exactly that order.

Figure 1: This chart, (courtesy of Emmanuel Dupoux), indicates at what age infants generally acquire various concepts about how the world works. It is consistent with the idea that abstract concepts, such as the fact that objects are subject to gravity and inertia, are acquired on top of less abstract concepts, like object permanence and the assignment of objects to broad categories. Much of this knowledge is acquired mostly by observation, with very little direct intervention, particularly in the first few weeks and months.
Figure 1: This chart, (courtesy of Emmanuel Dupoux), indicates at what age infants generally acquire various concepts about how the world works. Original figure and complete caption.

3 A Model Architecture for Autonomous Intelligence

The architecture has six connected roles. Perception estimates state; the world model predicts possible futures; a cost module evaluates them; the actor proposes and improves actions; short-term memory stores relevant experience; a configurator chooses tasks and subgoals. The modules are intended to exchange gradients, enabling planning through the learned model.

Figure 2: A system architecture for autonomous intelligence. All modules in this model are assumed to be “differentiable”, in that a module feeding into another one (through an arrow connecting them) can get gradient estimates of the cost’s scalar output with respect to its own output. The configurator module  takes inputs (not represented for clarity) from all other modules and configures them to perform the task at hand. The perception module  estimates the current state of the world. The world model module  predicts possible future world states as a function of imagined actions sequences proposed by the actor. The cost module  computes a single scalar output called “energy” that measures the level of discomfort of the agent. It is composed of two sub-modules, the intrinsic cost, which is immutable (not trainable) and computes the immediate energy of the current state (pain, pleasure, hunger, etc), and the critic, a trainable module that predicts future values of the intrinsic cost. The short-term memory module  keeps track of the current and predicted world states and associated intrinsic costs. The actor module  computes proposals for action sequences. The world model and the critic compute the possible resulting outcomes. The actor can find an optimal action sequence that minimizes the estimated future cost, and output the first action in the optimal sequence. See Section 3 for details.
Figure 2: A system architecture for autonomous intelligence. Original figure and complete caption.

3.1 Typical Perception-Action Loops

The agent can act through a fast learned policy or spend additional computation planning. These two modes correspond to different uses of the same perception, memory and world-model components.

3.1.1 Mode-1: Reactive behavior

Mode 1 maps an estimated state directly to an action using a policy. It is fast because it does not search through imagined futures. The observed result can still train the world model. Learning only by trying perturbed actions in the physical world can be slow and costly.

Figure 3: Mode-1 perception-action episode. The perception module estimates the state of the world  s [0] = Enc( x ) . The actor directly computes an action, or a short sequence of actions, through a policy module  a [0] =  A ( s [0]) . This reactive process does not make use of the world model nor of the cost. The cost module computes the energy of the initial state  f [0] = C( s [0])  and stores the pairs  ( s [0] , f [0])  in the short-term memory. Optionally, it may also predict the next state using the world model  s [1] = Pred( s [0] , a [0]) , and the associated energy  f [0] = C( s [0])  so that the world model can be adjusted once the next observation resulting from the action taken becomes available.
Figure 3: Mode-1 perception-action episode. Original figure and complete caption.
Equations Mathematical context · source page 9–Mathematical context · source page 10
(Mathematical context · source page 9)Original mathematical context on PDF page 9, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 10)Original mathematical context on PDF page 10, including inline formulas and their surrounding paragraphs.

3.1.2 Mode-2: reasoning and planning using the world model

Mode 2 proposes an action sequence, rolls it forward through the world model, scores the predicted states, and improves the actions. After planning, the agent executes only the first action or short segment, observes the result, and replans. This is the familiar model-predictive-control loop with a learned predictive model.

Figure 4: Mode-2 perception-action episode. The perception module estimates the state of the world s [0] . The actor proposes a sequence of actions  a [0] , a [1] , . . . , a [ t ] , a [ t  + 1] , . . . , a [ T ] . The world model recursively predicts an estimate of the world state sequence using  s [ t  + 1] = Pred( s [ t ] , a [ t ]) . The cost C ( s [ t ])  computes an energy for each predicted state in the sequence, the total energy being the sum of them. Through an optimization or search procedure, the actor infers a sequence of actions that minimizes the total energy. It then sends the first action in the sequence (or the first few actions) to the effectors. This is, in effect, an instance of classical model-predictive control with recedinghorizon planning. Since the cost and the model are differentiable, gradient-based methods can be used to search for optimal action sequences as in classical optimal control. Since the total energy is additive over time, dynamic programming can also be used, particularly when the action space is small and discretized. Pairs of states (computed by the encoder or predicted by the predictor) and corresponding energies from the intrinsic cost and the trainable critic are stored in the short-term memory for subsequent training of the critic.
Figure 4: Mode-2 perception-action episode. Original figure and complete caption.
Equation Mathematical context · source page 11
(Mathematical context · source page 11)Original mathematical context on PDF page 11, including inline formulas and their surrounding paragraphs.

3.1.3 From Mode-2 to Mode-1: Learning New Skills

A slow planner can teach a fast policy: use the planner’s selected actions as supervised targets. The policy can then act directly or initialize later searches. This is amortized inference—pay the search cost during learning so that future decisions need less computation.

Figure 5: Training a reactive policy module from the result of Mode-2 reasoning. Using Mode-2 is onerous, because it mobilizes all the resources of the agent for the task at hand. It involves running the world model for multiple time steps repeatedly. This diagram depicts how to train a policy module A ( s [ t ])  to approximate the action that results from Mode-2 optimization. The system first operates in Mode-2 and produces an optimal sequence of actions  (ˇ a [0] , . . . ,  ˇ a [ T ]) . Then the parameters of the policy module are adjusted to minimize a divergence  D (ˇ a [ t ]) , A ( s [ t ]))  between the optimal action and the output of the policy module. This results in a policy module that performs  amortized inference , and produces an approximation for a good action sequence. The policy module can then be used to produce actions reactively in Mode-1, or to initialize the action sequence prior to Mode-2 inference and thereby accelerate the optimization.
Figure 5: Training a reactive policy module from the result of Mode-2 reasoning. Original figure and complete caption.
Equations Source display–Mathematical context · source page 12
(Source display)Reactive policy rollout through the world model (unnumbered source display).
(Mathematical context · source page 12)Original mathematical context on PDF page 12, including inline formulas and their surrounding paragraphs.

3.1.4 Reasoning as Energy Minimization

Here, reasoning means searching for variables that satisfy constraints or reduce an energy score. Gradient descent is an option when the model is smooth; discrete choices may require other search methods. Calling reasoning energy minimization is a formulation, not a guarantee that the resulting optimization will be easy.

3.2 The Cost Module as the Driver of Behavior

The cost combines an intrinsic part with a learned critic. Intrinsic terms encode basic objectives such as avoiding damage or preserving energy; the critic predicts longer-term consequences and task-specific subgoals. A configurator can adjust weights among cost components. Keeping the intrinsic objective fixed is proposed to prevent unwanted objective drift, but it is not by itself a demonstrated safety guarantee.

Figure 6: Architecture of the cost module. The cost module comprises the intrinsic cost module which is immutable  IC i ( s )  (left) and the critic or Trainable Cost  TC j ( s )  (right), which is trainable. Both IC and TC are composed of multiple submodules whose output energies are linearly combined. Each submodule imparts a particular behavioral drive in the agent. The weights in the linear combination,  u i  and  v j , are determined by the configurator module and allow the agent to focus on different subgoals at different times.
Figure 6: Architecture of the cost module. Original figure and complete caption.
Equations 1, 2–Mathematical context · source page 14
(1, 2)Cost decomposition, source equations 1–2.
(3)Trainable critic decomposition, source equation 3.
(Mathematical context · source page 13)Original mathematical context on PDF page 13, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 14)Original mathematical context on PDF page 14, including inline formulas and their surrounding paragraphs.

3.3 Training the Critic

Store states, times and intrinsic costs in memory. To train the critic, retrieve an earlier state and a later cost, then adjust the critic to predict that future consequence. More elaborate targets could include discounted future costs. A differentiable key–value memory allows approximate retrieval and interpolation rather than requiring an exact lookup.

Figure 7: Training the critic. During planning episodes, the intrinsic cost module stores triplets (time, state, intrinsic energy):  ( τ, s τ , IC ( s τ ))  into the associative short-term memory. During critic training episodes, the critic retrieves a past state vector  s τ , together with an intrinsic energy at a later time  IC( s τ + δ ) . In the simplest scenario, the critic adjusts its parameters to minimize a divergence measure between the target  IC( s tau + δ )  and the predicted energy  C( s τ ) . In more complex schemes, it may use combinations of future intrinsic energies as targets. Note that the state sequence may contain information about the actions planned or taken by the agent.
Figure 7: Training the critic. Original figure and complete caption.
Equation Mathematical context · source page 15
(Mathematical context · source page 15)Original mathematical context on PDF page 15, including inline formulas and their surrounding paragraphs.

4 Designing and Training the World Model

The world model must learn from sufficiently diverse experience, represent several plausible outcomes, and predict at multiple timescales. These requirements interact: the agent’s own behavior affects the data it later observes, and long-horizon prediction often needs a different abstraction from short-horizon motion prediction.

4.1 Self-Supervised Learning

Self-supervision asks whether related observations fit together. An energy score can rate a proposed video continuation without explicitly generating every possible continuation. The proposed representation should retain useful information while making prediction easier. Learning objects, geometry and physical relationships through this process is presented as a hypothesis to test.

Figure 8: Self-Supervised Learning (SSL) and Energy-Based Models (EBM). SSL is a learning paradigm in which a learning system is trained to “fill in the blanks”, or more precisely to capture the dependencies between observed parts of the input and possibly unobserved parts of the input. Part of the input signal is observed and denoted  x  (in pink), and part of the input signal is either observed or unobserved and denoted  y  (in blue). In a temporal prediction scenario,  x  represents past and present observations, and  y  represent future observations. In a general pattern completion scenario, various parts of the input may be observed or unobserved at various times. The learning system is trained to capture the dependencies between  x  and  y  through a scalar-valued  energy function  F ( x, y ) that takes low values when  x  and  y  are consistent or compatible, and higher values if  x  and  y  are inconsistent or incompatible. In a video prediction scenario, the system would produce a low energy value if a video clip  y  is a plausible continuation of the video clip  x . This  energy-based model  (EBM) formulation enables the system to represent multi-modal dependencies in which multiple values of  y (perhaps an infinite set) may be compatible with a given  x . In the right panel, an energy landscape is represented in which dark discs represent data points, and closed lines represents contours (level sets) of the energy function.
Figure 8: Self-Supervised Learning (SSL) and Energy-Based Models (EBM). Original figure and complete caption.
Equation Mathematical context · source page 17
(Mathematical context · source page 17)Original mathematical context on PDF page 17, including inline formulas and their surrounding paragraphs.

4.2 Handling Uncertainty with Latent Variables

A latent variable represents information about the target that the context does not determine, such as camera motion between two views. Inference finds a latent setting that makes the pair compatible; minimizing over it produces a score for the pair. Latents can express alternative outcomes, but their capacity must be limited so they cannot simply encode any answer.

Figure 9: Latent-Variable Energy-Based Model (LVEBM). To evaluate the degree of compatibility between  x  and  y , an EBM may need the help of a latent variable z . The latent variable can be seen as parameterizing the set of possible relationships between an  x and a set of compatible  y . Latent variables represent information about  y  that cannot be extracted from  x . For example, if  x  is a view of an object, and  y  another view of the same object,  z  may parameterize the camera displacement between the two views. Inference consists in finding the latent that minimizes the energy  ˇ z  = argmin z ∈Z  E w ( x, y, z ) . The resulting energy  F w ( x, y ) =  E w ( x, y,  ˇ z ) only depends on  x  and  y . In the dual view example, inference finds the camera motion that best explains how  x  could be transformed into  y .
Figure 9: Latent-Variable Energy-Based Model (LVEBM). Original figure and complete caption.
Equations 4–Mathematical context · source page 20
(4)Latent-variable inference, source equation 4.
(5)Minimized latent-variable energy, source equation 5.
(Mathematical context · source page 19)Original mathematical context on PDF page 19, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 20)Original mathematical context on PDF page 20, including inline formulas and their surrounding paragraphs.

4.3 Training Energy-Based Models

Training must do more than make observed pairs have low energy. A constant representation would make every pair look compatible and solve that narrow objective uselessly. Contrastive methods push up scores for incompatible alternatives. Architectural constraints and regularization instead limit the extent of low-energy regions. Sampling useful negatives can be expensive; avoiding them requires a credible alternative constraint.

Figure 10: A few standard architectures and their capacity for collapse. (a) Deterministic generative architecture: cannot collapse because it can only produce a single output. For a given  x , only one value of  y  may have zero energy:  y  =  ̃ y . Other values of  y  will have higher energy if  D ( u,   ̃ y )  is larger than zero when  y  ̸ =  ̃ y . (b) Non-deterministic generative architecture: can collapse when the latent variable has excessive information capacity. If for a given  x  and for all  y  there exists a  z  that produces zero prediction energy (e.g. if  z  has the same or higher dimension as  y ), the entire  y  space will have low energy. The information capacity of  z  should be just enough so that varying  z  over its set will produce all the plausible   ̃ y  for a given  x . (c) Auto-encoder: can collapse if the system learns the identity function or if it can correctly reconstruct a region of  y  space that is much larger than the region of high data density, thereby giving low energy to an overly large region. (d) Simple joint embedding architecture: can collapse if the encoders ignore the inputs and produce representations that remain constant and equal, or if the encoders are invariant over overly broad regions of the space.
Figure 10: A few standard architectures and their capacity for collapse. Original figure and complete caption.
Figure 11: Contrastive and regularized methods for EBM training. A conceptual diagram of an energy landscape is shown on the left. Training samples are blue dots. The region of low energy is shown in orange (a level set of the energy function). Contrastive methods  (top right) push down on the energy of training samples (blue dots) and pulls up on the energies of suitably-placed contrastive samples (green dots). regularized methods  (bottom right) push down on the energy of training samples and use a regularizer term that minimizes the volume of low-energy regions. This regularization has the effect of “shrink-wrapping” the regions of high data density within the low-energy regions, to the extent that the flexibility of the energy function permits it. One issue with contrastive methods is that the energy will only be pulled up wherever contrastive samples have been placed. One must devise methods that preferentially place contrastive samples in regions of low energy, which is what Monte-Carlo and MCMC methods do. However, a disadvantage of contrastive methods is that the number of contrastive samples necessary to make an energy surface adopt a good shape may grow exponentially with the dimension of  y  space.
Figure 11: Contrastive and regularized methods for EBM training. Original figure and complete caption.
Equations 6, 7–Mathematical context · source page 24
(6, 7)Pairwise contrastive losses, source equations 6–7.
(8, 9)Group contrastive and likelihood losses, source equations 8–9.
(Mathematical context · source page 21)Original mathematical context on PDF page 21, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 22)Original mathematical context on PDF page 22, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 23)Original mathematical context on PDF page 23, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 24)Original mathematical context on PDF page 24, including inline formulas and their surrounding paragraphs.

4.4 Joint Embedding Predictive Architecture (JEPA)

JEPA encodes a context and a target, then predicts the target’s representation from the context representation and a latent variable. The mismatch defines energy. Predicting features lets the encoders discard details that do not help the prediction, while latents represent unresolved alternatives. This general architecture includes more possibilities than the later image- and video-masking implementations.

Figure 12: The Joint-Embedding Predictive Architecture (JEPA) consists of two encoding branches. The first branch computes  s x , a representation of  x  and the second branch  s y  a representation of  y . The encoders do not need to be identical. A predictor module predicts  s y  from  s x  with the possible help of a latent variable  z . The energy is the prediction error. Simple variations of the JEPA may use no predictor, forcing the two representations to be equal, or may use a fixed predictor with no latent, or may use simple latents such as discrete variables. The main advantage of JEPA is that it performs predictions in representation space, eschewing the need to predict every detail of  y , and enabling the elimination of irrelevant details by the encoders. More precisely, the main advantage of this architecture for representing multi-modal dependencies is twofold: (1) the encoder function  s y  = Enc( y )  may possess invariance properties that will make it produce the same  s y  for a set of different  y . This makes the energy constant over this set and allows the model to capture complex multi-modal dependencies; (2) The latent variable  z , when varied over a set  Z , can produce a set of plausible predictions  Pred( s x ,  Z ) =  {  ̃ s y  = Pred( s x , z )  ∀ z  ∈Z} If  x  is a video clip of a car approaching a fork in the road,  s x  and  s y  may represent the position, orientation, velocity and other characteristics of the car before and after the fork, respectively, ignoring irrelevant details such as the trees bordering the road or the texture of the sidewalk.  z  may represent whether the car takes the left branch or the right branch of the road.
Figure 12: The Joint-Embedding Predictive Architecture (JEPA) consists of two encoding branches. Original figure and complete caption.
Equations 10, 11, 12, 13–Mathematical context · source page 25
(10, 11, 12, 13)JEPA energy and latent inference, source equations 10–13.
(Mathematical context · source page 25)Original mathematical context on PDF page 25, including inline formulas and their surrounding paragraphs.

4.5 Training a JEPA

Four competing requirements shape the training objective: informative context features, informative target features, accurate feature prediction, and a limited latent channel. Too little information makes constant outputs attractive; too much unrestricted latent information lets the predictor ignore the context. The proposal gives several possible constraints rather than one final training recipe.

Figure 13: Non-contrastive training of JEPA. The main attraction of JEPAs is that they can be trained with non-contrastive methods. The basic principle of such training is that (1)  s x  should be maximally informative about  x ; (2)  s y  should be maximally informative about  y ; (3)  s y  should be easily predictable from  s x ; and (4)  z  should have minimal information content. Criteria 1, 2, and 4 collectively prevent a collapse of the energy function. Examples of such non-contrastive criteria for JEPA training include VICReg and Barlow Twins. As with every EBM, JEPAs can also be trained with contrastive methods. But doing so runs into the curse of dimensionality and limits the practical dimension of  s y .
Figure 13: Non-contrastive training of JEPA. Original figure and complete caption.
Equations Source display–Mathematical context · source page 28
(Source display)Finite possible target predictions for a discrete latent (unnumbered source display).
(Mathematical context · source page 26)Original mathematical context on PDF page 26, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 27)Original mathematical context on PDF page 27, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 28)Original mathematical context on PDF page 28, including inline formulas and their surrounding paragraphs.

4.5.1 VICReg

VICReg illustrates explicit anti-collapse constraints. Keep each feature’s batch variation above a threshold, reduce covariance between different features, and match the paired representations. An expansion network helps apply these constraints in a richer space. The terms encourage distinct and useful features; zero covariance alone does not establish complete statistical independence.

Figure 14: Training a JEPA with VICReg. VICReg is a non sample-contrastive method for training embeddings. The information content of the representations  s x  and  s y  is maximized by first mapping them to higher-dimensional embeddings  v x and  v y  through an expander (e.g. a trainable neural net with a few layers). The loss function drives the covariance matrix of the embeddings towards the identity (e.g. computed over a batch). VICReg can be seen as a  dimension-contrastive method  as opposed to sample-contrastive methods.
Figure 14: Training a JEPA with VICReg. Original figure and complete caption.
Equation Mathematical context · source page 29
(Mathematical context · source page 29)Original mathematical context on PDF page 29, including inline formulas and their surrounding paragraphs.

4.5.2 Biasing a JEPA towards learning “useful” representations

Predictability alone may not select the information a particular application needs. Auxiliary prediction heads can bias the representation toward task-relevant quantities available from the data. Architecture and training targets therefore remain important inductive choices.

4.6 Hierarchical JEPA (H-JEPA)

A short-horizon model may predict detailed movement, while a longer-horizon model predicts an abstract event such as arriving at a station. Stacked JEPAs are proposed to learn these levels, with temporal pooling supporting longer intervals. The hierarchy should support plans that become more detailed as execution approaches.

Figure 15: Hierarchical JEPA (H-JEPA) The ability of the JEPA to learn abstract representations in which accurate prediction can be performed allows hierarchical stacking. In this diagram JEPA-1 extracts low-level representations and performs short-term predictions. JEPA-2 takes the representations extracted by JEPA-1 as inputs and extracts higher-level representations with which longer-term predictions can be performed. More abstract representations ignore details of the inputs that are difficult to predict in the long term, enabling them to perform longer-term predictions with coarser descriptions of the world state.
Figure 15: Hierarchical JEPA (H-JEPA) The ability of the JEPA to learn abstract representations in which accurate prediction can be performed allows hierarchical stacking. Original figure and complete caption.
Equations Mathematical context · source page 30–Mathematical context · source page 31
(Mathematical context · source page 30)Original mathematical context on PDF page 30, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 31)Original mathematical context on PDF page 31, including inline formulas and their surrounding paragraphs.

4.7 Hierarchical Planning

A high-level actor chooses intermediate goals; lower levels find actions that reach them. For a commute, the high level might select a station while the low level controls the next turn. Lower levels must update their choices using new observations, since detailed conditions cannot all be known when the broad plan is formed.

Figure 16: Hierarchical JEPA for Mode-2 hierarchical planning. A complex task is defined by a high-level cost computed from a high-level world-state representation C ( s 2[4]) . A sequence of high-level abstract actions  ( a 2[2] , a 2[4])  is inferred that minimizes  C ( s 2[4]) . The inferred abstract actions are fed to lower-level cost modules  C ( s [2]) , C ( s [4])  which define subgoals for the lower layer. The lower layer then infers an action sequence that minimizes the subgoal costs. Although only a 2-layer hierarchy is shown here, it is straightforward to extend the concept to multiple levels. The process described here is sequential top-down, but a better approach would be to perform a joint optimization of the actions in all the layers.
Figure 16: Hierarchical JEPA for Mode-2 hierarchical planning. Original figure and complete caption.
Equations 14–Mathematical context · source page 32
(14)Hierarchical state encodings, source equation 14.
(Mathematical context · source page 32)Original mathematical context on PDF page 32, including inline formulas and their surrounding paragraphs.

4.8 Handling uncertainty

Uncertainty can come from randomness, chaos, missing observations, limited representations, or an inaccurate learned model. Latent variables generate different plausible futures. Planning should consider several futures and possibly penalize risk as well as average cost. The branching number can grow exponentially, so sampling, pruning and search remain major computational challenges.

Figure 17: Hierarchical JEPA for Mode-2 hierarchical planning in an uncertain environment. Realistic environments are not entirely predictable, even when using highly-abstract representations. Uncertainty about predictions can be handled by predictors with latent variables. The latent variables (red circles) contain information about the prediction that cannot be derived from the prior observation. The latent variables must be regularized to prevent an energy collapse and to force the system to predict as much as possible without the help of it. At planning time, latent variables are sampled from distributions obtained by applying a Gibbs distribution to the regularizers. Each sample leads to a different prediction. To produce consistent latent sequences, the parameters of the regularizer can be functions of previous states and retrieved memories. As the prediction progresses, the number of generated state trajectories may grow exponentially. If each latent variable has  k  possible discrete values, the number of possible trajectories will grow as k t , where  t  is the number of time steps. Directed search and pruning strategies must be employed. With multiple predicted trajectories, optimal action sequences can be computed that minimize the average cost, or a combination of average and variance of the cost so as to minimize risk.
Figure 17: Hierarchical JEPA for Mode-2 hierarchical planning in an uncertain environment. Original figure and complete caption.
Equation Mathematical context · source page 34
(Mathematical context · source page 34)Original mathematical context on PDF page 34, including inline formulas and their surrounding paragraphs.

4.8.1 World Model Architecture

Different prediction scales may need different machinery. Local motion can be represented by moving nearby features; interactions between objects may suit transformer-like representations. A separate model of the agent’s own body may be simpler than a model of the external world. These are architectural suggestions left for implementation and testing.

4.9 Keeping track of the state of the world

A memory query is compared with stored keys, the scores are normalized, and the corresponding values are combined. Writing updates or allocates stored values. The equations describe a differentiable associative-memory pattern. How to maintain accurate beliefs and allocate memory during complex planning remains open.

Equations 15, 16, 17, 18, 19, 20–Mathematical context · source page 36
(15, 16, 17, 18, 19, 20)Associative-memory retrieval and updates, source equations 15–20.
(Mathematical context · source page 35)Original mathematical context on PDF page 35, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 36)Original mathematical context on PDF page 36, including inline formulas and their surrounding paragraphs.

4.10 Data Streams

A learner may acquire data by passively watching, moving sensors, exploring its environment, responding to its own actions, or actively probing an uncertain situation. These modes provide different information. Curiosity and exploration objectives are suggested because observation alone may not supply every state transition needed for control.

5 Designing and Training the Actor

The actor searches over actions and unresolved latent variables, then teaches fast policies from the resulting plans. Gradients are useful when the model and cost behave smoothly; dynamic programming, beam search or tree search may be needed otherwise. Exploring uncertainty and selecting actions are connected optimization tasks, with potentially very different search costs.

Equation Mathematical context · source page 37
(Mathematical context · source page 37)Original mathematical context on PDF page 37, including inline formulas and their surrounding paragraphs.

6 Designing the Configurator

The configurator selects what the shared modules should attend to and accomplish. It may adjust routing, supply transformer tokens, weight cost components, and choose subgoals. Reusing one configurable model could share knowledge across tasks. How to train it to decompose a difficult task into suitable subgoals is explicitly left unanswered.

7 Related Work

The proposal draws on earlier model-based control, predictive representation learning and cognitive science. Its contribution is a suggested combination and research direction, rather than a claim that world models or planning began with JEPA.

7.1 Trained World Models, Model-Predictive Control, Hierarchical Planning

Learned models for control, model-based reinforcement learning, video prediction and hierarchical planning all provide predecessors. Generative models address uncertainty in observation space; JEPA instead proposes learning the space in which prediction is performed. The source reviews both historical methods and work current in 2022.

7.2 Energy-Based Models and Joint-Embedding Architectures

The paper uses “energy-based” broadly to mean learning compatibility scores, with normalized probabilistic models as one possible special case. Joint embeddings, Siamese networks, contrastive learning and more recent noncontrastive regularizers supply the representation-learning background.

7.3 Human and animal cognition

Developmental and cognitive research motivates learning abstract world knowledge and planning with simplified representations. Comparisons with attention, self-monitoring and consciousness are speculative analogies. They are not experimental evidence that the proposed modules reproduce the brain.

8 Discussion, Limitations, Broader Relevance

The author stresses that turning the diagram into a working system is a difficult, long-term research problem. The remaining sections separate missing mechanisms from broader speculation and arguments about alternative research directions.

8.1 What is missing from the Proposed Model?

Major missing pieces include how to train a useful hierarchy from video, constrain latent information, search actions and alternatives efficiently, maintain memory, and learn the configurator. Differentiability permits gradient computation but does not guarantee effective planning. The paper explicitly anticipates unforeseen obstacles beyond this list.

8.2 Broader Relevance of the Proposed Approach

The architecture is proposed as both an engineering direction and a possible framework for thinking about cognition. These broader interpretations should be distinguished from measured capabilities.

8.2.1 Could this Architecture be the Basis of a Model of Animal Intelligence?

The author sketches functional parallels between modules and brain systems, then speculates about emotion and conscious control. These suggestions motivate interdisciplinary discussion; the paper supplies neither a validated neural mapping nor a test of machine consciousness.

8.2.2 Could this be a Path towards Machine Common Sense?

Common sense is characterized as using learned world relationships to infer information missing from current perception or memory. The conjecture is that a sufficiently capable, configurable predictive hierarchy could support this behavior. It remains a conjecture about the full system.

8.3 Is it all about scaling? Is reward really enough?

Three debates frame the argument: whether scaling generative models is sufficient, whether reward alone supplies enough learning signal, and whether reasoning requires explicitly programmed symbol manipulation. The following claims state the author’s 2022 position.

8.3.1 Scaling is not enough

LeCun argues that scaling token prediction does not by itself solve uncertain continuous-world prediction or goal-directed search. The concerns motivate representation-space prediction and latent-variable planning. They are research judgments in a position paper, not a theorem ruling out every scaled, multimodal or search-augmented model.

8.3.2 Reward is not enough

A scalar reward provides less direct information than predicting many aspects of an observation. The proposal therefore assigns most learning to self-supervised world modeling and uses intrinsic cost to guide action. Reinforcement learning can still contribute; the argument challenges its sufficiency as the sole learning signal.

8.3.3 Do We Need Symbols for Reasoning?

Continuous representations can support gradient-based reasoning, while discrete high-level choices may require combinatorial search. Learning a useful continuous relaxation could make the latter easier. Whether this formulation captures every kind of human reasoning is expressly unresolved.

Acknowledgments

The author acknowledges colleagues and discussions that shaped the proposal. The work is presented as a starting point for further research and critique.

Appendix: Symbols and Notations

Filled circles denote observed or computed variables; hollow circles denote variables that must be inferred or sampled. Red rectangles contribute energy; rounded rectangles compute deterministic functions. A latent regularizer limits which hidden configurations are plausible. It can define a set of outputs or, after Gibbs normalization, induce a probability distribution. The vector excerpts retain the original symbols and formulas.

Figure 18: Symbols used in architectural diagrams. Architectural diagrams use symbols commonly used to draw factor graphs – circles for variables, rectangles for factors – plus rounded rectangles to represent deterministic functions. Filled circles represent observed variables, or variables that are outputs of deterministic functions. Hollow circles represent latent variables, i.e. variables that must be inferred by minimizing some cost, or sampled from a distribution. Red rectangles represent energy terms. These modules have an implicit scalar output that contributes additively to the total energy of the system. This is similar to the convention used for factor graphs. Rounded rectangles represent deterministic functions, which may have one or several inputs. Given a set of inputs, the output is assumed to be easily computable and unique. The function is generally assumed to be differentiable. It may contain trainable parameters. Non-deterministic functions are represented as shown on the right. They are composed of a deterministic function  g ( x, z )  in which one of the inputs is a latent variabl  z . The latent variable is seen as varying within a level set of a regularizing energy term  R ( z ) . When  z  varies in the level set Z h  =  { z | R ( z )  < h } , the output   ̃ y  will vary over the set  Y h  =  { y | y  =  g ( x, z ) ,  ∀ z  ∈Z h } . In some cases, the energy term can be transformed into a probability distribution (see text).
Figure 18: Symbols used in architectural diagrams. Original figure and complete caption.
Equations 21, 22, 23, 24–Mathematical context · source page 58
(21, 22, 23, 24)Latent level sets and induced distributions, source equations 21–24.
(Mathematical context · source page 58)Original mathematical context on PDF page 58, including inline formulas and their surrounding paragraphs.

Appendix: Amortized Inference for Latent Variables

Repeated latent optimization can be slow. Train an encoder to approximate its solution, then use that encoder either directly or to initialize refinement. This amortizes inference across future examples. Because the encoder can see the target, constraining latent information is especially important: otherwise it can copy the answer and destroy the learning task.

Figure 19: Amortized Inference with an EBM. An encoder is trained to produce   ̃ z  = Enc( s x , y ) to approximate the value of the latent that minimizes the energy  ˇ z  = argmin z ∈Z  E w ( x, y, z ) . The regularizer  R ( z )  plays the essential role of limiting the information that  z  contains about  y . This is particularly important here because the system has access to  y  and can “cheat” by carrying the complete information about  y  through the encoder.
Figure 19: Amortized Inference with an EBM. Original figure and complete caption.
Figure 20: Amortized Inference with a Regularized Generative Latent-Variable EBM architecture. An encoder is trained to produce   ̃ z  = Enc( s x , y )  to approximate the value  ˇ z  that minimizes the energy. The regularizer  R ( z )  plays the essential role of limiting the information that  z  contains about  y . This is particularly important here because the system has access to  y  and can “cheat” by carrying the complete information about  y  through the encoder.
Figure 20: Amortized Inference with a Regularized Generative Latent-Variable EBM architecture. Original figure and complete caption.
Equations Mathematical context · source page 59–Mathematical context · source page 60
(Mathematical context · source page 59)Original mathematical context on PDF page 59, including inline formulas and their surrounding paragraphs.
(Mathematical context · source page 60)Original mathematical context on PDF page 60, including inline formulas and their surrounding paragraphs.

Appendix: Loss functions for Contrastive Training of EBM

The table compares ways to shape energy using observed examples and contrastive alternatives. Likelihood methods sum or sample competing outputs; hinge and logistic losses compare scores; adversarial methods learn a negative-example generator; denoising corrupts data to create training pairs. The crucial design choices are how alternatives are generated and how their energies are penalized. The original table, every loss and the detailed source discussion are retained below.

Table 1: Contrastive training methods, negative-sample strategies, losses and complete source caption.
Table 1: Contrastive training methods and loss functions. Original table and complete caption.
Equations Source derivation–Mathematical context · source page 62
(Source derivation)Original row-by-row mathematical discussion of Table 1, PDF page 61.
(Mathematical context · source page 62)Original mathematical context on PDF page 62, including inline formulas and their surrounding paragraphs.
Concept demonstration

Open image ↗