At a glance
Imagine learning how a kitchen works by watching, then planning how to prepare a meal before moving a robot. LeCun proposes an architecture for that kind of agent. A world model predicts useful aspects of possible futures; a cost model scores them; an actor searches for actions; memory and a configurator supply context and goals. JEPA learns by predicting representations instead of reconstructing every detail of an observation. A hierarchy would support longer, more abstract plans. This is a research proposal that identifies important missing pieces, especially training the hierarchy and configurator. It does not report a completed autonomous system.
1 Prologue
The paper assembles existing ideas into a proposed research program. It explicitly does not claim that every component is new. The central question is how self-supervised world models, planning and intrinsic objectives could fit together in one learnable architecture.
2 Introduction
The three goals are to learn mainly through observation, reason through methods compatible with gradient-based learning, and represent plans at several levels of detail and time. The proposal connects these goals through a learned world model and hierarchical prediction.
2.1 Learning World Models
A world model predicts what may happen and fills in information that perception does not reveal. Such knowledge could let an agent learn a new task with fewer costly real-world trials: imagine a possible action, predict its consequence, and avoid an obviously poor choice before executing it.
2.2 Humans and Animals learn Hierarchies of Models
The developmental examples motivate a hierarchy: simpler perceptual regularities may support objects, physical relationships and more abstract concepts. A configurable shared world-model engine is proposed to reuse this knowledge across tasks. The infant timeline and brain interpretation are motivations, not an implementation or proof that a machine must learn in exactly that order.
3 A Model Architecture for Autonomous Intelligence
The architecture has six connected roles. Perception estimates state; the world model predicts possible futures; a cost module evaluates them; the actor proposes and improves actions; short-term memory stores relevant experience; a configurator chooses tasks and subgoals. The modules are intended to exchange gradients, enabling planning through the learned model.
3.1 Typical Perception-Action Loops
The agent can act through a fast learned policy or spend additional computation planning. These two modes correspond to different uses of the same perception, memory and world-model components.
3.1.1 Mode-1: Reactive behavior
Mode 1 maps an estimated state directly to an action using a policy. It is fast because it does not search through imagined futures. The observed result can still train the world model. Learning only by trying perturbed actions in the physical world can be slow and costly.
Equations Mathematical context · source page 9–Mathematical context · source page 10
3.1.2 Mode-2: reasoning and planning using the world model
Mode 2 proposes an action sequence, rolls it forward through the world model, scores the predicted states, and improves the actions. After planning, the agent executes only the first action or short segment, observes the result, and replans. This is the familiar model-predictive-control loop with a learned predictive model.
Equation Mathematical context · source page 11
3.1.3 From Mode-2 to Mode-1: Learning New Skills
A slow planner can teach a fast policy: use the planner’s selected actions as supervised targets. The policy can then act directly or initialize later searches. This is amortized inference—pay the search cost during learning so that future decisions need less computation.
Equations Source display–Mathematical context · source page 12
3.1.4 Reasoning as Energy Minimization
Here, reasoning means searching for variables that satisfy constraints or reduce an energy score. Gradient descent is an option when the model is smooth; discrete choices may require other search methods. Calling reasoning energy minimization is a formulation, not a guarantee that the resulting optimization will be easy.
3.2 The Cost Module as the Driver of Behavior
The cost combines an intrinsic part with a learned critic. Intrinsic terms encode basic objectives such as avoiding damage or preserving energy; the critic predicts longer-term consequences and task-specific subgoals. A configurator can adjust weights among cost components. Keeping the intrinsic objective fixed is proposed to prevent unwanted objective drift, but it is not by itself a demonstrated safety guarantee.
Equations 1, 2–Mathematical context · source page 14
3.3 Training the Critic
Store states, times and intrinsic costs in memory. To train the critic, retrieve an earlier state and a later cost, then adjust the critic to predict that future consequence. More elaborate targets could include discounted future costs. A differentiable key–value memory allows approximate retrieval and interpolation rather than requiring an exact lookup.
Equation Mathematical context · source page 15
4 Designing and Training the World Model
The world model must learn from sufficiently diverse experience, represent several plausible outcomes, and predict at multiple timescales. These requirements interact: the agent’s own behavior affects the data it later observes, and long-horizon prediction often needs a different abstraction from short-horizon motion prediction.
4.1 Self-Supervised Learning
Self-supervision asks whether related observations fit together. An energy score can rate a proposed video continuation without explicitly generating every possible continuation. The proposed representation should retain useful information while making prediction easier. Learning objects, geometry and physical relationships through this process is presented as a hypothesis to test.
Equation Mathematical context · source page 17
4.2 Handling Uncertainty with Latent Variables
A latent variable represents information about the target that the context does not determine, such as camera motion between two views. Inference finds a latent setting that makes the pair compatible; minimizing over it produces a score for the pair. Latents can express alternative outcomes, but their capacity must be limited so they cannot simply encode any answer.
Equations 4–Mathematical context · source page 20
4.3 Training Energy-Based Models
Training must do more than make observed pairs have low energy. A constant representation would make every pair look compatible and solve that narrow objective uselessly. Contrastive methods push up scores for incompatible alternatives. Architectural constraints and regularization instead limit the extent of low-energy regions. Sampling useful negatives can be expensive; avoiding them requires a credible alternative constraint.
Equations 6, 7–Mathematical context · source page 24
4.4 Joint Embedding Predictive Architecture (JEPA)
JEPA encodes a context and a target, then predicts the target’s representation from the context representation and a latent variable. The mismatch defines energy. Predicting features lets the encoders discard details that do not help the prediction, while latents represent unresolved alternatives. This general architecture includes more possibilities than the later image- and video-masking implementations.
Equations 10, 11, 12, 13–Mathematical context · source page 25
4.5 Training a JEPA
Four competing requirements shape the training objective: informative context features, informative target features, accurate feature prediction, and a limited latent channel. Too little information makes constant outputs attractive; too much unrestricted latent information lets the predictor ignore the context. The proposal gives several possible constraints rather than one final training recipe.
Equations Source display–Mathematical context · source page 28
4.5.1 VICReg
VICReg illustrates explicit anti-collapse constraints. Keep each feature’s batch variation above a threshold, reduce covariance between different features, and match the paired representations. An expansion network helps apply these constraints in a richer space. The terms encourage distinct and useful features; zero covariance alone does not establish complete statistical independence.
Equation Mathematical context · source page 29
4.5.2 Biasing a JEPA towards learning “useful” representations
Predictability alone may not select the information a particular application needs. Auxiliary prediction heads can bias the representation toward task-relevant quantities available from the data. Architecture and training targets therefore remain important inductive choices.
4.6 Hierarchical JEPA (H-JEPA)
A short-horizon model may predict detailed movement, while a longer-horizon model predicts an abstract event such as arriving at a station. Stacked JEPAs are proposed to learn these levels, with temporal pooling supporting longer intervals. The hierarchy should support plans that become more detailed as execution approaches.
Equations Mathematical context · source page 30–Mathematical context · source page 31
4.7 Hierarchical Planning
A high-level actor chooses intermediate goals; lower levels find actions that reach them. For a commute, the high level might select a station while the low level controls the next turn. Lower levels must update their choices using new observations, since detailed conditions cannot all be known when the broad plan is formed.
Equations 14–Mathematical context · source page 32
4.8 Handling uncertainty
Uncertainty can come from randomness, chaos, missing observations, limited representations, or an inaccurate learned model. Latent variables generate different plausible futures. Planning should consider several futures and possibly penalize risk as well as average cost. The branching number can grow exponentially, so sampling, pruning and search remain major computational challenges.
Equation Mathematical context · source page 34
4.8.1 World Model Architecture
Different prediction scales may need different machinery. Local motion can be represented by moving nearby features; interactions between objects may suit transformer-like representations. A separate model of the agent’s own body may be simpler than a model of the external world. These are architectural suggestions left for implementation and testing.
4.9 Keeping track of the state of the world
A memory query is compared with stored keys, the scores are normalized, and the corresponding values are combined. Writing updates or allocates stored values. The equations describe a differentiable associative-memory pattern. How to maintain accurate beliefs and allocate memory during complex planning remains open.
Equations 15, 16, 17, 18, 19, 20–Mathematical context · source page 36
4.10 Data Streams
A learner may acquire data by passively watching, moving sensors, exploring its environment, responding to its own actions, or actively probing an uncertain situation. These modes provide different information. Curiosity and exploration objectives are suggested because observation alone may not supply every state transition needed for control.
5 Designing and Training the Actor
The actor searches over actions and unresolved latent variables, then teaches fast policies from the resulting plans. Gradients are useful when the model and cost behave smoothly; dynamic programming, beam search or tree search may be needed otherwise. Exploring uncertainty and selecting actions are connected optimization tasks, with potentially very different search costs.
Equation Mathematical context · source page 37
6 Designing the Configurator
The configurator selects what the shared modules should attend to and accomplish. It may adjust routing, supply transformer tokens, weight cost components, and choose subgoals. Reusing one configurable model could share knowledge across tasks. How to train it to decompose a difficult task into suitable subgoals is explicitly left unanswered.
7 Related Work
The proposal draws on earlier model-based control, predictive representation learning and cognitive science. Its contribution is a suggested combination and research direction, rather than a claim that world models or planning began with JEPA.
7.1 Trained World Models, Model-Predictive Control, Hierarchical Planning
Learned models for control, model-based reinforcement learning, video prediction and hierarchical planning all provide predecessors. Generative models address uncertainty in observation space; JEPA instead proposes learning the space in which prediction is performed. The source reviews both historical methods and work current in 2022.
7.2 Energy-Based Models and Joint-Embedding Architectures
The paper uses “energy-based” broadly to mean learning compatibility scores, with normalized probabilistic models as one possible special case. Joint embeddings, Siamese networks, contrastive learning and more recent noncontrastive regularizers supply the representation-learning background.
7.3 Human and animal cognition
Developmental and cognitive research motivates learning abstract world knowledge and planning with simplified representations. Comparisons with attention, self-monitoring and consciousness are speculative analogies. They are not experimental evidence that the proposed modules reproduce the brain.
8 Discussion, Limitations, Broader Relevance
The author stresses that turning the diagram into a working system is a difficult, long-term research problem. The remaining sections separate missing mechanisms from broader speculation and arguments about alternative research directions.
8.1 What is missing from the Proposed Model?
Major missing pieces include how to train a useful hierarchy from video, constrain latent information, search actions and alternatives efficiently, maintain memory, and learn the configurator. Differentiability permits gradient computation but does not guarantee effective planning. The paper explicitly anticipates unforeseen obstacles beyond this list.
8.2 Broader Relevance of the Proposed Approach
The architecture is proposed as both an engineering direction and a possible framework for thinking about cognition. These broader interpretations should be distinguished from measured capabilities.
8.2.1 Could this Architecture be the Basis of a Model of Animal Intelligence?
The author sketches functional parallels between modules and brain systems, then speculates about emotion and conscious control. These suggestions motivate interdisciplinary discussion; the paper supplies neither a validated neural mapping nor a test of machine consciousness.
8.2.2 Could this be a Path towards Machine Common Sense?
Common sense is characterized as using learned world relationships to infer information missing from current perception or memory. The conjecture is that a sufficiently capable, configurable predictive hierarchy could support this behavior. It remains a conjecture about the full system.
8.3 Is it all about scaling? Is reward really enough?
Three debates frame the argument: whether scaling generative models is sufficient, whether reward alone supplies enough learning signal, and whether reasoning requires explicitly programmed symbol manipulation. The following claims state the author’s 2022 position.
8.3.1 Scaling is not enough
LeCun argues that scaling token prediction does not by itself solve uncertain continuous-world prediction or goal-directed search. The concerns motivate representation-space prediction and latent-variable planning. They are research judgments in a position paper, not a theorem ruling out every scaled, multimodal or search-augmented model.
8.3.2 Reward is not enough
A scalar reward provides less direct information than predicting many aspects of an observation. The proposal therefore assigns most learning to self-supervised world modeling and uses intrinsic cost to guide action. Reinforcement learning can still contribute; the argument challenges its sufficiency as the sole learning signal.
8.3.3 Do We Need Symbols for Reasoning?
Continuous representations can support gradient-based reasoning, while discrete high-level choices may require combinatorial search. Learning a useful continuous relaxation could make the latter easier. Whether this formulation captures every kind of human reasoning is expressly unresolved.
Acknowledgments
The author acknowledges colleagues and discussions that shaped the proposal. The work is presented as a starting point for further research and critique.
Appendix: Symbols and Notations
Filled circles denote observed or computed variables; hollow circles denote variables that must be inferred or sampled. Red rectangles contribute energy; rounded rectangles compute deterministic functions. A latent regularizer limits which hidden configurations are plausible. It can define a set of outputs or, after Gibbs normalization, induce a probability distribution. The vector excerpts retain the original symbols and formulas.
Equations 21, 22, 23, 24–Mathematical context · source page 58
Appendix: Amortized Inference for Latent Variables
Repeated latent optimization can be slow. Train an encoder to approximate its solution, then use that encoder either directly or to initialize refinement. This amortizes inference across future examples. Because the encoder can see the target, constraining latent information is especially important: otherwise it can copy the answer and destroy the learning task.
Equations Mathematical context · source page 59–Mathematical context · source page 60
Appendix: Loss functions for Contrastive Training of EBM
The table compares ways to shape energy using observed examples and contrastive alternatives. Likelihood methods sum or sample competing outputs; hinge and logistic losses compare scores; adversarial methods learn a negative-example generator; denoising corrupts data to create training pairs. The crucial design choices are how alternatives are generated and how their energies are penalized. The original table, every loss and the detailed source discussion are retained below.