Overview
Intuition. Imagine representing five switches using only two real-valued readouts. Five independent linear channels will not fit, but if most switches are off, shared directions can still give useful readings. Collisions occur when features overlap; the model trades that interference against the benefit of keeping more features.
Reading the math. The matrix W mixes input features x into a smaller hidden vector h = Wx. A decoder tries to reconstruct x, and ReLU clips negative outputs to zero. Sparsity means features are rarely active, not that the weight matrix contains mostly zeros. The resulting geometry is a learned compromise under the toy loss, rather than five perfectly independent channels.
Small networks trained on synthetic sparse features can represent more features than hidden dimensions by allowing controlled interference between their directions. A nonlinear output filters some of that interference. The experiments reveal phase transitions, geometric arrangements, and limited computation in superposition. They establish these behaviors in toy settings; how much the detailed geometry transfers to real language models remains uncertain. [1]

Key Results From Our Toy Models
The experiments demonstrate both monosemantic and polysemantic neurons, sharp changes in representation strategy, structured feature geometry, and an absolute-value computation using superposed features. These are existence results for simple ReLU networks, not claims that every real network follows the same organization.
Definitions and Motivation: Features, Directions, and Superposition
The conceptual picture separates four questions: whether activations decompose into features, whether those features are represented linearly, whether there are more features than dimensions, and whether features align with neuron coordinates. A privileged basis favors coordinate alignment; superposition can work against it.
Empirical Phenomena
Motivating observations include semantic directions in embeddings, interpretable neurons, recurring features across models, and neurons responding to unrelated inputs. No one observation alone determines a universal definition of a feature.
What are Features?
Candidate definitions include arbitrary input functions, human-understandable properties, or properties that sufficiently large models reliably represent with dedicated neurons. Each has limitations. The paper provisionally favors the last intuition while keeping the definition open.
Features as Directions
A linear representation stores feature values as a weighted sum of feature directions. The features themselves may be nonlinear functions of the original input. Linear accessibility makes them easy for later layers to read, although nonlinear encodings are possible and discussed in the appendix.
Privileged vs Non-privileged Bases
Purely linear surroundings permit a change of coordinates without changing behavior. Coordinatewise nonlinearities break that symmetry and privilege neuron axes, but do not guarantee that individual features align with them.

The Superposition Hypothesis
Superposition trades the benefit of representing extra features against interference between nonorthogonal directions. Sparse feature occurrence and nonlinear filtering can make the trade favorable. Almost-orthogonal vectors and compressed sensing motivate this possibility; the experiments test it directly.



Summary: A Hierarchy of Feature Properties
Decomposability and linearity describe how features combine. Superposition means their linear encoding is not invertible over arbitrary feature combinations; basis alignment is a further restriction. The paper hypothesizes the first two properties broadly while expecting the latter properties to vary by model and regime.
Demonstrating Superposition
The first experiment asks whether adding a simple output nonlinearity allows a narrow hidden space to recover more input features than a linear baseline can represent.
Experiment Setup
An autoencoder maps a known high-dimensional feature vector into a smaller hidden vector and reconstructs it. Knowing the synthetic ground-truth features makes representation and interference directly measurable.

The Feature Vector ( x )
Each input coordinate is zero with a chosen probability and otherwise uniform on [0,1]. Features may differ in importance in the reconstruction loss; most initial experiments give them the same sparsity. Sparse, numerous, unequally useful features are modeling assumptions rather than measurements from a real network.
The Model ( x → x′ )
Both initial models use a linear encoder and its transposed matrix as decoder, plus an output bias. The baseline output is linear; the comparison adds ReLU. Tied weights make feature directions identifiable, and negative biases let the nonlinear model suppress small interference.
The Loss
Training minimizes squared reconstruction error weighted by each feature’s importance. This explicitly controls which errors the model has an incentive to avoid.
Equation
Basic Results
The linear baseline keeps the most important dimensions. With dense inputs, the ReLU model behaves similarly; as inputs become sparse, it represents extra features nonorthogonally. Antipodal pairs appear first, followed by richer arrangements. The same qualitative pattern occurs in larger toy configurations.




Mathematical Understanding
A loss decomposition exposes competing reconstruction benefits and interference costs. In the sparse limit, zero- and one-active-feature cases dominate. ReLU removes negative interference in the one-feature case, and negative biases can suppress small positive interference. With additional simplifying constraints, the geometry resembles a generalized Thomson point-packing problem.


Superposition as a Phase Change
Two features in one dimension and three features in two dimensions exhibit distinct strategies: ignore a feature, allocate it a dedicated direction, or share space. Restricted analytic candidate solutions and trained models show sharp boundaries as sparsity and relative importance change. These candidate phase diagrams are not an exhaustive solution of every possible weight configuration.


The Geometry of Superposition
Equal feature statistics produce striking regular arrangements, while unequal statistics deform or change those arrangements. The authors explicitly caution that this elegant geometry may depend strongly on the toy setup.
Uniform Superposition
For equal-importance features, the squared encoder norm approximates how many features are represented. The resulting average dimensions-per-feature curve has persistent regions near one and one-half; the latter corresponds to antipodal pairs. Other feature-level fractions reveal additional structures.

Feature Dimensionality
Feature dimensionality compares a feature’s own squared norm with the total squared projection of all features onto its direction. It equals one-half for an equal antipodal pair. Empirical values cluster at fractions associated with regular configurations; their sum approximates hidden dimension in efficiently packed examples.
Equation

Why these geometric structures?
Fractions such as three-quarters, two-thirds, and two-fifths correspond to tetrahedra, triangles, and pentagons. Orthogonal combinations of smaller polytopes can explain split groups of dimensionalities. Features in different orthogonal factors do not interfere, potentially reducing damaging simultaneous interactions.

Aside: Polytopes and Low-Rank Matrices
A feature arrangement determines its Gram matrix of pairwise dot products, and that matrix determines interference. Conversely, discarded directions constrain possible arrangements. This geometry–matrix connection is useful even when the exact optimal packing is unknown.

Non-Uniform Superposition
Changing importance, sparsity, or co-occurrence deforms the uniform solutions. Correlated features tend to avoid interference if space permits; anticorrelated features can share space more readily. The report offers representative experiments rather than a complete non-uniform theory.
Perturbing a Single Feature
Changing one feature’s activation frequency deforms a five-feature pentagon until a competing two-pair solution becomes preferable and the weakest feature is dropped. Clean two-dimensional results require multiple optimization runs and selection of low-loss solutions; suboptimal runs help reveal competing minima.

Organization of Correlated and Anticorrelated Features
Two-dimensional experiments visualize how these feature groups arrange around the hidden space. Rotations and reflections align solutions for comparison; the figures preserve the resulting geometry.

Local Almost-Orthogonal Bases
Larger experiments can put each correlated group into a nearly orthogonal local basis even while different groups share space globally. This suggests a possible local simplification for interpretation, but does not establish such a property across the full distribution of real models.

Collapsing of Correlated Features
When correlated features are too dense or space is too limited, the model can preserve their shared principal component instead of their separate values. Increasing sparsity favors separating them in superposition. Principal-component compression and superposition therefore coexist and trade off in this toy setting.

Superposition and Learning Dynamics
The learning-dynamics experiments track how the final geometric structures form. They are preliminary observations of selected trajectories, not a general optimization theory.
Phenomenon 1: Discrete "Energy Level" Jumps
Individual features sometimes jump between characteristic dimensionalities while the loss drops. This motivates the idea that apparently smooth large-model learning might contain many smaller structural transitions, but that extrapolation is untested.

Phenomenon 2: Learning as Geometric Transformations
A six-feature, three-dimensional example passes through interpretable geometric rearrangements before reaching an octahedral arrangement. Different initializations can take different routes. Connections to initially linear learning and hierarchical feature splitting are suggestive, with different assumptions from the cited theorems.

Relationship to Adversarial Robustness
Analytically constructed attacks with L2 size 10% of average input norm become over three times more damaging as superposition appears in the tested setup. Interference provides an attack direction, but this does not establish the cause of real-model adversarial examples. Adversarial training reduces superposition; eliminating it in the preliminary experiment requires attacks around 80% of input norm.
Equations

Superposition in a Privileged Basis
Adding ReLU inside the hidden layer privileges neuron coordinates. Some feature directions then align with individual neurons, while sparse regimes still allow mixed representations. This connects the synthetic feature geometry to monosemantic and polysemantic neuron behavior.
Equations

Visualizing Superposition in Terms of Neurons
Weight plots reveal both dedicated and mixed neurons, sometimes in the same model. For the small five-neuron example, the displayed solutions are the best of one thousand training runs per setting. The structured mixtures are therefore low-loss examples rather than typical-run statistics.


Limitations of The ReLU Hidden Layer Toy Model Simulating Identity
The identity-reconstruction task does not need a hidden nonlinearity, so the network tries to bypass it using positive biases or averages of many inputs. This limits the relevance of that particular hidden-ReLU toy model and motivates a task requiring nonlinear computation.
Computation in Superposition
The next task computes coordinatewise absolute values. Two ReLU units can represent the positive and negative parts of one feature, giving a simple reference circuit against which computation in superposition can be examined.

Experiment Setup
Nonzero input values now range from −1 to 1, and the target is their absolute value. Encoder and decoder weights are no longer tied. A hidden ReLU and output ReLU train under the same importance-weighted squared-error objective.
Equation
Basic Results
With enough hidden units, the model learns the expected positive-part and negative-part circuit for each feature. Visualizations normalize equivalent hidden-unit rescalings before comparing weights.


Superposition vs Sparsity
As the task becomes sparser, a forty-neuron model approximates absolute values for more input features through mixed neurons. This demonstrates limited computation in superposition. Losses across sparsity settings are not directly comparable because the input distributions change. Some neurons have a dominant feature plus smaller secondary features, so their largest activations can conceal polysemanticity.


The Asymmetric Superposition Motif
One inspected motif uses unequal encoding magnitudes and reciprocal decoder weights to make interference asymmetric. A second neuron inhibits the harmful positive cross-effect, converting it into interference the output ReLU can suppress. This explains much of the selected circuit, with some smaller weights still not fully characterized.



The Strategic Picture of Superposition
The strategic motivation is to identify useful units for understanding model computation. If important features are hidden in superposition, inspecting neurons alone can leave substantial behavior unexplained.
Safety, Interpretability, & "Solving Superposition"
The authors describe feature enumeration and sparse decoding as a possible route toward more complete circuit analysis. They present this as a research objective: identifying features alone neither proves the absence of harmful behavior nor establishes that all relevant mechanisms have been captured.
Three Ways Out
Three broad approaches are to train models without superposition, recover an overcomplete dictionary afterward, or combine architectural changes with later decoding. Competitive performance and completeness remain challenges for every approach.
Approach 1: Creating Models Without Superposition
Hidden-activation L1 regularization can remove superposition in toy models at a reconstruction cost. Architectures such as mixture-of-experts networks might offer more selectively active neurons for a given compute budget. The suggestion that this could reduce the need for superposition is a hypothesis, not an evaluated capability result here.
Approach 2: Finding an Overcomplete Basis
Post-hoc sparse dictionary learning seeks an overcomplete set of directions explaining observed activations. It preserves the original model’s performance but introduces unknown dictionary size, large computational costs, and a gap between dictionary features and the network’s literal computation.
Approach 3: Hybrid Approaches
Hybrid approaches could reduce superposition slightly or make its decoding easier without eliminating it during training. Whether this can preserve competitive performance at useful scales remains open.
Additional Considerations
Sharp toy-model transitions suggest that complete removal is possible in some regimes. Even a less capable model with clear ground-truth features could aid research. Local bases, however, do not provide the full-distribution understanding required for strong global claims.
Discussion
The discussion separates possible interpretability applications from the remaining questions about real neural networks.
To What Extent Does Superposition Exist in Real Models?
The toy models reproduce several qualitative observations: mixed and dedicated neurons can coexist, sparse concepts encourage mixed representations, and top activations can appear cleaner than weaker ones. This supports their usefulness as models of possible mechanisms, while leaving real-model geometry and dynamics especially uncertain.
Open Questions
Open questions include statistical tests for superposition, controllable phase boundaries, realistic feature statistics, scaling behavior, feature counts, non-robust features, and the class of computations possible in compressed representations. The paper does not settle these broader questions.
Additional Connections
Reader-suggested connections include vector-symbolic architectures, hyperdimensional computing, and mathematical frames. These relate nonorthogonal high-dimensional directions to representation and computation, although the paper does not review them exhaustively.
Comments & Replications
The following external replications and later comments are preserved with attribution and should be distinguished from the original experiments.
Replication & Forthcoming Paper
Kshitij Sachan reports independent reproduction of the basic superposition and phase-diagram experiments and finds that activation choice changes the diagrams. A later update links the resulting Polysemanticity and Capacity in Neural Networks work, which develops a related constrained-optimization account.

Replication & Further Results
Tom McGrath reproduces key results and exactly analyzes the bias-free two-feature, one-dimensional case. The full loss surface reveals a same-direction confused-feature regime absent from the original restricted candidates, continuous movements of some minima, and other discontinuous transitions. This both supports and refines the original phase-diagram picture.
Equation

Replication
Jeffrey Wu and Dan Mossing report independent reproduction of the basic results, fractional-dimensionality patterns, and discrete changes during learning.
Replication
Spencer Becker-Kahn reports independent reproduction of key basic plots and small-model results consistent with the geometry and learning-dynamics account.
Engineering Monosemanticity in Toy Models
Adam Jermyn, Evan Hubinger, and Nicholas Schiefer report follow-up toy architectures and initialization methods that can produce monosemantic representations with sufficient capacity. Their results motivate architectural research but do not establish a cost-free solution for real language models.
Fractional Dimensionality and "Pressure"
Tom Henighan and Chris Olah observe cleaner geometric structure when more features compete for representation and when models train longer. This preliminary pressure effect is especially visible at high sparsity and lacks a complete explanation.

Replication
Marius Hobbhahn reports reproduction of the basic reconstruction and privileged-basis results, and most findings from the later memorization and double-descent study.
Extracting Features with Sparse Autoencoders
Lee Sharkey, Dan Braun, and Beren Millidge recover nearly ground-truth directions from a synthetic sparse-mixture dataset using an L1-regularized autoencoder. Dead units, losses, and comparisons across dictionary sizes offer tentative ways to estimate feature count. Their initial language-model activation results are inconclusive.
Linear Representation in Othello
Neel Nanda reports that Othello board states become linearly readable when expressed as the current player’s pieces versus the opponent’s, rather than absolute colors. Linear interventions provide supporting evidence that the model uses this representation. The example shows that conclusions about linearity depend on choosing appropriate features.
Leverage Score and Feature Dimensionality
Fred Zhang relates feature dimensionality to statistical leverage scores. Leverage scores sum to matrix rank; when feature vectors are suitably isotropic, the paper’s simpler directional statistic approximates them. This explains the observed sum under a geometric condition, rather than asserting it for every arrangement.
Equations
Code
The source provides notebooks reproducing selected core diagrams and theoretical phase boundaries. They are not a complete release of every experiment; the external replications used independent implementations.
Acknowledgments
The authors acknowledge independent replications, theoretical feedback, related-work suggestions, and research support.
Author Contributions
Contributors developed the reconstruction models, phase diagrams, geometric analyses, learning trajectories, adversarial experiments, nonlinear-computation examples, theoretical connections, and illustrations.
Citation Information
The source citation identifies Nelson Elhage and colleagues, Toy Models of Superposition, Transformer Circuits Thread, 2022. The original citation listing is retained.
Elhage, et al., "Toy Models of Superposition", Transformer Circuits Thread, 2022.
@article{elhage2022superposition,
title={Toy Models of Superposition},
author={Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and Hatfield-Dodds, Zac and Lasenby, Robert and Drain, Dawn and Chen, Carol and Grosse, Roger and McCandlish, Sam and Kaplan, Jared and Amodei, Dario and Wattenberg, Martin and Olah, Christopher},
year={2022},
journal={Transformer Circuits Thread},
note={https://transformer-circuits.pub/2022/toy_model/index.html}
}
Nonlinear Compression
A constructed nonlinear encoding packs two bounded coordinates into one by quantizing one coordinate and using the remaining precision for the other. Smooth approximations outperform simple linear compression in the illustrated dense two-coordinate setting. This is a representational possibility, not evidence that real networks commonly learn such schemes; computation, precision, and learning costs matter.
Equation

Connection between compressed sensing lower bounds and the toy model
The appendix claims a compressed-sensing dimension lower bound after converting sufficiently accurate reconstruction into sparse recovery under a restricted-isometry assumption. The original theorem and derivation are preserved. Its support-recovery argument needs stronger conditions: small reconstruction error alone does not identify support, and the stated threshold using the (k+1)th largest entry is zero for exactly k-sparse inputs. The cited uniform sparse-recovery bound should therefore not be read as an unconditional guarantee about the trained toy models’ average loss.