Literature
Display
← All papers

Toy Models of Superposition

Nelson Elhage · Tristan Hume · Catherine Olsson · Nicholas Schiefer · Tom Henighan · Shauna Kravec · Zac Hatfield-Dodds · Robert Lasenby · Dawn Drain · Carol Chen · Roger Grosse · Sam McCandlish · Jared Kaplan · Dario Amodei · Martin Wattenberg · Christopher Olah

· Transformer Circuits Thread ↗

Interpretability14 min text · CondensedOriginal paper ↗
Contents

Overview

Intuition. Imagine representing five switches using only two real-valued readouts. Five independent linear channels will not fit, but if most switches are off, shared directions can still give useful readings. Collisions occur when features overlap; the model trades that interference against the benefit of keeping more features.

Reading the math. The matrix W mixes input features x into a smaller hidden vector h = Wx. A decoder tries to reconstruct x, and ReLU clips negative outputs to zero. Sparsity means features are rarely active, not that the weight matrix contains mostly zeros. The resulting geometry is a learned compromise under the toy loss, rather than five perfectly independent channels.

Small networks trained on synthetic sparse features can represent more features than hidden dimensions by allowing controlled interference between their directions. A nonlinear output filters some of that interference. The experiments reveal phase transitions, geometric arrangements, and limited computation in superposition. They establish these behaviors in toy settings; how much the detailed geometry transfers to real language models remains uncertain. [1]

Key Results From Our Toy Models

The experiments demonstrate both monosemantic and polysemantic neurons, sharp changes in representation strategy, structured feature geometry, and an absolute-value computation using superposed features. These are existence results for simple ReLU networks, not claims that every real network follows the same organization.

Definitions and Motivation: Features, Directions, and Superposition

The conceptual picture separates four questions: whether activations decompose into features, whether those features are represented linearly, whether there are more features than dimensions, and whether features align with neuron coordinates. A privileged basis favors coordinate alignment; superposition can work against it.

Empirical Phenomena

Motivating observations include semantic directions in embeddings, interpretable neurons, recurring features across models, and neurons responding to unrelated inputs. No one observation alone determines a universal definition of a feature.

What are Features?

Candidate definitions include arbitrary input functions, human-understandable properties, or properties that sufficiently large models reliably represent with dedicated neurons. Each has limitations. The paper provisionally favors the last intuition while keeping the definition open.

Features as Directions

A linear representation stores feature values as a weighted sum of feature directions. The features themselves may be nonlinear functions of the original input. Linear accessibility makes them easy for later layers to read, although nonlinear encodings are possible and discussed in the appendix.

Privileged vs Non-privileged Bases

Purely linear surroundings permit a change of coordinates without changing behavior. Coordinatewise nonlinearities break that symmetry and privilege neuron axes, but do not guarantee that individual features align with them.

The Superposition Hypothesis

Superposition trades the benefit of representing extra features against interference between nonorthogonal directions. Sparse feature occurrence and nonlinear filtering can make the trade favorable. Almost-orthogonal vectors and compressed sensing motivate this possibility; the experiments test it directly.

Summary: A Hierarchy of Feature Properties

Decomposability and linearity describe how features combine. Superposition means their linear encoding is not invertible over arbitrary feature combinations; basis alignment is a further restriction. The paper hypothesizes the first two properties broadly while expecting the latter properties to vary by model and regime.

Demonstrating Superposition

The first experiment asks whether adding a simple output nonlinearity allows a narrow hidden space to recover more input features than a linear baseline can represent.

Experiment Setup

An autoencoder maps a known high-dimensional feature vector into a smaller hidden vector and reconstructs it. Knowing the synthetic ground-truth features makes representation and interference directly measurable.

The Feature Vector ( x )

Each input coordinate is zero with a chosen probability and otherwise uniform on [0,1]. Features may differ in importance in the reconstruction loss; most initial experiments give them the same sparsity. Sparse, numerous, unequally useful features are modeling assumptions rather than measurements from a real network.

The Model ( x → x′ )

Both initial models use a linear encoder and its transposed matrix as decoder, plus an output bias. The baseline output is linear; the comparison adds ReLU. Tied weights make feature directions identifiable, and negative biases let the nonlinear model suppress small interference.

The Loss

Training minimizes squared reconstruction error weighted by each feature’s importance. This explicitly controls which errors the model has an incentive to avoid.

EquationL=∑x∑iIi(xi−xi′)2L = \sum_x \sum_i I_i (x_i - x'_i)^2

Basic Results

The linear baseline keeps the most important dimensions. With dense inputs, the ReLU model behaves similarly; as inputs become sparse, it represents extra features nonorthogonally. Antipodal pairs appear first, followed by richer arrangements. The same qualitative pattern occurs in larger toy configurations.

Mathematical Understanding

A loss decomposition exposes competing reconstruction benefits and interference costs. In the sparse limit, zero- and one-active-feature cases dominate. ReLU removes negative interference in the one-feature case, and negative biases can suppress small positive interference. With additional simplifying constraints, the geometry resembles a generalized Thomson point-packing problem.

Superposition as a Phase Change

Two features in one dimension and three features in two dimensions exhibit distinct strategies: ignore a feature, allocate it a dedicated direction, or share space. Restricted analytic candidate solutions and trained models show sharp boundaries as sparsity and relative importance change. These candidate phase diagrams are not an exhaustive solution of every possible weight configuration.

The Geometry of Superposition

Equal feature statistics produce striking regular arrangements, while unequal statistics deform or change those arrangements. The authors explicitly caution that this elegant geometry may depend strongly on the toy setup.

Uniform Superposition

For equal-importance features, the squared encoder norm approximates how many features are represented. The resulting average dimensions-per-feature curve has persistent regions near one and one-half; the latter corresponds to antipodal pairs. Other feature-level fractions reveal additional structures.

Feature Dimensionality

Feature dimensionality compares a feature’s own squared norm with the total squared projection of all features onto its direction. It equals one-half for an equal antipodal pair. Empirical values cluster at fractions associated with regular configurations; their sum approximates hidden dimension in efficiently packed examples.

Equation

Di = ∣∣Wi∣∣2∑j(Wi^⋅Wj)2D_i ~=~ \frac{||W_i||^2}{\sum_j (\hat{W_i} \cdot W_j)^2}

Why these geometric structures?

Fractions such as three-quarters, two-thirds, and two-fifths correspond to tetrahedra, triangles, and pentagons. Orthogonal combinations of smaller polytopes can explain split groups of dimensionalities. Features in different orthogonal factors do not interfere, potentially reducing damaging simultaneous interactions.

Aside: Polytopes and Low-Rank Matrices

A feature arrangement determines its Gram matrix of pairwise dot products, and that matrix determines interference. Conversely, discarded directions constrain possible arrangements. This geometry–matrix connection is useful even when the exact optimal packing is unknown.

Non-Uniform Superposition

Changing importance, sparsity, or co-occurrence deforms the uniform solutions. Correlated features tend to avoid interference if space permits; anticorrelated features can share space more readily. The report offers representative experiments rather than a complete non-uniform theory.

Perturbing a Single Feature

Changing one feature’s activation frequency deforms a five-feature pentagon until a competing two-pair solution becomes preferable and the weakest feature is dropped. Clean two-dimensional results require multiple optimization runs and selection of low-loss solutions; suboptimal runs help reveal competing minima.

Correlated and Anticorrelated Features

Co-occurrence strongly affects which features share dimensions. Correlated features prefer separate directions, then neighboring directions if necessary, and can eventually merge into a principal component. Anticorrelated features favor shared directions with negative interference.

Setup for Exploring Correlated and Anticorrelated Features

Correlated groups share the decision to activate while retaining independently sampled nonzero values. Anticorrelated groups permit at most one active member. These controlled constructions isolate co-occurrence from other natural-data structure.

Organization of Correlated and Anticorrelated Features

Two-dimensional experiments visualize how these feature groups arrange around the hidden space. Rotations and reflections align solutions for comparison; the figures preserve the resulting geometry.

Local Almost-Orthogonal Bases

Larger experiments can put each correlated group into a nearly orthogonal local basis even while different groups share space globally. This suggests a possible local simplification for interpretation, but does not establish such a property across the full distribution of real models.

Collapsing of Correlated Features

When correlated features are too dense or space is too limited, the model can preserve their shared principal component instead of their separate values. Increasing sparsity favors separating them in superposition. Principal-component compression and superposition therefore coexist and trade off in this toy setting.

Superposition and Learning Dynamics

The learning-dynamics experiments track how the final geometric structures form. They are preliminary observations of selected trajectories, not a general optimization theory.

Phenomenon 1: Discrete "Energy Level" Jumps

Individual features sometimes jump between characteristic dimensionalities while the loss drops. This motivates the idea that apparently smooth large-model learning might contain many smaller structural transitions, but that extrapolation is untested.

Phenomenon 2: Learning as Geometric Transformations

A six-feature, three-dimensional example passes through interpretable geometric rearrangements before reaching an octahedral arrangement. Different initializations can take different routes. Connections to initially linear learning and hierarchical feature splitting are suggestive, with different assumptions from the cited theorems.

Relationship to Adversarial Robustness

Analytically constructed attacks with L2 size 10% of average input norm become over three times more damaging as superposition appears in the tested setup. Interference provides an attack direction, but this does not establish the cause of real-model adversarial examples. Adversarial training reduces superposition; eliminating it in the preliminary experiment requires attacks around 80% of input norm.

Equations

(WTW)0  =  (1, 0, 0, 0, ...)(W^TW)_0 ~~=~~ (1,~ 0,~ 0,~ 0,~ ...)

(WTW)0  =  (1, ϵ, −ϵ, ϵ, ...)(W^TW)_0 ~~=~~ (1,~ \epsilon,~ -\epsilon,~ \epsilon,~ ...)

Superposition in a Privileged Basis

Adding ReLU inside the hidden layer privileges neuron coordinates. Some feature directions then align with individual neurons, while sparse regimes still allow mixed representations. This connects the synthetic feature geometry to monosemantic and polysemantic neuron behavior.

Equationsh = ReLU(Wx)h~=~\text{ReLU}(Wx)x′ = ReLU(WTh+b)x'~=~\text{ReLU}(W^Th+b)

Visualizing Superposition in Terms of Neurons

Weight plots reveal both dedicated and mixed neurons, sometimes in the same model. For the small five-neuron example, the displayed solutions are the best of one thousand training runs per setting. The structured mixtures are therefore low-loss examples rather than typical-run statistics.

Limitations of The ReLU Hidden Layer Toy Model Simulating Identity

The identity-reconstruction task does not need a hidden nonlinearity, so the network tries to bypass it using positive biases or averages of many inputs. This limits the relevance of that particular hidden-ReLU toy model and motivates a task requiring nonlinear computation.

Computation in Superposition

The next task computes coordinatewise absolute values. Two ReLU units can represent the positive and negative parts of one feature, giving a simple reference circuit against which computation in superposition can be examined.

Experiment Setup

Nonzero input values now range from −1 to 1, and the target is their absolute value. Encoder and decoder weights are no longer tied. A hidden ReLU and output ReLU train under the same importance-weighted squared-error objective.

Equation

h=ReLU(W1x)h = \text{ReLU}(W_1x) y′=ReLU(W2h+b)y' = \text{ReLU}(W_2h+b)

Basic Results

With enough hidden units, the model learns the expected positive-part and negative-part circuit for each feature. Visualizations normalize equivalent hidden-unit rescalings before comparing weights.

Superposition vs Sparsity

As the task becomes sparser, a forty-neuron model approximates absolute values for more input features through mixed neurons. This demonstrates limited computation in superposition. Losses across sparsity settings are not directly comparable because the input distributions change. Some neurons have a dominant feature plus smaller secondary features, so their largest activations can conceal polysemanticity.

The Asymmetric Superposition Motif

One inspected motif uses unequal encoding magnitudes and reciprocal decoder weights to make interference asymmetric. A second neuron inhibits the harmful positive cross-effect, converting it into interference the output ReLU can suppress. This explains much of the selected circuit, with some smaller weights still not fully characterized.

The Strategic Picture of Superposition

The strategic motivation is to identify useful units for understanding model computation. If important features are hidden in superposition, inspecting neurons alone can leave substantial behavior unexplained.

Safety, Interpretability, & "Solving Superposition"

The authors describe feature enumeration and sparse decoding as a possible route toward more complete circuit analysis. They present this as a research objective: identifying features alone neither proves the absence of harmful behavior nor establishes that all relevant mechanisms have been captured.

Three Ways Out

Three broad approaches are to train models without superposition, recover an overcomplete dictionary afterward, or combine architectural changes with later decoding. Competitive performance and completeness remain challenges for every approach.

Approach 1: Creating Models Without Superposition

Hidden-activation L1 regularization can remove superposition in toy models at a reconstruction cost. Architectures such as mixture-of-experts networks might offer more selectively active neurons for a given compute budget. The suggestion that this could reduce the need for superposition is a hypothesis, not an evaluated capability result here.

Approach 2: Finding an Overcomplete Basis

Post-hoc sparse dictionary learning seeks an overcomplete set of directions explaining observed activations. It preserves the original model’s performance but introduces unknown dictionary size, large computational costs, and a gap between dictionary features and the network’s literal computation.

Approach 3: Hybrid Approaches

Hybrid approaches could reduce superposition slightly or make its decoding easier without eliminating it during training. Whether this can preserve competitive performance at useful scales remains open.

Additional Considerations

Sharp toy-model transitions suggest that complete removal is possible in some regimes. Even a less capable model with clear ground-truth features could aid research. Local bases, however, do not provide the full-distribution understanding required for strong global claims.

Discussion

The discussion separates possible interpretability applications from the remaining questions about real neural networks.

To What Extent Does Superposition Exist in Real Models?

The toy models reproduce several qualitative observations: mixed and dedicated neurons can coexist, sparse concepts encourage mixed representations, and top activations can appear cleaner than weaker ones. This supports their usefulness as models of possible mechanisms, while leaving real-model geometry and dynamics especially uncertain.

Open Questions

Open questions include statistical tests for superposition, controllable phase boundaries, realistic feature statistics, scaling behavior, feature counts, non-robust features, and the class of computations possible in compressed representations. The paper does not settle these broader questions.

Additional Connections

Reader-suggested connections include vector-symbolic architectures, hyperdimensional computing, and mathematical frames. These relate nonorthogonal high-dimensional directions to representation and computation, although the paper does not review them exhaustively.

Comments & Replications

The following external replications and later comments are preserved with attribution and should be distinguished from the original experiments.

Replication & Forthcoming Paper

Kshitij Sachan reports independent reproduction of the basic superposition and phase-diagram experiments and finds that activation choice changes the diagrams. A later update links the resulting Polysemanticity and Capacity in Neural Networks work, which develops a related constrained-optimization account.

Replication & Further Results

Tom McGrath reproduces key results and exactly analyzes the bias-free two-feature, one-dimensional case. The full loss surface reveals a same-direction confused-feature regime absent from the original restricted candidates, continuous movements of some minima, and other discontinuous transitions. This both supports and refines the original phase-diagram picture.

Equation

∫DReLU(f(x))dx=∫D∩f(x)>0f(x)dx\int_D \text{ReLU}(f(x)) dx = \int_{D \cap f(x)>0} f(x) dx

Replication

Jeffrey Wu and Dan Mossing report independent reproduction of the basic results, fractional-dimensionality patterns, and discrete changes during learning.

Replication

Spencer Becker-Kahn reports independent reproduction of key basic plots and small-model results consistent with the geometry and learning-dynamics account.

Engineering Monosemanticity in Toy Models

Adam Jermyn, Evan Hubinger, and Nicholas Schiefer report follow-up toy architectures and initialization methods that can produce monosemantic representations with sufficient capacity. Their results motivate architectural research but do not establish a cost-free solution for real language models.

Fractional Dimensionality and "Pressure"

Tom Henighan and Chris Olah observe cleaner geometric structure when more features compete for representation and when models train longer. This preliminary pressure effect is especially visible at high sparsity and lacks a complete explanation.

Replication

Marius Hobbhahn reports reproduction of the basic reconstruction and privileged-basis results, and most findings from the later memorization and double-descent study.

Extracting Features with Sparse Autoencoders

Lee Sharkey, Dan Braun, and Beren Millidge recover nearly ground-truth directions from a synthetic sparse-mixture dataset using an L1-regularized autoencoder. Dead units, losses, and comparisons across dictionary sizes offer tentative ways to estimate feature count. Their initial language-model activation results are inconclusive.

Linear Representation in Othello

Neel Nanda reports that Othello board states become linearly readable when expressed as the current player’s pieces versus the opponent’s, rather than absolute colors. Linear interventions provide supporting evidence that the model uses this representation. The example shows that conclusions about linearity depend on choosing appropriate features.

Leverage Score and Feature Dimensionality

Fred Zhang relates feature dimensionality to statistical leverage scores. Leverage scores sum to matrix rank; when feature vectors are suitably isotropic, the paper’s simpler directional statistic approximates them. This explains the observed sum under a geometric condition, rather than asserting it for every arrangement.

Equations

Di = ∣∣Wi∣∣2∑j(Wi^⋅Wj)2, D_i ~=~ \frac{||W_i||^2}{\sum_j (\hat{W_i} \cdot W_j)^2},

τi=max⁡x:∥x∥2=1⟨Wi,x⟩2∑j=1n⟨Wj,x⟩2. \tau_i = \max_{x : \|x\|_2 = 1} \frac{\langle W_i , x \rangle^2}{\sum_{j=1}^n \langle W_j , x \rangle^2}.

Code

The source provides notebooks reproducing selected core diagrams and theoretical phase boundaries. They are not a complete release of every experiment; the external replications used independent implementations.

Acknowledgments

The authors acknowledge independent replications, theoretical feedback, related-work suggestions, and research support.

Author Contributions

Contributors developed the reconstruction models, phase diagrams, geometric analyses, learning trajectories, adversarial experiments, nonlinear-computation examples, theoretical connections, and illustrations.

Citation Information

The source citation identifies Nelson Elhage and colleagues, Toy Models of Superposition, Transformer Circuits Thread, 2022. The original citation listing is retained.

Elhage, et al., "Toy Models of Superposition", Transformer Circuits Thread, 2022.
@article{elhage2022superposition,
   title={Toy Models of Superposition},
   author={Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and Hatfield-Dodds, Zac and Lasenby, Robert and Drain, Dawn and Chen, Carol and Grosse, Roger and McCandlish, Sam and Kaplan, Jared and Amodei, Dario and Wattenberg, Martin and Olah, Christopher},
   year={2022},
   journal={Transformer Circuits Thread},
   note={https://transformer-circuits.pub/2022/toy_model/index.html}
}

Nonlinear Compression

A constructed nonlinear encoding packs two bounded coordinates into one by quantizing one coordinate and using the remaining precision for the other. Smooth approximations outperform simple linear compression in the illustrated dense two-coordinate setting. This is a representational possibility, not evidence that real networks commonly learn such schemes; computation, precision, and learning costs matter.

Equation

t=⌊Zx⌋+yZ t = \frac{\lfloor Zx \rfloor + y}{Z}

Connection between compressed sensing lower bounds and the toy model

The appendix claims a compressed-sensing dimension lower bound after converting sufficiently accurate reconstruction into sparse recovery under a restricted-isometry assumption. The original theorem and derivation are preserved. Its support-recovery argument needs stronger conditions: small reconstruction error alone does not identify support, and the stated threshold using the (k+1)th largest entry is zero for exactly k-sparse inputs. The cited uniform sparse-recovery bound should therefore not be read as an unconditional guarantee about the trained toy models’ average loss.

Equation

∥x−x^∥1≤C(k)min⁡x′,∥x′∥0≤k∥x−x′∥1\|x - \hat{x}\|_1 \leq C(k) \min_{x', \|x'\|_0 \leq k} \|x - x'\|_1

Concept demonstration

Open image ↗