Literature
Display
← All papers

A Toy Model of Interference Weights

Chris Olah · Nicholas L. Turner · Tom Conerly

Published · Transformer Circuits Thread ↗

Interpretability4 min text · CondensedOriginal paper ↗
Contents

Overview

This preliminary note studies weight superposition: compressing features into fewer dimensions also entangles the connections between them. Toy experiments reproduce patterns seen in language-model dictionaries and compare possible definitions of interference. These are exploratory findings, not a solved method for globally interpreting full models [1].

Introduction

Even a correct feature dictionary can produce apparent connections that arise from representational overlap. Lifting neuron weights into feature coordinates therefore does not automatically reveal a clean computational circuit.

Should we care about interference weights?

Interference is physically present and can affect robustness, including adversarial inputs. Whether it can be ignored when studying learned objectives is less clear: the authors tentatively distinguish incidental interference from connections favored by training, but explicitly regard that distinction as uncertain. Imperfect features, new learning during optimization, and interactions between weights complicate the claim.

Revisiting Toy Models

The original superposition toy model can be reinterpreted as independent identity computations compressed through a small shared bottleneck [2]. Its effective feature-to-feature weight matrix allows direct study of signal and interference.

Equations

h=Wxh=Wx

x′=ReLU(WTh+b)x' = \text{ReLU}(W^Th+b)

x′=ReLU(WTWh+b)x'=\text{ReLU}(W^TWh+b)

Reproducing Phenomenology from Towards Monosemanticity

Feature-to-logit weights in a one-layer transformer pass through compressed activation spaces. This predicts interference even when individual features are interpretable, offering an explanation for weight distributions reported in Towards Monosemanticity [3].

Toy Model Setup

The initial example compresses 100 features into 20 residual dimensions with feature density 0.02. It is deliberately undertrained because this better resembles the motivating language-model observations; converged cases appear in an appendix.

Interference Weights and Decomposition

Two definitions are compared: deviation from an ideal weight matrix optimized in expanded feature space, or individual weights whose removal does not worsen loss. These are different decompositions. The experiments mainly use each weight’s marginal effect on loss as ground truth.

Weight Histogram

Weight histograms show a large interference mass alongside useful connections, qualitatively resembling real feature-to-logit histograms. Colors and aggregation differ between the toy and language-model plots, so their agreement is qualitative.

Comparing Two Models with a Scatter Plot

Across random seeds, useful identity connections agree while interference varies. Converged models show stronger geometric structure but preserve this separation between stable useful weights and seed-dependent interference.

Getting More Realistic Phenomenology

A harder example is needed because real useful and interference weights overlap in magnitude. A simple small-weight threshold would discard useful connections or retain interference.

A More Sophisticated Toy Model

An untied encoder–decoder approximates a more complex random block-structured circuit. The illustrated case compresses 128 features into 16 dimensions across eight blocks. It reproduces overlap and seed variability, but the chosen regime does not preserve every desired property at full convergence.

Equations

y=ReLU(Ax+v)y = ReLU(A x + v)

h=Wdownxh = W_{down} x

y′=ReLU(Wuph+b)y' = ReLU(W_{up} h+b)

What Should We Do about Interference Weights?

Exact optimization or ablation-based definitions scale quadratically with feature count and are too expensive globally. Cheap candidates include weight magnitude, expected attribution, target-weighted attribution, and activation frequency. Evaluate not only classification precision but how much beneficial loss effect is retained. Summed individual effects are only a proxy for jointly pruning weights.

Conclusion

Interference may be a major obstacle to global circuit analysis, whereas prompt-specific attribution graphs partly avoid it. Toy ground truth and sampled real-model ablations offer ways to develop cheaper filtering rules; no scalable definitive solution is demonstrated.

Author Contributions

Chris Olah led experiments, figures, and writing. Nick Turner investigated detection heuristics in other settings, and Tom Conerly contributed exposition.

Acknowledgments

The report builds on earlier superposition research and feedback from colleagues. Its account credits the prior observations and discussions that motivated these experiments.

Appendix 1: Non-Linear Weight Sharing

Compressing computations need not make model parameters a linear sum of independent components. Two antipodal features can share a single multiplying weight, and partial sharing creates more complicated cases. This challenges a naive compressed-sensing or additive-parameter view [7].

Appendix 2: More Fully Converged Histograms and Isotropic Superposition Geometry

Fully converged isotropic models produce structured weight distributions: density 0.25 yields pentagonal geometry, whereas density 0.001 shows a different apparent upper bound on interference. The main figures’ undertraining is therefore a substantive modeling choice.

Appendix 3: Correlated Feature Toy Models

With two mutually exclusive groups of correlated features, a low-rank component excites the active group and suppresses the other. Removing this component harms loss even though it partly compensates for interference. Useful structure need not reduce to independent pairwise weights.

Appendix 4: Different Definitions of Interference Weights

Individual ablations can be misleading when weights jointly carry a signal, and an optimized ideal model may learn connections absent from the compressed model. Learning sparse masks on existing weights avoids some of this problem [5]. The appendix treats these definitions as competing research choices, with no settled canonical answer.

Concept demonstration

Open image ↗