Literature
Display
← All papers

A Mathematical Framework for Transformer Circuits

Nelson Elhage · Neel Nanda · Catherine Olsson · Tom Henighan · Nicholas Joseph · Ben Mann · Amanda Askell · Yuntao Bai · Anna Chen · Tom Conerly · Nova DasSarma · Dawn Drain · Deep Ganguli · Zac Hatfield-Dodds · Danny Hernandez · Andy Jones · Jackson Kernion · Liane Lovitt · Kamal Ndousse · Dario Amodei · Tom Brown · Jack Clark · Jared Kaplan · Sam McCandlish · Chris Olah

· Transformer Circuits Thread ↗

Interpretability12 min text · CondensedOriginal paper ↗
Contents

Overview

This foundational study rewrites small attention-only transformers as sums of paths from input tokens to output logits. Zero-layer models learn bigrams; one-layer models add factored skip-trigram rules; two-layer models can compose heads into an induction mechanism that copies continuations seen earlier in context. These results concern deliberately simplified models, and do not amount to reverse-engineering a full language model. [1]

Summary of Results

The summary separates concrete reverse-engineering results from reusable conceptual tools.

Reverse Engineering Results

The main reverse-engineering result is a progression from token-pair statistics to context-dependent rules and then to composed algorithms. Expanded weight products expose these behaviors directly. In particular, a previous-token head and a later induction head can implement the pattern A B … A → B.

Conceptual Take-Aways

Heads contribute independently and additively to the residual stream. Each separates where to attend, governed by its query–key circuit, from what to transmit, governed by its output–value circuit. Matrix products describe communication through the residual stream; composing heads changes either their attention pattern or the information they carry.

Transformer Overview

The framework chooses mathematically equivalent representations that make the computation easier to inspect. Such representations need not be efficient implementations. The notation appendix records the activation spaces, weight shapes, and tensor-product conventions.

Model Simplifications

The main experiments remove MLP layers, a major simplification motivated partly by the difficulty of interpreting them. Biases are omitted from the presentation and can be represented with an added constant coordinate. LayerNorm contributes input-dependent scale factors that cannot simply be discarded; the appendix explains how the remaining affine operations can be absorbed into weights.

High-Level Architecture

An autoregressive decoder embeds input tokens, repeatedly adds attention and MLP outputs to a shared residual stream, and unembeds that stream into next-token logits. Heads within one attention layer operate in parallel.

Virtual Weights and the Residual Stream as a Communication Channel

The residual stream is an additive communication space linking embeddings, heads, MLPs, and the output. Its linear structure allows direct analysis of how one component’s output is read by another, while separating the changing normalization scales.

Virtual Weights

Multiplying a component’s output matrix by a later component’s input matrix produces a virtual weight. This product measures their direct communication through the residual stream even when many layers separate them.

Subspaces and Residual Stream Bandwidth

Individual heads read and write relatively small subspaces. Information persists until another component changes it, and many components must share the residual stream’s limited dimensions. Exploratory weight patterns suggest that some components may delete stored information, but this memory-management interpretation remains tentative.

Attention Heads are Independent and Additive

Concatenating head results and applying one output matrix is algebraically equivalent to applying each head’s own output block and summing the results. The latter form exposes independent additive contributions.

Equation

WOH[rh1rh2...]  =  [WOh1, WOh2, ...]⋅[rh1rh2...]  =  ∑iWOhirhiW_O^H \left[\begin{matrix}r^{h_1}\\r^{h_2}\\... \end{matrix}\right] ~~=~~ \left[W_O^{h_1},~ W_O^{h_2},~ ... \right]\cdot\left[\begin{matrix}r^{h_1}\\r^{h_2}\\...\end{matrix}\right] ~~=~~ \sum_i W_O^{h_i} r^{h_i}

Attention Heads as Information Movement

For a fixed attention pattern, a head applies one linear operation across token positions and another across feature coordinates. Their tensor product combines information routing with its output–value transformation. The query–key product separately computes the attention scores before the masked softmax.

Equations
Source diagram 7
View original figure
Source diagram 8
View original figure

Observations about Attention Heads

This decomposition makes several properties explicit: fixed-pattern attention is linear, query and key factors matter through their product, and value and output factors likewise combine into one low-rank map. Value-composed heads form virtual heads with composed routing patterns and weight products.

Zero-Layer Transformers

With no attention or MLP layers, embedding followed by unembedding predicts the next token solely from the present token, approximating bigram statistics. The same direct path exists inside deeper models, where other components can share responsibility for those statistics. Unembedding is therefore not generally the inverse of embedding.

Equation

T = WUWET ~=~ W_U W_E

One-Layer Attention-Only Transformers

One attention layer adds rules of the form source … destination → output. The destination chooses which earlier source to attend to, and that source changes the output logits. These factored skip-trigrams can express useful behaviors despite the model’s shallow architecture.

The Path Expansion Trick

Expanding the product of embedding, attention layer, and unembedding gives an additive sum over complete paths. One path supplies direct bigram effects; each remaining first-order path passes through an attention head.

Splitting Attention Head terms into Query-Key and Output-Value Circuits

The expanded query–key matrix scores source–destination token pairs. The expanded output–value matrix maps each attended source token to output-logit changes. Given equal attention weight, the same source has the same direct logit effect at different destinations, while softmax competition still involves all available sources.

OV and QK Independence (The Freezing Attention Patterns Trick)

Run the model once to collect its attention patterns, then reuse those patterns. The second run is linear in its token representations for the attention-only analysis. This thought experiment separates routing from transmitted information; it does not make attention independent of the input in the original model.

Interpretation as Skip-Trigrams

Expanded matrices contextualize weights as token-pair effects, but remain enormous: a vocabulary near 50,000 creates matrices with about 2.5 billion entries. Low rank, correlated tokens, overlapping head roles, and query-specific score normalization complicate their interpretation. The source links complete matrix-entry dumps alongside selected examples.

Copying / Primitive In-Context Learning

Many one-layer heads copy an earlier token when the current token makes it a plausible continuation. Some also handle tokenizer fragments, such as copying a whole word into a differently segmented spelling. This is a primitive use of context, weaker than learning an arbitrary continuation from an earlier example.

Other Interesting Skip-Trigrams

Selected skip-trigrams capture indentation and code idioms, argument patterns, bracket closure, markup, fixed expressions, and website paths. Their apparent sophistication often depends on the tokenizer exposing useful units such as indentation or escape prefixes.

Primarily Positional Attention Heads

A minority of heads predominantly use position, such as attending to the current or previous token. Sinusoidal position coordinates can implement relative offsets through rotations; the exact treatment depends on the position-encoding scheme.

Skip-Trigram "Bugs"

Factoring a three-token rule into two pairwise interactions can create unwanted cross-combinations. A head supporting keep … in → mind and keep … at → bay can also encourage mismatched continuations. These examples make a specific failure mechanism visible, but the paper does not quantify its aggregate loss cost.

Summarizing OV/QK Matrices

Despite their large expanded size, query–key and output–value matrices have head-sized rank and sometimes recognizable clusters. Matrix decompositions may summarize them better than inspecting individual entries; the paper treats this as an open methodological direction.

Detecting Copying Behavior

Positive eigenvalues, positive diagonal entries, and self-token rankings provide evidence for copying behavior. In one twelve-head model, ten heads show strong copying signatures. Positive eigenvalues alone are not a conclusive definition of copying, because nonorthogonal eigenvectors can produce counterintuitive token-level effects.

Do We "Fully Understand" One-Layer Models?

The framework removes much of the algorithmic mystery from one-layer attention-only models by exposing interpretable tables. It does not compress every learned fact into a small human-readable account; understanding the lookup structure differs from retaining the entire model in one’s head.

Two-Layer Attention-Only Transformers

A second attention layer permits one head to change another head’s inputs. This compositional ability can create algorithms unavailable to a single attention layer, including the minimal induction circuit studied here.

Three Kinds of Composition

Query composition changes the destination-side attention computation; key composition changes the source-side computation; value composition changes the transmitted information. The first two enrich routing, while the third yields virtual attention heads.

Path Expansion of Logits

The two-layer logit expansion contains a direct path, individual-head paths, and value-composed two-head paths. The latter are mathematically available but contribute comparatively little in the particular small model analyzed.

Path Expansion of Attention Scores QK Circuit

Expanding the second layer’s query–key computation reveals paths on both the query and key sides. These higher-order routing terms capture how first-layer outputs change which positions later heads attend to, beyond what the logit expansion alone reveals.

Analyzing a Two-Layer Model

The example two-layer model contains strong key composition between a previous-token head and later induction heads. The originally published composition diagram underestimated additional interactions; the retained correction below updates that part of the analysis.

Correction

A low-rank linear-algebra bug affected the original composition-strength plot. It strongly highlighted induction-related interactions while missing others. Read that plot together with the corrected figure and explanation later in the article. The accompanying attention examples are exploratory views of a short passage, rather than sufficient evidence by themselves.

Source diagram 23
View original figure

Induction Heads

Induction heads are the main composed algorithm identified in these small models. They extend simple copying by conditioning the copied continuation on a previous occurrence of the current token.

Function of Induction Heads

An induction head searches for an earlier matching token and copies the token that followed it: A B … A → B. When there is no match it can default to the start token. Repeated random-token sequences test this behavior beyond ordinary word statistics, providing stronger evidence than a familiar prose example alone.

Source diagram 25
View original figure

How Induction Heads Work

In the minimal circuit, an earlier previous-token head writes each token’s identity into the next position. A later head uses this shifted information as a key, matches its current query, and copies the continuation through its output–value circuit. Other models can use broader context or positional-pointer mechanisms.

Checking the Mechanistic Theory

The proposed mechanism predicts both an output–value copying matrix and a matching matrix in the shifted-key path. Induction heads score strongly on both eigenvalue-based diagnostics. This corroborates the mechanistic account, while retaining the limitations of eigenvalues as a universal copying test.

Term Importance Analysis

The retained ablation algorithm freezes attention patterns and repeatedly replaces head outputs with those from the preceding run to limit value-path depth. Loss differences estimate the marginal contribution of successive orders. Second-order value paths have small effects in this model; this says nothing against important query or key composition.

Algorithm for measuring marginal loss reduction of nnth order terms
Step 1: Run model, save all attention patterns.
Step 2: Run model, forcing all attention patterns to be the version you recorded, and instead of adding attention head outputs to residual stream, save the output, and then replace it with a zero tensor of the same shape. Record resulting loss.
Step nn: Run model, forcing all attention patterns to be the version you recorded, and instead of adding attention head outputs to residual stream, save the output, and replace it with the value you saved for this head last time. Record resulting loss.

Virtual Attention Heads

A value-composed virtual head has the product of two attention patterns and the product of their output–value maps. Such paths could combine semantic routing operations and grow rapidly in number with depth. Their wider importance is a hypothesis here, not an empirical result for large models.

Where Does This Leave Us?

The framework supplies tools for primarily attentional circuits in full transformers, but most parameters and many head interactions involve MLPs. Understanding these simplified models is therefore a foothold for larger-model analysis, not a complete account of it.

Circuits

The approach extends the earlier vision Circuits project while adapting to attention, bilinear routing, and the linear residual stream. Attention heads form comparatively large inspectable units, but adding MLPs restores the difficult problem of understanding nonlinear feature representations.

The Logit Lens

The logit lens applies the final unembedding at intermediate residual states. This framework instead decomposes those states into contributions and follows their weight products backward, exploiting the same additive structure more extensively. [18]

Attention Head Analysis

Earlier attention studies visualize patterns and relate heads to linguistic structures. This work seeks an end-to-end mechanism in smaller autoregressive models. It shares observations such as previous-token heads, default attention positions, and the value of scaling attention weights by value-vector magnitude.

Criticism of Attention as Explanation

Attention weight alone need not measure a token’s total causal contribution. Induction provides a concrete example: its visible copying depends on an earlier head constructing the keys. Saturated softmax scores can also hide that dependence from local gradient attribution.

Bertology Generally

Bertology includes attention analysis and probes of linguistic information in representations. This paper mainly intersects the attention-analysis branch because it studies attention-only models and emphasizes mechanistic decomposition over supervised probing.

Mathematical Framework

Many mathematical ingredients, including path expansions and tensor formulations of attention, have precedents. The contribution is their combination and use in reverse-engineering concrete transformer behavior. [31]

Other Interpretability Directions

Other interpretability approaches examine individual neurons, training-data influence, saliency, and feature visualization. They provide complementary questions and tools, each with its own known failure modes.

Interpretability Interfaces

Interactive views of attention, activations, and weights help researchers explore high-dimensional structure and test hypotheses. Summary statistics alone can conceal the distinctions needed for a mechanistic account.

Recent Architectural Changes

The framework also interprets architectural variants. Convolving nearby positions before computing attention can make induction-like routing expressible without the same inter-head composition. Talking-head attention shares components across query–key and output–value maps.

Comments & Replications

The comments contain later author updates and corrections. They are preserved alongside the original analysis so the reader can distinguish the initial result from its subsequent qualification.

Summary of Follow-Up Research

A February 2023 update surveys follow-up work on MLP polysemanticity, toy superposition, feature extraction, attention circuits, induction, and in-context optimization. These later studies extend the original foothold but do not resolve full-model interpretability.

Correction: Attention Head Composition Diagram

The bug incorrectly transferred a cyclic matrix-product identity to the Frobenius norm. The resulting statistic emphasized a special alignment relevant to induction rather than general head composition. The corrected plot keeps the important previous-token key path and reveals additional interactions, including induction heads using a head that attends across several recent tokens.

Additional Resources

Companion materials include a problem set, the PySvelte visualization library, and a description of Garcon, the model-inspection infrastructure used for these studies.

Acknowledgments

The authors acknowledge detailed feedback, infrastructure support, and reader-reported corrections.

Author Contributions

The framework emerged from joint theoretical and empirical work; collaborators contributed toy-model analysis, large-model tooling, visualization, training infrastructure, writing, and research-cluster support.

Citation Information

The original citation identifies Nelson Elhage and colleagues, A Mathematical Framework for Transformer Circuits, Transformer Circuits Thread, 2021. The source citation listing is retained.

Elhage, et al., "A Mathematical Framework for Transformer Circuits", Transformer Circuits Thread, 2021.
@article{elhage2021mathematical,
   title={A Mathematical Framework for Transformer Circuits},
   author={Elhage, Nelson and Nanda, Neel and Olsson, Catherine and Henighan, Tom and Joseph, Nicholas and Mann, Ben and Askell, Amanda and Bai, Yuntao and Chen, Anna and Conerly, Tom and DasSarma, Nova and Drain, Dawn and Ganguli, Deep and Hatfield-Dodds, Zac and Hernandez, Danny and Jones, Andy and Kernion, Jackson and Lovitt, Liane and Ndousse, Kamal and Amodei, Dario and Brown, Tom and Clark, Jack and Kaplan, Jared and McCandlish, Sam and Olah, Chris},
   year={2021},
   journal={Transformer Circuits Thread},
   note={https://transformer-circuits.pub/2021/framework/index.html}
}

Additional Intuition and Observations

These appendices extend the framework to MLPs, convolution-like structure, activation spaces, and position-based algorithms.

MLP Layers

MLP nonlinearities prevent the same complete linear expansion. For a one-layer transformer, however, weight products still expose which token and attention paths influence each neuron before its activation and how that neuron affects logits afterward. Extending this approach depends on interpreting the neurons and their interactions, which was the main unresolved obstacle.

Equations

aprem = WIm⋅(Id+∑h∈H1Ah⊗WOVh)⋅ WEa^m_\text{pre} ~=~ W^m_I \cdot \bigg(Id+\sum_{h\in H_1} A^h \otimes W_{OV}^h\bigg)\cdot~W_E

aprem = WImWE + ∑h∈H1Ah⊗(WImWOVhWE)a^m_\text{pre} ~=~ W^m_I W_E ~+~ \sum_{h\in H_1} A^h \otimes \left(W^m_I W_{OV}^h W_E\right)

Virtual Weights and Convolution-like Structure

Virtual weights can resemble convolution kernels indexed by attention patterns instead of fixed spatial offsets. Fixed-offset patterns recover ordinary convolution, while dynamic patterns provide a looser analogy to content-dependent or graph-based convolutions. Organizing weights this way may make circuits easier to inspect.

Equations

y=(Id⊗WId+∑hAh⊗Wh)x+…y = (\text{Id} \otimes W_\text{Id} + \sum_h A^h \otimes W_h) x + …

W∗x = ∑vAv⊗WvW \ast x ~=~ \sum_v A^v \otimes W_v

Activation Properties

The appendix distinguishes spaces with a privileged coordinate basis from spaces whose coordinates can be freely changed in the idealized model. It also identifies bottlenecks where many components communicate through fewer dimensions. These properties guide which interpretation methods are appropriate; a bottleneck is not itself an explanation of the encoded features.

Pointer Arithmetic with Positional Embeddings

When position embeddings enter the residual stream, a head can copy a position and another can transform it to address a neighboring token. The authors observed an induction-like instance in GPT-2. Their main experiments exclude this route by adding position information only to query and key computations.

Identity Attention Heads?

Representing the residual identity path as an extra identity attention head could simplify the notation. The article keeps it separate because its fixed, full-rank behavior differs from learned attention heads.

Notation

The notation reference specifies the model’s vectors, matrices, and tensor operations.

Variable Definitions

The original reference table records model and attention activations, parameter shapes, combined query–key and output–value weights, MLP quantities, and functions. It also marks the basis properties of the different spaces.

Main Model Activations and Parameters

Variable

Shape / Type

Description

T(t)T(t)

[ncontext, nvocab][n_\text{context},~ n_\text{vocab}]

Transformer logits, for tokens tt [activation, privileged basis]

tt

[ncontext, nvocab][n_\text{context},~ n_\text{vocab}]

One-hot encoded tokens [activation, privileged basis]

xnx^n

[ncontext, dmodel][n_\text{context},~ d_\text{model}]

“Residual stream” or “embedding” vectors of model at layer nn (one vector per context token) [activation, not privileged basis]

WEW_E

[dmodel,nvocab][d_\text{model}, n_\text{vocab}]

Token embedding [parameter]

WPW_P

[dmodel,ncontext][d_\text{model}, n_\text{context}]

Positional embedding [parameter]

WUW_U

[nvocab,dmodel][n_\text{vocab}, d_\text{model}]

Unembedding / softmax weights  [parameter]

Attention Heads Activations and Parameters

Variable

Shape / Type

Description

HnH_n

Set

Set of attention heads at layer nn

h(x)h(x)

[ncontext, dmodel] [n_\text{context},~ d_\text{model}]

Output of attention head hh [activation, not privileged basis]

AhA^h

[ncontext, ncontext][n_\text{context},~ n_\text{context}]

Attention pattern of attention head hh [activation, privileged basis]

qh,kh,vh,rhq^h, k^h, v^h, r^h

[ncontext, dhead][n_\text{context},~ d_\text{head}]

Query, key, value and result vectors of attention head hh (one vector per context token) [activation, not privileged basis]

WQh,WKh,WVhW^h_Q, W^h_K, W^h_V

[dhead, dmodel][d_\text{head},~ d_\text{model}]

Query, key, and value weights of attention head hh [parameter]

WOhW^h_O

[dmodel, dhead][d_\text{model},~ d_\text{head}]

Output weights of attention head hh [parameter]

WOVhW^h_{OV}

[dmodel, dmodel][d_\text{model},~ d_\text{model}]

WOVh=WOhWVhW^h_{OV} = W^h_{O}W^h_{V} [parameter, low-rank]

WQKhW^h_{QK}

[dmodel, dmodel][d_\text{model},~ d_\text{model}]

WQKh=WQThWKhW^h_{QK} = W^h_{Q^T}W^h_{K} [parameter, low-rank]

MLP Layer Activations and Parameters

Variable

Shape / Type

Description

m(x)m(x)

[ncontext, dmodel] [n_\text{context},~ d_\text{model}]

Output of MLP layer mm [activation, not privileged basis]

ama^m

[ncontext, dmlp][n_\text{context},~ d_\text{mlp}]

Activations of MLP layer mm [activation, privileged basis]

WImW^m_I

[dmlp, dmodel][d_\text{mlp},~ d_\text{model}]

Input weights for MLP layer mm [parameter]

WOmW^m_O

[dmodel, dmlp][d_\text{model},~ d_\text{mlp}]

Output weights for MLP layer mm [parameter]

Functions

Variable

Shape / Type

Description

GeLU()\text{GeLU}()

Function

Gaussian Error Linear Units [77]

softmax∗()\text{softmax}^*()

Function

Softmax with autoregressive mask for attention distribution creation.

Tensor Product / Kronecker Product Notation

A tensor product separates operations across token positions from operations across each token’s vector coordinates. Equivalent interpretations use left–right matrix multiplication, a Kronecker product on flattened activations, or a higher-order tensor. The mixed-product identity underlies the path expansions.

Technical Details

The technical notes document the models, normalization treatment, and efficient low-rank calculations.

Model Details

The examples use zero-, one-, and two-layer decoder-only attention models, usually with twelve heads of sixty-four dimensions; one stated example uses thirty-two heads of 128 dimensions. Context length is 2,048 with dense attention. Sinusoidal position embeddings are added at the query and key inputs rather than into the residual stream.

Handling Layer Normalization

LayerNorm’s mean subtraction and learned affine parameters can be incorporated into neighboring matrices, but its input-dependent normalization scale remains. These scales reweight paths and complicate comparisons between paths passing through different layers. In the first layer, normalization can instead be applied directly to the token-embedding table.

Working with Low-Rank Matrices

Keep large low-rank matrices factored and perform operations around their narrowest dimension. Nonzero eigenvalues can be obtained from the smaller reversed product, and a short sequence of singular-value decompositions yields the full product’s decomposition. These valid identities do not justify cyclically reordering factors inside a Frobenius norm, the mistake identified in the correction.

Concept demonstration

Open image ↗