Overview
Across 34 transformer models, induction-head formation coincides with an early improvement in using longer contexts. Architectural interventions, small-model ablations, weight analysis, and selected larger-model examples connect the two. Evidence that induction supplies most of this improvement is strongest for small attention-only models; the corresponding claim for large models remains an indirect hypothesis. [7]
Key Concepts
The definitions distinguish loss-based context learning, behavioral induction heads, and comparisons of model outputs over training.
In-context Learning
The paper measures in-context learning primarily as the difference between average next-token loss at positions 500 and 50, rather than few-shot task accuracy. With that late-minus-early convention, better use of context makes the score more negative; parts of the narrative instead describe the positive loss reduction. This broad metric captures benefits from additional context but does not isolate learning a particular new task. [13]
Induction Heads
An induction head empirically performs A B … A → B on repeated random-token sequences: it attends to the earlier continuation B and increases B’s logit. Requiring both prefix matching and copying separates this behavior from a head that merely repeats plausible tokens. The behavioral definition permits different implementations and does not require perfect scores.

Per-Token Loss Analysis
For each model checkpoint, record losses on a fixed collection of 10,000 sampled tokens and treat them as a vector of behavior. PCA projects these vectors into a small space to compare training trajectories. The first two components summarize major changes but omit other behavioral variation.

Arguments that induction heads are the mechanism for the majority of in-context learning.
Six arguments combine synchronized development, architectural changes, ablations, concrete generalization examples, mechanistic analysis, and extrapolation across scale. The evidence table distinguishes some contribution from majority contribution. The latter is causally supported in small attention-only models and remains much less secure in large models with MLPs.
Argument 1: Transformer language models undergo a “phase change” during training, during which induction heads form and simultaneously in-context learning improves dramatically.
During an early training window, induction scores rise, the context-loss gap grows, the loss curve develops a bend, and per-token-loss trajectories turn in PCA space. Standard one-layer models lack the same transition. Timing varies with the model setup: the overview emphasizes roughly 2.5–5 billion tokens, while the small-model details describe about 1–3 billion. Co-occurrence is evidence of a connection, not by itself proof of the complete causal story.





Looking at the Phase Change More Closely
The transition does not align with the later scheduled weight-decay change, and a books-only dataset reproduces the qualitative pattern. An illustrated passage shows improved prediction of repeated continuations and worse prediction when a familiar prefix receives a new continuation. This is an intuitive example, not a representative quantitative evaluation.


Assessing the Evidence
Other mechanisms may emerge with induction, and large models have only fourteen or fifteen saved checkpoints, limiting temporal resolution. A stable late-minus-early loss gap also does not imply a stable mechanism: both losses continue improving. The causal interpretation is therefore stronger near the transition than as a complete explanation at the end of training.
Argument 2: When we change the transformer architecture in a way that shifts when induction heads form or whether they can form, the dramatic improvement in in-context learning shifts in a precisely matching way.
A learned mixture of the current and previous token’s key gives a single attention layer access to the information needed for induction. With this smeared-key architecture, one-layer models acquire induction and the context-loss improvement, while two-layer versions acquire them earlier. The targeted one- and two-layer intervention supports the mechanism but does not establish that it remains the only important mechanism at large scale.
Equation

Argument 3: When we directly “knock out” induction heads in small models at test-time, the amount of in-context learning greatly decreases.
Pattern-preserving head ablations remove an individual head’s output while keeping attention patterns from the original run. In small attention-only models, induction heads account for nearly all measured context improvement; evidence is less decisive when nonlinear MLP interactions intervene. No full-scale-model head-ablation experiment is reported. Marginal ablations can also miss redundant contributions or shift the comparison by harming ordinary prediction.

Argument 4: Despite being defined narrowly as copying random sequences, induction heads can implement surprisingly abstract types of in-context learning.
Selected heads in a forty-layer model satisfy the random-sequence induction test and also contribute to literal repetition, translation, or a synthetic category task. These examples show that the empirical induction definition can include more abstract behavior. They establish possibilities, not a population estimate of how much large-model in-context learning induction explains.
Head | Layer Depth | Copying score (?) | Prefix matching score (?) |
Literal copying head | 21 / 40 | 0.89 | 0.75 |
Translation head | 7 / 40 | 0.20 | 0.85 |
Pattern-matching head | 8 / 40 | 0.69 | 0.94 |
Behavior 1: Literal sequence copying
The literal-copying example attends to earlier occurrences of repeated names, phrases, and sentences. Attention and direct-path logit attribution provide complementary views: where information is read and how that head directly changes the relevant output prediction.

Behavior 2: Translation
A selected translation head follows semantically corresponding words across English, French, and German passages. Word order and tokenization produce imperfect alignment, and direct logit effects are not uniformly sharp. The head contributes to correct translation in these examples, potentially alongside later processing.

Behavior 3: Pattern matching
The synthetic task labels four combinations of month or color with animal or fruit. A selected head often attends to an earlier example of the same category rather than simply the same word; about 65% of its attention from the label-predicting colons goes to the correct category in related test sequences. This is an attention-allocation result, not 65% end-to-end task accuracy.

What’s going on with more abstract heads that are also induction heads?
Literal induction may be a special case of matching and copying in a more abstract representation. Earlier layers could supply those representations, or the head could implement a broadly similar nearest-neighbor rule. The examples are consistent with these explanations but do not fully identify the larger circuits.
Argument 5: For small models, we can explain mechanistically how induction heads work, and can show they contribute to in-context learning. Furthermore, the actual mechanism of operation suggests natural ways in which it could be re-purposed to perform more general in-context learning.
A mechanism that retrieves earlier matching contexts and their continuations should help prediction as more context becomes available. Weight-level analysis establishes such a mechanism for small attention-only circuits. It supports some contribution generally, while majority contribution requires additional evidence about competing mechanisms.
Summary of Reverse Engineering Induction Heads
The output–value circuit copies the attended token. The query–key circuit matches a current token against information shifted by an earlier previous-token head, usually through key composition. Broader matches can also use query-side context. A positional-pointer alternative observed in GPT-2 is unavailable in the principal models because their positional information is not written into the residual stream.

What About More Complex Induction Heads?
The authors cannot fully reverse-engineer the larger, more abstract induction examples. They hypothesize that richer query–key paths and more abstract representations extend the same matching-and-copying operation.
Argument 6: Extrapolation from small models suggests induction heads are responsible for the majority of in-context learning in large models.
Similar training curves and induction signatures across scales motivate extrapolation from small models. The main alternative is that larger models develop additional compositional mechanisms during the same transition. Similar aggregate measurements do not rule out those mechanisms becoming more important than induction.
Model Analysis Table
The analysis combines small attention-only models, small models with MLPs, a full-scale series, and targeted smeared-key variants. There is one training run per model configuration. Models sharing a dataset see the same examples in the same order without repeating training data; separate books-only variants test one dataset change. The complete comparison panels are retained.
Small Attention-Only Transformers
This experiment series studies small attention-only transformers, because these are the models we have the strongest theoretical understanding of (see the framework paper). For these small models, we're able to capture snapshots with a high time resolution over early training. These experiments are the ones that most cleanly demonstrate the phase change and link it to induction heads. Note how (1) many models undergo a discontinuous phase change (seen most clearly in the PCA plot and in-context learning score); (2) it appears to be caused by induction heads (see ablations); and (3) the only model without a phase change is the one-layer model. More...
One Layer
(Attention-Only)
Two Layer
(Attention-Only)
Three Layer
(Attention-Only)
Four Layer
(Attention-Only)
Five Layer
(Attention-Only)
Six Layer
(Attention-Only)





PCA of Token Losses
The vector of per-token losses is a way to map different neural network behavior to the same vector space. We take 10,000 individual token predictions per model, and project them onto the first two principal components. This shows how the large-scale-behavior of multiple networks evolve over training. More...





Models Over Training





Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.





Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...






Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...






"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...





Attention Heads Over Training





"Prefix matching score"
Each line is an attention head. Y axis is the average fraction of a head's attention weight given to the token we expect an induction head to attend to -- the token where the prefix matches the present context -- on random, synthetic data. Heads are colored by layer depth: early; late. More...





Trace of QK eigenvalues
Each line is an attention head. Y axis is the trace of the in the attention head's QK circuit term, scaled by how well matches an ideal previous token head. Only defined for models more than one layer deep. Heads are colored by layer depth: early; late. More...




Ablation to "before-and-after" vector
Each line is an attention head. Y axis is the projection
of the observed per-token loss differences when ablating
an attention head onto the difference in per-token
losses before and after the bump. More...
Heads are colored by type:
previous token;
induction;
other.




Ablation attribution to "in-context learning score"
Each line is an attention head. Y axis is the change in
the "in-context learning score" observed when ablating
each attention head. More...
Heads are colored by type:
previous token;
induction;
other.






Small Transformers (with MLPs)
This experiment series studies small normal transformers (with MLP layers, unlike the above). For these small models, we're able to capture snapshots with a high time resolution over early training. The main observation to take away is that the results are essentially the same as the attention-only models (where we have a stronger theoretical understanding. More...
One Layer
(with MLPs)
Two Layer
(with MLPs)
Three Layer
(with MLPs)
Four Layer
(with MLPs)
Five Layer
(with MLPs)
Six Layer
(with MLPs)





PCA of Token Losses
The vector of per-token losses is a way to map different neural network behavior to the same vector space. We take 10,000 individual token predictions per model, and project them onto the first two principal components. This shows how the large-scale-behavior of multiple networks evolve over training. More...





Models Over Training





Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.





Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...






Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...






"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...





Attention Heads Over Training





"Prefix matching score"
Each line is an attention head. Y axis is the average fraction of a head's attention weight given to the token we expect an induction head to attend to -- the token where the prefix matches the present context -- on random, synthetic data. Heads are colored by layer depth: early; late. More...





Trace of QK eigenvalues
Each line is an attention head. Y axis is the trace of the in the attention head's QK circuit term, scaled by how well matches an ideal previous token head. Only defined for models more than one layer deep. Heads are colored by layer depth: early; late. More...




Ablation to "before-and-after" vector
Each line is an attention head. Y axis is the projection
of the observed per-token loss differences when ablating
an attention head onto the difference in per-token
losses before and after the bump. More...
Heads are colored by type:
previous token;
induction;
other.




Ablation attribution to "in-context learning score"
Each line is an attention head. Y axis is the change in
the "in-context learning score" observed when ablating
each attention head. More...
Heads are colored by type:
previous token;
induction;
other.






Full-scale Transformers
This experiment series analyzes a sweep of typical transformers, spaced exponentially in size, up to a 13 billion parameter model. The goal is to show that the phenomena we observed in small models can be seen in larger models as well. For these models, we unfortunately have lower time resolution (note that the x-axis of most plots has changed to a log scale). We are also unable to provide complete ablations. The main observation to make is that results are roughly the same as seen in the small models. More...
4-Layer
Transformer
(13M Params)
6-Layer
Transformer
(42M Params)
10-Layer
Transformer
(200M Params)
16-Layer
Transformer
(810M Params)
24-Layer
Transformer
(2.7B Params)
40-Layer
Transformer
(13B Params)





PCA of Token Losses
The vector of per-token losses is a way to map different neural network behavior to the same vector space. We take 10,000 individual token predictions per model, and project them onto the first two principal components. This shows how the large-scale-behavior of multiple networks evolve over training. More...





Models Over Training





Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.





Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...






Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...






"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...





Attention Heads Over Training





"Prefix matching score"
Each line is an attention head. Y axis is the average fraction of a head's attention weight given to the token we expect an induction head to attend to -- the token where the prefix matches the present context -- on random, synthetic data. Heads are colored by layer depth: early; late. More...





"Smeared Key" Architecture Modification + Controls
This experiment series analyzes "smeared key" models, which are designed so that a single attention layer can implement induction heads (see Argument 2). For each smeared key model, a regular control model is provided (trained exactly the same) for comparison. The key observations are that (1) a one-layer smeared-key model experiences the phase change, while a regular one-layer model does not; and (2) the two-layer smeared-key model experiences a phase change earlier than a baseline model. More...
One Layer
(Vanilla)
One Layer
(Smeared Keys)
Two Layer
(Vanilla)
Two Layer
(Smeared Keys)



Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.



Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...




Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...




"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...



Small Attention-Only Transformers (different dataset)
This experiment series studies transformers of the same architecture as the small attention-only transformers, on a different dataset consisting only of internet books. More...
One Layer
(Attention-Only)
Two Layer
(Attention-Only)
Three Layer
(Attention-Only)
Four Layer
(Attention-Only)
Five Layer
(Attention-Only)
Six Layer
(Attention-Only)





Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.





Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...






Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...






"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...





Small Transformers (with MLPs) (different dataset)
This experiment series studies small normal transformers (with MLP layers), on a different dataset consisting only of internet books. More...
One Layer
(with MLPs)
Two Layer
(with MLPs)
Three Layer
(with MLPs)
Four Layer
(with MLPs)
Five Layer
(with MLPs)
Six Layer
(with MLPs)





Loss over training
Loss curve showing after the model is trained on tokens of training data. More...
The orange band denotes phase change interval.





Loss over training, broken down by context index
Heatmap of , the average loss of token in context after elapsed tokens of training. More...






Derivative of loss WRT context index
Heatmap of . The log vertical partial derivative of the above graph. More...






"In-context learning score"
)
How much better the model is at predicting the 500th token than the 50th? We use this as a proxy for how good the model is at in-context learning. More...





Model Details
The following specifications describe the small, full-scale, and modified-key model families.
Small models
Small models have one to six layers, twelve heads per layer, residual width 768, an 8,192-token context, and a 65,536-token vocabulary. They train for about ten billion tokens with 200 saved checkpoints. The weight-decay reduction around five billion tokens occurs after the reported small-model transition; the learning-rate warmup can overlap its early range.
Full-scale models
Full-scale models range from four to forty layers and include dense and local attention. Width and head count grow with scale, while context and vocabulary match the smaller series. Only fourteen or fifteen exponentially spaced checkpoints are available, so developmental curves are much less finely resolved.
Table of model properties for full-scale models
The retained table specifies six full-scale models from thirteen million to thirteen billion non-embedding parameters, including their layer counts, widths, head counts, and head dimensions.
Non-embedding parameter counts | Activation dimension | Attention heads per layer | Attention dimension | |
4 | 13M | 512 | 8 | 64 |
6 | 42M | 768 | 12 | 64 |
10 | 200M | 1280 | 20 | 64 |
16 | 810M | 2048 | 32 | 64 |
24 | 2.7B | 3072 | 48 | 64 |
40 | 13B | 5120 | 40 | 128 |
Smeared key models
Smeared-key models interpolate each head’s key between the current and preceding token using a learned scalar gate. The first token is unchanged. Other proportions and training follow the small models; the reported variants have one or two layers.
Equation
Unexplained Curiosities
Several results remain unexplained, including the similar context-loss gaps across model sizes and unusual individual heads. They are observations for follow-up, not resolved consequences of the proposed mechanism.
Seemingly Constant In-Context Learning Score
After the transition, the context-loss gap is surprisingly similar across scales and training time. Changing the comparison positions broadly preserves this pattern. Larger models still predict better at every position; much of their advantage appears in the first few tokens. Equal later improvements therefore do not mean equal overall capability or identical information use.


Phase Change Effect on Loss Derivatives
Loss derivatives change their relative ordering around the transition, further suggesting a major developmental change. The source’s verbal descriptions of the direction are inconsistent, so the retained derivative plot supplies the precise comparison.

Additional Curiosities
Unresolved cases include a late-forming non-induction head that resembles the transition in ablation space, diffuse ablation effects, a loss spike, an induction head with an opposite-signed score effect, and prefix-matching heads that suppress rather than copy. These exceptions complicate a single-mechanism account.
Discussion
The discussion considers safety motivations and connections between individual circuits, learning dynamics, and aggregate scaling trends.
Safety Implications
The safety motivation is to understand abrupt capability changes and inference-time adaptation. The small models studied provide an induction-based explanation for much of their measured adaptation; the study observes no evidence of an internal optimizer, but does not exclude such mechanisms in other settings.
Linking Learning Dynamics, Scaling Laws, and Mechanistic Interpretability
The transition connects an identifiable circuit, changing training behavior, and an aggregate loss phenomenon. This makes it a concrete setting for studying how mechanistic explanations relate to learning dynamics and scaling trends.
Scaling Laws
Scaling laws describe aggregate relationships, while circuit analysis seeks their internal causes. The unusual behavior of standard one-layer transformers may partly reflect their inability to express the same induction circuit, but the proposed connection does not derive a general scaling law.

Phase Changes & Discontinuous Model Behavior
Related phenomena include abrupt task gains, grokking, and double-descent trend reversals. This study contributes a circuit-level interpretation of one training transition, rather than equating all these phenomena.
Learning Dynamics
Work on solvable linear-network learning dynamics and loss-surface geometry suggests ways to connect evolving circuits to optimization. The induction transition provides a test case for those connections, not a complete theory of neural-network training.
Universality
Recurring features or circuits across models make shared interpretability research possible. Induction adds evidence at the levels of attention pattern, weight-level mechanism, and computed information, while related representation-similarity results address broader forms of convergence.
Attention Patterns in Translation-like Tasks
Translation and speech-recognition systems can attend to the source item whose continuation is about to be generated. These earlier patterns resemble aspects of induction, although the encoder–decoder visualization and task setup differ.
Comments & Replications
Two external researchers report replications and related observations. Their results are retained with separate attribution.
Replication
Adam Scherlis reports reliable induction circuits in two-layer attention-only models. Replacing a previous-token head’s scores with the ideal pattern recovers 99% of its ablation loss difference. A simple induction pattern recovers about 65%, and adding a longer-prefix pattern recovers another ten percentage points. These results support the circuit while exposing behavior beyond the minimal one-token match.
Replication
Tom Lieberum identifies candidate induction heads in publicly available GPT-2 and GPT-Neo models. He also reports that a synthetic task can learn induction with context length four, suggesting that usefulness and training distribution affect the required context length. This is an external extension, not a threshold established on natural text.
Change Log
The source records September 2022 corrections to prefix-matching wording and the copying evaluator, plus March 2023 typo fixes. Because some directional descriptions remain confusing, the edited text uses the explicit A B … A → B pattern throughout.
Acknowledgments
The authors thank external reviewers and replicators, along with colleagues supporting the research and infrastructure.
Author contributions
Catherine Olsson led experiments and analysis with mentorship from Chris Olah. Collaborators developed the smeared-key models, activation evaluators, visualization, model-inspection tools, distributed training, and writing.
Citation Information
The source citation is Catherine Olsson and colleagues, In-context Learning and Induction Heads, Transformer Circuits Thread, 2022. The original citation listing is retained.
Olsson, et al., "In-context Learning and Induction Heads", Transformer Circuits Thread, 2022.
@article{olsson2022context,
title={In-context Learning and Induction Heads},
author={Olsson, Catherine and Elhage, Nelson and Nanda, Neel and Joseph, Nicholas and DasSarma, Nova and Henighan, Tom and Mann, Ben and Askell, Amanda and Bai, Yuntao and Chen, Anna and Conerly, Tom and Drain, Dawn and Ganguli, Deep and Hatfield-Dodds, Zac and Hernandez, Danny and Johnston, Scott and Jones, Andy and Kernion, Jackson and Lovitt, Liane and Ndousse, Kamal and Amodei, Dario and Brown, Tom and Clark, Jack and Kaplan, Jared and McCandlish, Sam and Olah, Chris},
year={2022},
journal={Transformer Circuits Thread},
note={https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html}
}
Where induction heads form
The appendix locates induction behavior by layer and compares empirical scores with weight-based diagnostics.
Small attention-only models
In the small attention-only models, the strongest induction heads appear in the final layer.

Small models with MLPs:
In five- and six-layer models with MLPs, induction heads appear more often in the second-to-last layer.

Full-scale models:
Induction appears relatively earlier in the larger models. In the twenty-four- and forty-layer models, most identified heads occur before the depth midpoint. Plots emphasize a sample of one hundred heads biased toward high prefix-matching scores; translucent points show the rest.

Distribution of scores
Copying and prefix-matching scores are positively associated, especially among strong prefix matchers. Small-model distributions separate more sharply than large-model distributions. The plotting subsample should not be mistaken for an unbiased census.

Validating head activation evaluators
The following comparisons test whether behavioral evaluators track the expected weight-matrix signatures in small attention-only models.
Copying
The copying score correlates with a signed eigenvalue summary of the output–value matrix. This supports the behavioral proxy in the tested models without turning eigenvalue positivity into a universal copying guarantee.

Prefix matching
The prefix-matching score correlates with the trace of the previous-token key-composition term in the query–key circuit, linking the empirical behavior to the proposed mechanism.

Data Collection
At each checkpoint, the study collects unaltered per-token losses, small-model head-ablation losses, behavioral head scores, and a weight-based prefix-matching statistic. The full-scale series omits head ablations because of their cost.
(1) Per-token losses (unaltered models)
Unaltered models process the same 10,000 examples of length 512. One fixed random token per example yields a 10,000-dimensional behavioral vector; averaging across examples by position yields a 512-dimensional context-loss curve.
(2) Per-token losses (attention head ablations)
A full ablation would remove a head’s influence on all downstream computations. The reported pattern-preserving procedure instead runs once to save attention patterns, then zeroes one head’s result while forcing those saved patterns on the second run. It preserves routing effects and removes transmitted-output paths. Nonlinear MLPs still complicate causal interpretation.
(3) Head activation evaluators
The copying evaluator tests direct logit changes on twenty-five random tokens after mean subtraction and ReLU. Prefix matching uses a twenty-five-token sequence repeated four times with a start token, and previous-token attention is measured on natural data. Scores average ten examples. The copying test accidentally omitted the start token, a documented source of anomalously low scores for some heads.
(4) Trace of QK eigenvalues (for previous-token QK-circuit term)
The weight-based statistic combines pure key-composition paths from earlier heads, weighting each by how strongly it attends to the previous token, then takes the trace of the resulting matching matrix. It is defined only for heads beyond the first layer and covers this restricted family of paths rather than every possible induction mechanism.
Analyses
The analyses turn checkpoint losses and head measurements into training curves, behavior-space trajectories, and ablation comparisons.
Per-token losses
Loss views include checkpoint-average loss, loss by context position, the derivative with respect to log position, and the position-500 minus position-50 score. PCA jointly projects checkpoint loss vectors to visualize the dominant training trajectories.
Attention head measurements
Head plots track behavioral prefix scores and the weight-based matching trace. Small-model plots include every head; full-scale plots emphasize a subset. The weight-decay change creates a separate visible bend after the induction transition.