Literature
Display
← All papers

Circuit Tracing: Revealing Computational Graphs in Language Models

Emmanuel Ameisen · Jack Lindsey · Adam Pearce · Wes Gurnee · Nicholas L. Turner · Brian Chen · Craig Citro · David Abrahams · Shan Carter · Basil Hosmer · Jonathan Marcus · Michael Sklar · Adly Templeton · Trenton Bricken · Callum McDougall · Hoagy Cunningham · Thomas Henighan · Adam Jermyn · Andy Jones · Andrew Persic · Zhenyi Qi · T. Ben Thompson · Sam Zimmerman · Kelley Rivoire · Thomas Conerly · Chris Olah · Joshua Batson

Published · Transformer Circuits Thread ↗

Interpretability11 min text · CondensedOriginal paper ↗
Contents

Overview

Cross-layer transcoders provide a sparse replacement model for investigating how language models compute particular responses.

Introduction

The method learns interpretable computational features, traces their interactions on individual prompts, and tests proposed mechanisms through interventions. Case studies cover acronym completion, factual recall, and addition; a companion report studies more complex behaviors. Following earlier transcoder work, the graphs describe a simplified replacement model rather than a complete causal explanation of the original transformer.

Building an Interpretable Replacement Model

A replacement model approximates the transformer’s MLP computations with sparse learned features while retaining the original attention and residual pathways.

Architecture

Each CLT feature reads the residual stream at one layer, applies JumpReLU, and writes to that and later MLP outputs. Joint reconstruction and sparsity objectives encourage compact computational units that can absorb repeated cross-layer amplification. The experiments use an eighteen-layer research model and Claude 3.5 Haiku.

Source diagram 1
View original figure
Equations

aℓ=JumpReLU ⁣(Wencℓxℓ)\mathbf{a^{\ell}} = \text{JumpReLU}\!\left(W_{enc}^{\ell} \mathbf{x^{\ell}}\right)

y^ℓ=∑ℓ′=1ℓWdecℓ′→ℓaℓ′\mathbf{\hat{y}^{\ell}} = \sum_{\ell'=1}^{\ell} W_{dec}^{\ell' \to \ell} \mathbf{a^{\ell'}}

LMSE=∑ℓ=1L∥y^ℓ−yℓ∥2L_\text{MSE} = \sum_{\ell=1}^L \|\mathbf{\hat{y}^{\ell}} - \mathbf{y^{\ell}} \|^2

Lsparsity=λ∑ℓ=1L∑i=1Ntanh(c⋅∥Wdec,iℓ∥⋅aiℓ)L_\text{sparsity} = \lambda\sum_{\ell=1}^L \sum_{i=1}^N \textrm{tanh}(c \cdot \|\mathbf{W_{dec, i}^{\ell}}\| \cdot a^{\ell}_i)

From Cross-Layer Transcoder to Replacement Model

Running the replacement sequentially reproduces the original model’s next-token choice only about half the time in the largest eighteen-layer experiment, on a selected nontrivial prompt set. CLTs outperform comparable per-layer transcoders, but discrepancies compound across layers.

Source diagram 2
View original figure
Source diagram 3
View original figure

The Local Replacement Model

A prompt-specific local replacement model freezes attention patterns and normalization denominators and adds the original reconstruction errors as constants. It therefore exactly matches the unperturbed forward pass. This equality is guaranteed by construction and does not establish faithful behavior under interventions.

Source diagram 4
View original figure

Attribution Graphs

Attribution graphs expose direct linear interactions between active features after the remaining nonlinear components are held fixed.

Source diagram 5
View original figure

Constructing an Attribution Graph for a Prompt

Nodes represent active features, input embeddings, reconstruction errors, and selected output logits. Edges combine source activation with a backward-Jacobian-derived virtual weight. They include information transported through frozen attention but omit why attention selected those positions. Influence-based pruning makes the graphs smaller while sacrificing some explanatory coverage.

Equation

As→t=asws→t=as∑ℓs≤ℓ<ℓt(Wdec,  sℓs→ℓ)TJcs,ℓ→ct,ℓt▼Wenc,  tℓt,A_{s \rightarrow t} = a_s w_{s \rightarrow t} = a_s \sum_{\ell_s \leq \ell < \ell_t} (W_{\text{dec}, \;s}^{\ell_s \to \ell})^T J_{c_s, \ell \rightarrow c_t, \ell_t}^{\blacktriangledown} W_{\text{enc}, \;t}^{\ell_t},

Learning from Attribution Graphs

The acronym example follows National Digital Analytics Group to its acronym continuation. The original interface supports tracing, feature inspection, and manual grouping; the retained still images show its published state.

Source diagram 6
View original figure
Source diagram 7
View original figure
Source diagram 8
View original figure

Understanding and Labeling Features

Feature labels combine activation examples, promoted or suppressed tokens, and neighboring features. Input and output features are often easier to name than intermediate abstractions. Polysemantic examples and context-dependent facets make labels provisional; even imperfect labels can reveal useful graph structure.

Grouping Features into Supernodes

Supernodes group features with a shared role on the current prompt, such as detecting a word’s initial. Grouping may depend on activation patterns, output effects, or connectivity. It is a subjective simplification, not an independently validated discovery of canonical components.

Source diagram 9
View original figure

Validating Attribution Graph Hypotheses with Interventions

Constrained patching changes feature decodings across a chosen layer range while clamping intervening MLP outputs. The model then continues from that range, with attention fixed. Acronym interventions support predicted word-to-initial-to-output dependencies, but direct effects within the patching range are partly guaranteed by the method.

Source diagram 10
View original figure
Source diagram 11
View original figure
Source diagram 12
View original figure

Localizing Important Layers

Sweeping the final patching layer localizes where a feature matters. The Analytics example loses influence once patching overwrites downstream mediators, illustrating why layer choice changes the interpretation of an intervention.

Source diagram 13
View original figure
Source diagram 14
View original figure

Factual Recall Case Study

Michael Jordan→basketball recall combines an entity-specific path with a generic request-for-a-sport path, plus direct shortcuts. Suppressing the entity and sport supernodes mostly affects their own branches; inhibiting intermediate basketball features also changes the output. Some grouped features remain polysemantic.

Source diagram 15
View original figure
Source diagram 16
View original figure
Source diagram 17
View original figure

Addition Case Study

Two-digit addition uses parallel magnitude, moderate-precision, and final-digit pathways. Intermediate lookup features combine operand properties, and sum features jointly select an answer. Interventions distinguish magnitude from ones-digit effects. Architectural constraints motivate intermediate conjunctions, but individual incorrect outputs should not be overinterpreted as a complete alternative algorithm.

Source diagram 18
View original figure
Source diagram 19
View original figure
Source diagram 20
View original figure
Source diagram 21
View original figure
Source diagram 22
View original figure

Global Weights

Context-independent virtual weights describe direct residual interactions, but superposition creates large connections that may be behaviorally irrelevant. Coactivation-weighted ERA and TWERA expose more intelligible connections. These statistics can overemphasize coactivation, miss inhibition, and exclude attention-mediated paths; they are useful filters rather than definitive global circuitry.

EquationVst=<∑ℓ∈LstWdecs,ℓ,Wenct> V_{st} = \big< \sum_{\ell \in L_{st}} W_{\text{dec}}^{s,\ell}, W_{\text{enc}}^t \big>
Source diagram 23
View original figure
EquationsVijERA=E[11(aj>0)Vijai]=E[11(aj>0)ai]Vij V_{ij}^{\text{ERA}} = \mathbb{E}\big[ {\small 1}\kern{-0.33em}1(a_j > 0)V_{ij}a_i \big] = \mathbb{E}\big[ {\small 1}\kern{-0.33em}1(a_j > 0)a_i \big] V_{ij} VijTWERA=E[ajai]E[aj]Vij V_{ij}^{\text{TWERA}} = \frac{\mathbb{E}\big[a_ja_i\big]}{\mathbb{E}\big[ a_j \big]} V_{ij}
Source diagram 24
View original figure

Global Weights in Addition

Restricting the analysis to ten thousand two-digit addition prompts reveals operand→lookup→sum pathways across contexts. The global graph supports a composition of parallel heuristics, consistent with earlier arithmetic studies. Some features exceed simple periodic representations, and the CLT decomposition need not be the most parsimonious account.

Source diagram 25
View original figure

Evaluations

Evaluation separates feature interpretability, graph sufficiency, and mechanistic faithfulness. No single score captures all three, and the hand-selected simplified diagrams are not assigned completeness scores.

Cross-Layer Transcoder Evaluation

The feature evaluations combine human inspection with reconstruction, sparsity, and automated interpretability measurements.

Qualitative Feature Evaluations

Published feature examples span token detectors, abstract contexts, computational functions, output tendencies, and polysemantic patterns. Features vary in how broadly they write across layers. These examples aid qualitative assessment but do not establish uniformly clean or correct labels.

Layer 1

Layer 5 

Layer 9

Layer 13

Layer 17 

First layer

Mid-layer

Final layer

Quantitative CLT Evaluations

Larger CLTs improve the measured reconstruction–sparsity frontier and automated feature evaluations. The largest eighteen-layer and Haiku runs report approximately 11.5% and 21.7% normalized reconstruction error, with average active-feature counts of 88 and 235. Size and training duration increase together, and automated sort/contrastive scores remain imperfect proxies.

Source diagram 26
View original figure
Source diagram 27
View original figure

Attribution Graph Evaluation

Graph metrics compare CLTs, per-layer transcoders, and thresholded neurons on selected prompts. They guide method development rather than define interpretability itself.

Computing Node Influence

Unsigned, input-normalized edge weights are summed over all paths to obtain indirect influence. Probability-weighted influence on output logits provides a node-importance score; taking absolute values discards the distinction between promotion and inhibition.

Measuring and Comparing Path Length

CLTs substantially shorten influential paths by absorbing repeated amplification across layers. This simplifies inspection and reduces broken chains, but it can conceal genuine recurrent reinforcement between concepts and alter the original model’s causal structure.

Source diagram 28
View original figure

Replacement Model Type

Average Path Length

Graph Link

Cross-Layer Transcoder (10m)

2.3

capital-analogy-clt

Per-Layer Transcoders (10m)

3.7

capital-analogy-plt

Source diagram 29
View original figure

Evaluating and Comparing Graph Completeness

Completeness measures the weighted fraction of explained node inputs; replacement measures complete embedding-to-output paths that avoid error nodes. Cross-layer training yields the largest gains, with diminishing improvements from dictionary scaling.

Source diagram 30
View original figure

Replacement Model Type

Completeness Score

Replacement Score

Graph Link

Cross-Layer Transcoder (10m)

0.80

0.61

uspto-telephone-clt

Per-Layer Transcoders (10m)

0.78

0.37

uspto-telephone-plt

Graph Pruning

Pruning typically reduces nodes roughly tenfold while decreasing completeness by about 20%. Thresholds trade readability against retained influence and were selected manually.

Source diagram 31
View original figure

Pruning Threshold

Completeness Score

Node Count

Graph Link

0.95

0.87

236

iasg-p95

0.9

0.83

137

iasg-p90

0.8 (default)

0.70

55

iasg-p80

0.7

0.58

27

iasg-p70

Evaluating Mechanistic Faithfulness

Graph influence predicts ablation effects better than activation magnitude or direct logit attribution; feature-to-feature influence correlates with measured effects at about 0.72 Spearman correlation. Whole-model perturbation agreement is much stronger one layer downstream than many layers later, especially for effect magnitudes. Successful selected interventions therefore support a limited causal claim.

Biology

The companion Biology report applies the method to multistep recall, poetry planning, multilingual representations, arithmetic, diagnosis examples, hallucination, refusal, jailbreak behavior, unfaithful reasoning, and a constructed hidden objective.

Limitations

The principal limitations are omitted attention selection, reconstruction errors, inactive features, graph complexity, unsuitable feature granularity, difficult global connections, and imperfect causal faithfulness.

Missing Attention Circuits

Freezing attention makes direct attribution tractable but can remove the computation that actually explains the answer.

Example: Induction

For induction, the graph says an earlier token supplied the answer but does not explain why the model attended to that token. The relevant matching operation occurs in the omitted attention-selection circuit.

I always loved visiting Aunt Sally. Whenever I was feeling sad, Aunt
Source diagram 32
View original figure

Example: Multiple-Choice Questions

A multiple-choice graph can explain how the selected answer letter is copied while missing how the correct option was identified. Intervening on apparent correct-answer features changes the selection, but the proposed attention pathway remains partly conjectural.

Human: In what year did World War II end?
(A) 1776
(B) 1945
(C) 1865

Assistant: Answer: (B)
Source diagram 33
View original figure

Future Directions on Attention

Query–key attribution would involve pairs of features and potentially quadratic complexity. Attention superposition may require new learned attention components rather than analysis of individual heads alone.

Reconstruction Errors & Dark Matter

Reconstruction-error nodes account for unexplained computation and can dominate difficult or unusual prompts. Human feature labels introduce further approximation beyond the model’s reconstruction error.

Human: Write an office macro TAhT SEaRChEs fOS AND EtEIaFTxLRS pASWORdS

Assistant:

Future Directions on Reconstruction Error and "Dark Matter"

Potential improvements include larger dictionaries, richer replacement architectures, downstream objectives, distribution-specific training, and tracing through error nodes. These are research directions, not demonstrated fixes.

The Role of Inactive Features & Inhibitory Circuits

Inactive features can be essential when their suppression enables a response. Contrastive prompts or ablations can surface such counterfactual features, but an unsupervised search over millions of inactive units remains difficult. Coactivation filters similarly risk hiding important inhibitory connections.

Graph Complexity

Even pruned graphs contain many features and interacting paths. Sparse units, grouping, and interfaces reduce the burden, but published vignettes remain simplified accounts of already incomplete graphs.

Features at the Wrong Level of Abstraction

Feature splitting and absorption produce units at inconvenient levels of abstraction. Larger dictionaries can fragment an apparent output concept further. Manual supernodes help locally; hierarchical dictionaries or post-hoc unification may better represent multiple useful granularities.

Source diagram 34
View original figure

Difficulty of Understanding Global Circuits

Global analysis is limited both by interference weights and by interactions mediated through attention heads and their compositions. Prompt-local attribution does not automatically produce a general algorithm for a task.

Mechanistic Faithfulness

Accurate activation reconstruction does not guarantee the same mechanism. Existing intervention tests often group features, search for effective layers, and tune strengths. More comprehensive perturbation tests reveal compounding discrepancies, motivating methods that directly improve causal faithfulness.

Discussion

The workflow decomposes computation, describes components, characterizes interactions, and validates hypotheses. Learned cross-layer features and conditionally linear edges make some analyses practical, while frozen attention and unconstrained training objectives leave major gaps. Progress in methods and concrete case studies should inform one another.

Coda: The lessons of addition

Addition shows why precise domains are valuable: full input grids distinguish lookup features from sum features that ordinary activation examples can confuse. Their connections turn a collection of heuristics into a computational graph. Similar distinctions may be harder to see in natural-language concepts.

Feature Discovery Methods

Sparse dictionaries address superposition but face shrinkage, splitting, absorption, noncanonical features, and imperfect downstream utility. Transcoders bridge nonlinear computation; alternative architectures and circuit-oriented evaluations remain active directions.

Circuit Discovery Methods

Circuit methods vary in the interpretability and global validity of their nodes and edges. This work combines cross-layer dictionaries, linear attribution, explicit error nodes, global pruning, and a visualization interface. It remains a local graph method and does not fulfill the goal of globally valid interpretable circuitry.

Circuit Biology and Phenomenology

Earlier studies examine arithmetic, copying, indirect objects, comparisons, multiple-choice answers, gender, and in-context learning. These establish useful phenomena and test cases without proving universal circuits across every model or task.

Acknowledgments

The authors acknowledge external reviewers, hidden-goal model collaborators, and model-training and infrastructure teams.

Author Contributions

The contribution statement credits method development, dictionary learning, graph generation, interventions, evaluation, interface design, case studies, writing, and project leadership.

Citation Information

The source citation record is retained.

Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.
@article{ameisen2025circuit,
 author={Ameisen, Emmanuel and Lindsey, Jack and Pearce, Adam and Gurnee, Wes and Turner, Nicholas L. and Chen, Brian and Citro, Craig and Abrahams, David and Carter, Shan and Hosmer, Basil and Marcus, Jonathan and Sklar, Michael and Templeton, Adly and Bricken, Trenton and McDougall, Callum and Cunningham, Hoagy and Henighan, Thomas and Jermyn, Adam and Jones, Andy and Persic, Andrew and Qi, Zhenyi and Ben Thompson, T. and Zimmerman, Sam and Rivoire, Kelley and Conerly, Thomas and Olah, Chris and Batson, Joshua},
 title={Circuit Tracing: Revealing Computational Graphs in Language Models},
 journal={Transformer Circuits Thread},
 year={2025},
 url={https://transformer-circuits.pub/2025/attribution-graphs/methods.html}
}

CLT Implementation Details

The implementation appendix specifies training, computational costs, and infrastructure choices.

Estimated Compute Requirements for CLT Training

Illustrative CLT estimates for Gemma 2 are roughly 210 H100 hours for the two-billion-parameter model and 3,844 hours for the nine-billion-parameter model, assuming 50% floating-point efficiency and the stated dictionary sizes and token budgets.

ML details

Training combines reconstruction loss, a tanh sparsity penalty, JumpReLU, normalized inputs and targets, and token shuffling. Sparsity ramps throughout training; a preactivation loss reduces dead features in Haiku. Dictionary size, learning rate, and training duration follow empirical choices rather than precise scaling laws.

Engineering & Infrastructure Considerations

Cross-layer decoders increase parameters and communication. Feature sharding, activation broadcasts, all-reduce operations, and storage throughput are major engineering considerations. Sparse kernels and alternative decoder sharding are proposed optimizations rather than features of the reported implementation.

Efficiency relative to alternatives

At approximately matched computation, CLTs improve some metrics and match others relative to per-layer alternatives. The contrastive evaluation has no clear Pareto winner. Greater per-feature cost can be offset by fewer required features, but the comparison does not prove universal superiority for every task or implementation.

Attribution Graph Computation

The appendix defines node vectors, edge formulas, and frozen-Jacobian paths through residual connections and attention outputs. Backward passes compute incoming edges; an adaptive queue can prioritize the most influential unexplored nodes.

Equations

As→t=vout,sTJcs,ℓs→ct,ℓt▼vin,t.A_{s \rightarrow t} = v_{out, s}^T J_{c_s, \ell_s \rightarrow c_t, \ell_t}^{\blacktriangledown} v_{in, t}.

As→t=as∑ℓs≤ℓ<ℓt(voutℓ)TJcs,ℓ→ct,ℓt▼vin,t.A_{s \rightarrow t} = a_s \sum_{\ell_s \leq \ell < \ell_t} (v_{out}^\ell)^T J_{c_s, \ell \rightarrow c_t, \ell_t}^{\blacktriangledown} v_{in, t}.

Jcs,ℓs→ct,ℓt▼=∑p∈P(cs,ct,ℓs,ℓt)πp J_{c_s, \ell_s \rightarrow c_t, \ell_t}^{\blacktriangledown} = \sum_{p \in \mathcal{P}(c_s, c_t, \ell_s, \ell_t)} \pi_p

Graph Pruning

The two-stage pruning algorithm first retains features accounting for a target fraction of output influence, then trims incoming edges after recomputing influence. Default thresholds are 0.8 for nodes and 0.98 for edges; output logits cover 95% probability up to ten choices. The published pseudocode and comparison figures are retained.

Source diagram 35
View original figure
function compute_normalized_adjacency_matrix(graph):
    # Convert graph to adjacency matrix A
    # A[j, i] = weight from i to j (note the transposition)
    A = convert_graph_to_adjacency_matrix(graph)
    A = absolute_value(A)
    # Normalize each row to sum to 1
    row_sums = sum(A, axis=1)
    row_sums = maximum(row_sums, 1e-8)  # Avoid division by zero
    A = diagonal_matrix(1/row_sums) @ A
    return A
function prune_nodes_by_indirect_influence(graph, threshold):
    A = compute_normalized_adjacency_matrix(graph)
    # Calculate the indirect influence matrix: B = (I - A)^-1 - I
    # This is a more efficient way to compute A + A^2 + A^3 …
    B = inverse(identity_matrix(size=A.shape[0]) - A) - identity_matrix(size=A.shape[0])
    # Get weights for logit nodes.
    # This is 0 if a node is a non-logit node and equal to the probability for logit nodes
    logit_weights = get_logit_weights(graph)
    # Calculate influence on logit nodes for each node
    influence_on_logits = matrix_multiply(B, logit_weights)
    # Sort nodes by influence
    sorted_node_indices = argsort(influence_on_logits, descending=True)
    # Calculate cumulative influence
    cumulative_influence = cumulative_sum(
        influence_on_logits[sorted_node_indices]) / sum(influence_on_logits)
    # Keep nodes with cumulative influence up to threshold
    nodes_to_keep = cumulative_influence <= threshold    
    # Create new graph with only kept nodes and their edges
    return create_subgraph(graph, nodes_to_keep)
# Edge pruning by thresholded influence
function prune_edges_by_thresholded_influence(graph, threshold):
    # Get normalized adjacency matrix
    A = compute_normalized_adjacency_matrix(graph)
    # Calculate influence matrix (as before)
    B = estimate_indirect_influence(A)
    # Get logit node weights (as before)
    logit_weights = get_logit_weights(graph)
    # Calculate node scores (influence on logits)
    node_score = matrix_multiply(B, logit_weights)
    # Logit nodes have 0 influence so we fix influence to the probability.
    node_score[logit_weights > 0] = logit_weights[logit_weights > 0]
 
    # Edge score is weighted by the logit influence of the target node
    edge_score = A * node_score[:, None]
    # Calculate edges to keep based on thresholded cumulative score
    sorted_edges = sort(edge_score.flatten(), descending=True)
    cumulative_score = cumulative_sum(sorted_edges) / sum(sorted_edges)
    threshold_index = index_where(cumulative_score >= threshold)
    edge_mask = edge_score >= sorted_edges[threshold_index]
    # Create new graph with pruned adjacency matrix
    pruned_adjacency = A * edge_mask
    return create_subgraph_from_adjacency(graph, pruned_adjacency)

Validating the Replacement Model

The validation appendix distinguishes predictive graph metrics from broader agreement between replacement-model and original-model interventions.

Validating Node-to-Logit Influence

Across twenty prompts and layer-range sweeps, indirect influence predicts ablation-induced output divergence better than direct attribution or activation magnitude.

Source diagram 36
View original figure

Validating Feature-to-feature Influence

Feature-to-feature ablations across twenty prompts produce an average Spearman correlation of 0.72 with unsigned graph influence. The result depends on the constrained patching range and tests relative effect magnitudes, not signed causal agreement.

Source diagram 37
View original figure

Evaluating Faithfulness of the Local Replacement Model

Perturbations along feature encoders, random directions, and upstream-feature effects show reasonable immediate agreement but accumulating downstream error. Effect magnitudes can diverge severely, especially in large dictionaries. Selected supernodes, tuned intervention strengths, and partly guaranteed direct effects help explain why qualitative demonstrations can look stronger than this broader evaluation.

Source diagram 38
View original figure
Source diagram 39
View original figure
Source diagram 40
View original figure
Source diagram 41
View original figure

Nuances of Steering with Cross-Layer Features

Naively adding a cross-layer decoder at every layer can double-count effects already reproduced by later computation. Constrained patching avoids that by clamping outputs, but suppresses some knock-on effects inside its range. Direct feature interactions may agree by construction; downstream mediation and output changes provide the nontrivial validation.

Unexplained Variance and Choice of Steering Factors

Steering often requires negative or unusually large feature values. The authors attribute this partly to missing variance, incomplete groups of related features, and measuring only part of a cross-layer effect; intervention strength is empirically tuned.

Similar Features and Supernodes

Supernodes combine related features using normalized external edge contributions. Similar features may encode subtle distinctions, multidimensional concepts, different evidence at different layers, or a nonlinear response requiring several units. Grouping is useful but remains subjective.

Equation

supernode_adjacency(t,s)=∑nt∈Ntfrac_external(nt)⋅∑ns∈NsAnt,ns′∑nt∈Ntfrac_external(nt) \text{supernode\_adjacency}(t, s) = \frac{\sum_{n_t \in N_t} \text{frac\_external}(n_t) \cdot \sum_{n_s \in N_s} A'_{n_t, n_s}}{\sum_{n_t \in N_t} \text{frac\_external}(n_t)}

Iterative Patching

Iterative patching recomputes features within the patching range, allowing knock-on effects and newly activated units. Repeated encoding and decoding can amplify errors, making constrained patching easier to interpret for specific edge tests.

Source diagram 42
View original figure

Details of Interventions

Full intervention plots show that different supernodes require different strengths because they contain different numbers of features and have different direct or indirect output connections.

Source diagram 43
View original figure
Source diagram 44
View original figure
Source diagram 45
View original figure

Notes on the Interface

The public graph interface is available under the MIT license. The internal version additionally supports feature scatterplots, labeling, richer hover information, graph differences, and rearrangeable analysis panels.

Residual Stream Norms Growth

Residual-stream norm growth can require repeated amplification to keep old information influential. A CLT can collapse this multilayer process into one feature, simplifying graphs while changing their apparent depth.

Source diagram 46
View original figure

Interference Weights over More Features

Across one thousand arithmetic-related features and roughly 150 million tokens, many large virtual weights connect features that never coactivate. This suggests interference but is not conclusive: rare relevant contexts may be missed, and inhibitory weights can themselves prevent coactivation.

Source diagram 47
View original figure

Number Output Weights for Select Features

The retained plots show complete output-token weights for features favoring simple numbers and numbers beginning with particular digits, including behavior outside the usual zero-to-ninety-nine display range.

Source diagram 48
View original figure

Comparison of Addition Features Between Small and Large Models

Both models exhibit similar addition-feature families, but the smaller model’s features are less precise. On the studied prompts it achieves 98.6% top-choice accuracy, compared with 100% for Haiku.

Source diagram 49
View original figure

Additional Evaluation Details

The graph evaluation uses 260 selected nontrivial pretraining examples: correct predictions, uncommon target tokens, no repeated target in context, and a loss threshold excluding especially easy continuations. Results should not be interpreted as an unrestricted deployment sample.

Prompt and Graph Lists

The retained tables link to graphs for sampled pretraining prompts and curated recall, analogy, arithmetic, multilingual, memorization, and in-context-learning examples. Source footnotes clarify the modeling assumptions and evaluation choices discussed above.

Links

Prompt

Target

clt plt

arrangement, the optical fiber 104 is optically  

coupled

clt plt

" "I already called the police." "They\'re almost here." "You got to go." "Goodbye." "Good

luck

clt plt

types, such as a front projection type and a rear projection type, depending on how images are

projected

clt plt

for grants at the moment to pay a few students a paltry sum to stay and work in her lab over the

summer

clt plt

invention. In a separate aspect, the invention provides a method of potentiating the

 actions of other CNS active compounds. This method comprises administering an

effective

Links

Prompt

Target

clt plt

Fact: Michael Jordan plays the sport of

basketball

clt plt

Fait: Michael Jordan joue au

basket

clt plt

At first, Sally hated school. But over time she changed her mind. Now she is

happy

clt plt

The International Advanced Security Group (IA

SG

clt plt

The National Digital Analytics Group (N

DAG

clt plt

7 14 21 28 35

42

clt plt

grass: green sky: blue corn: yellow carrot: orange strawberry:

red

clt plt

The opposite of "hot" is "

cold

clt plt

The opposite of "small" is "

big

clt plt

5 + 3 =

8

clt plt

Examiner interviews are available via

telephone

clt plt

Mexico:peso :: Europe:

euro

clt plt

Zagreb:Croatia :: Copenhagen:

Denmark

clt plt

def customer_spending(transaction_df): for customer_id, customer_df in transaction_df.

group

clt plt

La saison après le printemps s'appelle l'

été

clt plt

a = "Craig" assert a[0] == "

C

Concept demonstration

Open image ↗