Literature
Display
← All papers

Tracing Attention Computation Through Feature Interactions

Harish Kamath · Emmanuel Ameisen · Isaac Kauvar · Rodrigo Luger · Wes Gurnee · Adam Pearce · Sam Zimmerman · Joshua Batson · Thomas Conerly · Chris Olah · Jack Lindsey

Published · Transformer Circuits Thread ↗

Interpretability4 min text · CondensedOriginal paper ↗
Contents

Overview

Earlier attribution graphs explained information flow while freezing attention patterns. This report adds QK attributions and head loadings to explain both why a head attends to a token and how it contributes to an output. Case studies in Claude 3.5 Haiku recover induction, antonym selection, multiple-choice answering, and statement-validation mechanisms.

The problem: transcoder-based attribution graphs omit attentional computations

Freezing attention makes feature interactions linear but discards the computation selecting attention destinations. Summing over heads also hides which heads carried a graph edge. In many tasks, these omitted decisions are the central computation, so merely tracing an output back to an earlier matching token is insufficient.

High-level strategy

Expand each pre-softmax attention score into bilinear interactions between query-side and key-side representation features. Separately decompose attribution-graph edges into the contributions of individual attention heads. Earlier work explored related QK decompositions [1] [2]; this report combines them with systematic head loadings.

QK attributions

Reconstruct residual streams from SAE features, bias, and reconstruction error, then expand their query–key dot products. The terms sum to the attention score after accounting for normalization and any position-dependent transforms. They explain scores, not the full softmax competition: relevant negative contributions at alternative positions still require manual inspection.

An illustration of how we visualize QK attributions. In a circuits graph, for any edge that crosses context positions, we can use the head loadings of that edge to index into a specific (query ctx, key ctx, layer, head) position, and then use the (un)marginalized list of features to inspect the QK circuit.

Computing attention head contributions to an attribution graph

Head loadings identify how much each head mediates a graph edge. They connect an explanation of an attention pattern to the computation that uses it.

“Checkpointing” attention paths with features

Paths through multiple attention layers grow exponentially. Checkpointing representations at each layer restricts an edge to paths through at most one attention layer. The authors favor residual-stream dictionaries, implemented with weakly causal crosscoders, over alternatives that accumulated more error. This trades linear MLP replacement for local gradient-based attributions, which can be prompt-specific.

Head loadings

For adjacent-layer features, separate residual, MLP, and attention contributions. Within the attention term, retain each head’s contribution using the source feature activation and decoder direction, the fixed attention weight, and the downstream gradient. SAE encoders infer features but are not treated as replacement-model weights.

Equation

∑h∈headsasat(vt⊤OhVhvs)⋅attentionh(ps,pt)\sum_{h \in \text{heads}} a_s a_t \left(\mathbf{v_t}^\top O_h V_h \mathbf{v_s}\right) \cdot \text{attention}_h(p_s, p_t)

Examples

The following interventions test mechanisms in selected prompts. A head’s role in one example does not imply that it performs only that role across the model’s full distribution.

Induction

In an aunt-name completion, the old graph showed that the model copied a previously seen name. The new analysis asks why it selected that name and how preceding-title information reached the name token.

I always loved visiting Aunt Sally. Whenever I was feeling sad, Aunt

Transforming “Sally” to “say Sally” on the second “Aunt” token

Induction-related heads combine a generic tendency to attend to names with a more specific match between family-title features and features tagging a name as following that title. Removing the latter key-side features suppresses copying the specific name. Several heads must be steered; compensating downstream heads may explain this redundancy [3].

How the model’s top prediction changes as we vary the scale of “name of aunt/uncle” features on the key side. As we steer negatively, the model stops performing induction, and begins to predict generic aunt names instead.

Copying “Aunt” to “this is the name of an Aunt” on the “Sally” token

Earlier heads move title information onto name tokens through interactions between query-side first-name features and key-side title features. The role of positional information in selecting the most recent relevant title remains unexamined.

The “previous token” head works as a precursor to the induction mechanism.

Multiple parallel QK interactions

Multiple independent QK interactions can operate in parallel: name identity and family-role matching contribute separately, rather than through a single rank-one relation. Generic name-attending biases add further terms.

I always loved visiting Aunt Mary Sue. Whenever I was feeling sad, Aunt Mary

Antonyms

For a French antonym completion, attention combines explicit tags identifying the requested word with broader adjective-search heuristics. Inhibiting query-side opposite features reduces the antonym prediction and increases synonyms; suppressing key-side adjective features has a weaker similar effect.

How the model’s top prediction changes as we vary the scale of “opposite” features on the query side. As we steer negatively, the model stops predicting “the opposite of petit” and begins to predict petit, as well as French synonyms of petit.

Multiple Choice

Multiple-choice heads attend to correct-answer features, reinforced when the query context requests an answer; false-statement features inhibit competing options. Moving or suppressing correct-answer features changes the selected letter. The upstream calculation of correctness remains partly hidden by reconstruction-error nodes [4].

Human: In what year did World War II end?
(A) 1776
(B) 1945
(C) 1865

Assistant: Answer: (
A visual of how the model uses “correct/incorrect answer” features to determine which multiple choice answer to predict.
How the model’s answer to a multiple choice question changes as we steer “correct answer” features on the key side of different answers, where we inject the “correct answer” feature into the incorrect answers. From left to right: (a) steer on the B answer tokens negatively, (b) steer on the B answer tokens negatively, and on the A answer tokens positively,  (c) steer on the B answer tokens negatively, and on the C answer tokens positively. Note that “X answer tokens” means “tokens spanning from the content of the answer, to the beginning of the content of the next answer” – see the previous figure for a visual.

Correctness circuits

Concordance and discordance heads distinguish compatible from incompatible properties, such as a banana’s stated color, without clear evidence of first computing and comparing an explicit answer. Targeted QK steering flips both attention and the model’s yes/no response. Related heads check an arithmetic consistency property. These cases suggest distinct validation heuristics, not a universal correctness circuit [5].

Human: Answer with yes or no. The color of a banana is yellow?

Assistant:
How the model’s top prediction changes as we intervene on inputs to attention, steering negatively on “yellow” and injecting “red” on the query side. This causes the concordance head to stop attending, the discordance head to start attending, and the model output to flip.
Human: Answer with yes or no. 18+5=23?

Assistant:

Related work

Prior research includes bilinear sparse QK decompositions, indirect-object identification circuits, attention-output SAEs, and targeted patching studies of binding, multiple choice, and arithmetic validation [6] [7] [8]. The present contribution systematically joins QK explanations to the heads carrying graph edges.

Future work

Open problems include simplifying approximately low-rank QK explanations, automatically selecting informative softmax counterfactuals, scaling beyond quadratic context costs, and determining whether computational units span heads. Broader case studies are needed to establish generality.

Acknowledgments

Julius Tarng contributed figure assistance, and Adam Jermyn contributed conceptual discussions about attention.

Citation Information

The original article supplies its academic citation and detailed notes. Key qualifications concern position-dependent attention, nonlinear MLP attributions, compensating heads, and the distinction between validating a proposed answer and computing one.

Kamath, Ameisen, et al., "Tracing Attention Computation", Transformer Circuits, 2025.
@article{kamath2025tracing, author={Kamath, Harish and Ameisen, Emmanuel and Kauvar, Isaac and Luger, Rodrigo and Gurnee, Wes and Pearce, Adam and Zimmerman, Sam and Batson, Joshua and Conerly, Thomas and Olah, Chris and Lindsey, Jack}, title={Tracing Attention Computation Through Feature Interactions}, journal={Transformer Circuits Thread}, year={2025}, url={https://transformer-circuits.pub/2025/attention-qk/index.html} }
Concept demonstration

Open image ↗