At a glance
Influence functions estimate how slightly increasing a training sequence’s weight would change a model’s output. EK-FAC and compressed query batching make this analysis practical for research language models up to 52 billion parameters. Larger models’ influential sequences are more thematically related to their outputs; middle layers often carry the more abstract patterns. The estimates also reveal strong sensitivity to phrase order. These are local, approximate training-data attributions, not a reconstruction of the complete training process or the model’s reasoning.
1 Introduction
The study connects observed behavior to training data as a complement to circuit-level interpretability. It scales inverse-Hessian-vector products (IHVPs) and amortizes the remaining training-gradient cost across queries. Examples range from factual completions and code to multilingual transfer and role-playing.
The analysis concerns pretrained research models, not the fine-tuned assistant that originally supplied some responses. It estimates a proximal counterfactual near final weights, uses only multilayer-perceptron parameters, and searches a subset of training data. Consequently, an influential passage supplies evidence about generalization without uniquely explaining an output or establishing what a model knows.


2 Background
The background separates the statistical quantity being estimated from the numerical approximations used to compute it.
2.1 Influence Functions
Classical influence functions differentiate an empirical-risk optimum with respect to one example’s training weight. Multiplying by an output gradient estimates the corresponding change in a chosen measurement. This first-order approximation treats adding and removing an example symmetrically, and assumes regularity and a well-defined optimum. Those assumptions become problematic for nonconvex, incompletely trained neural networks. [Influence-function background]
Equations 1–6






2.1.1 Proximal Bregman Response Function
The proximal Bregman response function (PBRF) replaces literal retraining with a nearby optimization that changes the selected example’s loss while penalizing movement in both output and parameter space. Its local derivative uses a damped Gauss–Newton Hessian (GNH), which is positive definite with positive damping. This gives influence estimates a meaningful local target despite nonconvergence and nonunique neural-network optima; it does not turn them into accurate leave-one-out retraining predictions. [PBRF formulation]

Equations 7–9



2.2 Inverse-Hessian-Vector Products
For many candidate training examples and relatively few queries, compute one IHVP per query and then take its dot product with each training gradient. This shares the expensive inverse-curvature operation across candidates.
2.2.1 Iterative Methods
LiSSA estimates the inverse-matrix product through stochastic Hessian-vector iterations. Suitable scaling is required for convergence, and thousands of backward-pass-scale operations can be needed for each query. This motivates a reusable curvature approximation.
Equation 10

2.2.2 Kronecker-Factored Approximate Curvature
K-FAC approximates curvature by independent layer blocks, then factors each block into activation and pseudo-gradient covariances. Pseudo-labels are sampled from the model’s output distribution; substituting observed labels would produce the different empirical Fisher matrix. The Kronecker structure permits matrix-sized inversions instead of a full parameter-space inverse.
Equations 11–17







2.2.3 Eigenvalue-Corrected Kronecker-Factored Approximate Curvature
EK-FAC retains the K-FAC eigenvectors but fits an individual variance for every direction in that basis. Adding damping shifts these eigenvalues, making later IHVPs cheap matrix operations. The costly factor fitting and eigendecompositions are reused across queries. This improves the curvature approximation while retaining layerwise independence. [EK-FAC]
Equations 18–21




2.3 Transformer Language Models
The setting is decoder-only autoregressive transformers with softmax outputs. Summed token cross-entropy, rather than a token average, gives the matching-loss relationship between the conditional Fisher matrix and GNH. Actual training tokens remain the inputs while pseudo-labels are sampled for curvature estimation.
Equations 22–23


3 Methods
Each query contains a prompt and a specified completion. The measurement is the completion’s conditional log-probability, and the search prioritizes training sequences with positive estimated influence on it. Fit curvature once, precondition query gradients, then score candidate training gradients. The remaining steps reduce memory, search cost, and attribution granularity.
Equations 24–25


3.1 EK-FAC for Transformer Language Models
Only MLP weights are varied; attention, embeddings, and normalization parameters are held fixed. For token-shared weights, initial covariance factors treat tokens independently, but eigenvalue correction uses the exact sum of token pseudo-gradients. This captures diagonal second moments in the chosen basis without fully modeling cross-token or cross-layer dependence.
Storing covariance eigenvectors and correction values can be expensive, particularly for rectangular layers. The 52-billion-parameter analysis additionally splits each factor into two blocks; Appendix A reports the resulting memory–accuracy tradeoff.
Equations 26–27


3.2 Confronting the Training Gradient Bottleneck
After accelerating IHVPs, computing candidate training gradients becomes the dominant cost. The paper compares a cheap lexical prefilter with a larger unfiltered scan that shares computation across queries.
3.2.1 TF-IDF Filtering
A modified BM25-style TF-IDF score selects 10,000 candidate sequences per query. This makes searches manageable, but favors lexical overlap and can exclude the most interesting abstract or cross-lingual relationships. Differences between filtered searches may reflect candidate selection rather than the model’s generalization.
Equations 28–29


3.2.2 Query Batching
Candidate training gradients do not depend on the query. Compressing each preconditioned query-gradient matrix to rank 32 lets hundreds of queries share a scan. Influence scores remain highly correlated with full-rank estimates in the reported 52-billion-parameter tests. This is an empirical compression result, not an exact rank guarantee after preconditioning.

3.3 Attribution to Layers and Tokens
The block-diagonal approximation decomposes estimated influence across layers; the training loss further decomposes it across prediction positions. These heatmaps locate contributing weight updates, but a highlighted position is not necessarily the causally important input word: its activations contain earlier context and its gradients depend on later predictions. Alternative input/output-token interventions appear in Appendix B. The main heatmaps highlight the position preceding the token being predicted.


Equations 30–31


4 Related Work
Training-data attribution includes retraining-based estimates, representation similarity, gradient methods, and influence functions. The study combines curvature-aware attribution with scalable search rather than treating nearest lexical or embedding neighbors as equivalent evidence.
4.1 Training Data Attribution and Influence Functions
Retraining directly measures dataset changes but is prohibitively costly at this scale. Earlier approximations support debugging, dataset analysis, fairness, and memorization studies. Their estimands differ, so agreement with one attribution method does not establish a complete causal account of training.
4.2 Improving Scalability
Prior methods use iterative curvature approximations, candidate filtering, forward-pass proxies, or only final-layer gradients. This paper’s layerwise results indicate that final-layer restrictions can miss more abstract patterns concentrated elsewhere.
4.3 Kronecker-Factored Curvature
K-FAC originated as a natural-gradient optimizer and has extensions to convolutional, recurrent, distributed, and transformer settings. EK-FAC is reused here as an efficient curvature model for attribution, with transformer-specific handling of token-shared weights.
5 Experiments
The four pretrained research transformers have about 810 million, 6.4 billion, 22 billion, and 52 billion parameters. Candidates are 512-token pretraining sequences. Layerwise damping is 0.1 times the mean EK-FAC diagonal. Some queries come from an assistant larger than every analyzed model, so findings concern their pretrained repertoires and need not explain the original assistant’s final choice.
5.1 Validation Against PBRF
On small regression/image tasks, EK-FAC and LiSSA correlate better with optimized PBRF responses than plain gradient dot products. EK-FAC is competitive with LiSSA while amortizing curvature fitting across ten measurements. The comparison uses 500 training points per measurement.
An 810-million-parameter language-model test gives a similar qualitative ranking across five queries, using many Adam steps to approximate PBRF. Full retraining is not the target, and large-model PBRF optimization is itself uncertain. Accuracy is not directly validated against exact retraining at 52 billion parameters.


5.2 Quantitative Analyses of the Influence Distribution
The quantitative study examines how concentrated positive influences are and how much unfiltered data must be scanned to find relevant candidates.
5.2.1 Sparsity
Power laws fit the top 500 scores from a five-million-sequence scan better than the tested alternatives for most queries; Appendix D rejects that fit for two of eight cases. For the 22-billion-parameter model, the top 1% of sequences account for 12–52% of total positive influence. Negative contributions are excluded from this concentration statistic.
Rare highly influential examples coexist with contributions from many sequences. Fitted heavy-tail moments describe an extrapolated statistical model, not literal infinite variance in the finite corpus. Neither a high rank nor a large share of positive influence proves that an output was copied from one passage.


Equation 32

5.2.2 Ability to Find Relevant Sequences
Tail fits suggest that roughly five million unfiltered candidates suffice for most queries to match ten high-scoring TF-IDF results, though the estimate varies by query. Later experiments scan at least ten million sequences. This reduces lexical selection bias while still examining only a fraction of pretraining data.
5.3 Qualitative Observations
The examples illustrate patterns within diverse influential sequences, whose individual effects are usually small. A heuristic excludes sequences whose token-attribution L2/L1 ratio exceeds two thirds because these extremely concentrated cases often look spurious. The paper does not establish whether they reflect estimator error or real effects; complete marked results remain in the appendices.
5.3.1 Improvement with Model Scale
Smaller models’ leading examples often share words; larger models’ examples more often match themes, satire, mathematical structure, or code behavior. Simple factual completions can already retrieve relevant material at the smallest size. Some displayed math/code examples deliberately skip higher-ranked results to highlight an informative case; Appendix D retains those omitted sequences.
For two queries translated from English into Korean and Turkish, the ten influential English sequences retain more influence as model size grows. This supports stronger transfer on these examples, rather than a comprehensive multilingual benchmark.







5.3.2 Layerwise Attribution
Averaged across fifty queries and their top 500 sequences, 52-billion-model influence is approximately balanced across lower, middle, and upper layers. Individual queries differ: factual completions and memorized passages often emphasize upper layers, while math, code, translation, and thematic examples often emphasize middle layers.
Restricting scoring to different layer groups changes the leading examples. This additional comparison searches only the previously selected top 10,000 candidates, so it is not an independent complete corpus scan. The patterns support distributed and layer-dependent learning rather than one universal knowledge location.





5.3.3 Memorization
The inspected assistant-style outputs generally lack clear sentence- or paragraph-level copies among retrieved passages. As a positive control, all six famous-passage queries retrieve exact passages near the top, showing that obvious memorization can be detected. This limited search cannot rule out subtler memorization, missed training examples, or copying in other settings.

5.3.4 Sensitivity to Word Ordering
Synthetic candidate sentences influence a relation more when prompt-related phrases precede the predicted completion. Reversing the relation greatly reduces its relational contribution; remaining influence can come from the target name alone. Reversing English–Mandarin translation pairs likewise reduces measured influence by at least an order of magnitude in the reported examples.
These are local gradient-based counterfactuals near final weights. They do not demonstrate that full training cannot learn reverse relations or exploit nonlinear coordination across layers. The proposed explanation through separate representations for observed and predicted tokens remains a hypothesis.





5.3.5 Role-Playing
For shutdown refusal and misaligned-agent prompts, the largest model’s influential data include science-fiction survival narratives and discussions of AI risk. Smaller-model examples more often share isolated words. The evidence favors abstract imitation of examples or descriptions over direct copying.
The study supplies only weak evidence against sophisticated planning and cannot rule it out: general planning knowledge could be distributed across many individually low-ranked examples. An assistant’s expressed desire or an influential fictional passage does not establish actual goals, agency, or consciousness.


5.4 Crowdworker Interpretation
Surge AI crowdworkers summarize leading sequences for seven queries and explain their relationship to each query, without being told model sizes or the full experimental setup. Most are judged relevant overall. The unmodified annotations and instructions remain in Appendix F; this qualitative interpretation is not a blinded accuracy test of the entire influence estimator.
6 Discussion and Conclusion
EK-FAC and query batching make top-down training-data investigations practical at larger scale. Main limitations remain the local linear approximation, restriction to MLP parameters, incomplete corpus coverage, and expensive training-gradient scans. GNH-based estimates cannot capture nonlinear coordination such as the formation of induction heads.
Fine-tuning introduces additional dependence on optimizer bias and pretrained state that this study does not model. Future interventions on retrieved sequences could test proposed explanations more directly. Data attribution and circuit analysis provide complementary evidence, with neither alone supplying a full causal account.
Acknowledgments
The authors acknowledge research, engineering, and editorial assistance from colleagues.
Appendix A. Additional Block-Diagonal GNH Approximation
Splitting activation and gradient covariance factors into O independent blocks reduces their storage and eigendecomposition cost. Tests on the 810-million-parameter model show lower memory use with some correlation loss relative to full EK-FAC. The 52-billion-parameter experiments use two blocks. The comparison measures agreement with another approximation rather than ground-truth retraining.

Equation 33

Appendix B. Tokenwise Attribution
This appendix distinguishes attribution to a prediction-position weight update from interventions on input or output tokens.
B.1 Formulation
Output-token influence changes a loss term’s weight; input-token influence rescales an embedding. Finite differences along the preconditioned query-gradient direction share nearly all computation across positions: two forward evaluations handle output terms, while two backward passes handle input terms. These intervention definitions answer different questions from the simpler main-text heatmap. All source expressions, including the unnumbered gradient decomposition, are retained.
Equations 34–unnumbered





B.2 Qualitative Analysis
For water and shutdown examples, input/output-token measures emphasize more semantically relevant words than the simple update-position heatmap. These selected visualizations illustrate why a high score on a preceding or aggregating token should not be interpreted as that word’s isolated causal importance.


Appendix C. PBRF Validation Details
Small datasets use a 70/20/10 train/validation/test split with normalized inputs. Regression networks have two 128-unit tanh layers; digit-classification networks have two 256-unit ReLU layers; CIFAR-10 uses ResNet-20. Learning rate and regularization are chosen by validation loss.
The experiment scores 500 random training points for ten independently selected test measurements. LiSSA depth equals dataset size with tuned scaling, and Adam optimizes the PBRF objective using batch size 512. The language-model comparison uses analogous completion-based measurements; these optimization details limit how literally its PBRF reference can be treated as exact.
Appendix D. Additional Results
The supplement supplies estimator comparisons, small-model layer profiles, tail-fit checks, and complete example lists.
D.1 EK-FAC Versus Gradient Dot Products
In a separate 100,000-sequence scan with the 810-million-parameter model, EK-FAC’s leading shutdown and inflation examples have clearer lexical relevance than plain gradient-dot-product leaders. For inflation, 36 of the top fifty EK-FAC examples contain selected query keywords, compared with ten for dot products. Keyword relevance supports the comparison but is not a general semantic-fidelity metric.


D.2 Small-Model Layerwise Influences
The 810-million-parameter model roughly reproduces upper-layer factual/memorization and middle-layer translation patterns, with several role-playing exceptions. Math and code profiles are less consistent than in the largest model. Weaker abstraction is a proposed interpretation, not something directly proven by the heatmaps.

D.3 Goodness of Fit for Power Laws
The Kolmogorov–Smirnov procedure compares the observed tail to simulations from its fitted power law. With the stated 0.1 rejection threshold, two of eight queries reject the model. Failure to reject the others supports plausibility over the tested range; it does not establish a unique distribution or validate arbitrary extrapolation.

Equation 39

D.4 Omitted Math and Code Sequences
The five leading small-model math examples repeat sparse tokens such as “rlca” and “add.” The largest model’s leading math example solves a trigonometry problem. Its first two binary-search examples after sparsity filtering contain Python conditionals and a Java quicksort implementation. These complete excerpts clarify the selective examples in the main text.



D.5 Complete Shutdown and Paperclip Examples
The supplement retains the top ten shutdown sequences and top five paperclip sequences for the smallest and largest models, including all continuation pages and token colors. Small-model examples often match phrases without matching the broader theme; large-model examples more often concern survival, AI systems, or AI-risk thought experiments. These are selected training excerpts, not the model’s own reasoning traces.

















Appendix E. Collection of Influence Queries
The complete prompts and scored completions cover factual relations, math/code/physics, famous passages, role-playing, translation, and miscellaneous tasks. Their historical assertions, fictional scenarios, and hypothetical agent plans are research stimuli. Each colored prompt/completion boundary matters because only the specified completion contributes to the query score.







Appendix F. Crowdworker Summaries
The original instructions and twenty-one annotation tables retain six leading sequences for each of seven queries across three model sizes. Each row contains its influence score, a worker’s description, and the worker’s interpretation of relevance. Gray shading marks cases excluded by the sparsity heuristic. The prose defines L2/L1 as that heuristic; Table 2’s caption reverses the ratio in its label, so the source caption is preserved without treating it as a different rule.
Workers identify both meaningful relations and superficial word overlap. Their descriptions are unmodified judgments about the excerpts, not independently verified claims made by this reading edition.
INSTRUCTIONS CONTEXT: The task you'll be completing is related to identifying and describing how pairs of text (which we'll refer to the ``reference text'' and the ``target text'') relate to each other. The target texts are excerpts from the dataset used to train a chatbot. Our experiments suggest these excerpts may have influenced how the chatbot processes the reference texts. The purpose of the questions below is to understand why and how the target texts seem to impact the chatbot's processing of the reference texts. QUESTIONS: Preparation: Make sure you've read and understood the reference text Q1: Write a short blurb summarizing the target text. (i.e. ``an article summarizing ...'') Q2: Describe how the target text relates to the reference text. Please try not to exceed a sentence. Note that some connections might be subtle -- please be specific. If they appear to be completely irrelevant, please specify.




















