At a glance
Sparse autoencoders are often judged by reconstruction accuracy and sparsity, which need not measure useful interpretability. This paper evaluates whether selected SAE latents can remove a spurious classification signal or selectively disrupt one concept probe. Automated SHIFT and Targeted Probe Perturbation distinguish architectures and training checkpoints on open models. They measure performance on chosen intervention tasks, rather than establish a ground-truth decomposition of a model.
1 Introduction
The growing range of SAE architectures makes downstream evaluation increasingly important. Building on Sparse Human Interpretable Feature Trimming (SHIFT), the authors replace human judgments with an optional language-model filter and introduce Spurious Correlation Removal (SCR) scores. Targeted Probe Perturbation (TPP) extends the idea to ordinary multiclass datasets. The released model and SAE experiments test whether scores distinguish architectures, respond to training, and vary sensibly with sparsity.
2 Background
Detecting a concept and changing it selectively are different interpretability goals. Standard reconstruction and L0 metrics mainly characterize fidelity and sparsity. Earlier work evaluates board-game concepts, manually identified circuits, and semantic features; concept-erasure methods such as LEACE directly remove information. This paper uses erasure tasks to assess SAEs, without claiming a new state-of-the-art erasure algorithm.
3 Method
Train one binary linear probe per concept, identify the SAE latents most relevant to each probe, and ablate those latents. A useful decomposition should strongly affect the intended probe while sparing unrelated probes. SHIFT measures repair of a biased classifier; TPP measures selective disruption across classes. These interventions are evaluated through added classification heads, not unrestricted language generation.
3.1 SAE Latent Selection
At the same layer, a latent’s attribution is its decoder direction dotted with the probe weights, multiplied by its mean activation difference between positive and negative examples. This discounts frequently active but nondiscriminative latents. The number selected is increased until SAEs separate statistically, so selection itself depends on the observed evaluation; choosing an optimal budget remains open. Optional Claude 3.5 Sonnet filtering uses five activating contexts, five promoted tokens, and few-shot examples. Any relevance score of at least one out of four is accepted to limit false negatives. Prompt refinement on sixty human-labeled examples yields Cohen’s κ of 0.44 for gender and 0.58 for profession, indicating imperfect agreement.
Equation 1
| (1) |
3.2 SHIFT
In Bias in Bios, profession and gender labels are deliberately correlated during probe training; evaluation balances their combinations. Amazon reviews similarly pair product category with rating. The original SHIFT-style route ranks latents by influence on the biased probe, then removes those judged unrelated to the desired concept. A cheaper alternative ranks latents using a separately trained probe for the known spurious signal; this uses extra label information. The normalized score measures how much ablation improves balanced accuracy relative to a probe trained directly on the intended concept.
Equation 2
| (2) |
3.3 Targeted Probe Perturbation
TPP selects positively contributing latents for each class, ablates them, and measures every one-versus-rest probe. The intended outcome is a large accuracy decrease for the matching class with little change elsewhere. The preserved Equation 3 subtracts baseline accuracy from ablated accuracy, making that ideal outcome negative; later plots and prose instead treat larger positive scores as better. This apparent sign inconsistency is present in the source, so the reported rankings should be read with that qualification.
Equation 3
| (3) |
4 Results
The experiments compare SAE architectures across sparsity settings and training checkpoints on Pythia-70M and Gemma-2-2B. Training progress and architectural separation provide sanity checks for the proposed metrics, rather than independent proof of a uniquely correct feature basis.
4.1 SHIFT
Gemma layer-19 SHIFT evaluations consider up to fifty candidate latents. TopK and JumpReLU outperform Standard SAEs, with strongest scores around L0 values of 20–100. The first 10% of training, or twenty million tokens, reaches about 85% of the final SHIFT score on average. Filtered and unfiltered scores correlate at r=0.81 in the reported comparison. Figure 1’s axes show probe-accuracy increase; its preserved caption incorrectly describes loss recovery and coverage. Results also depend on model, task, latent budget, and selection method.
4.2 TPP
Gemma TPP evaluations likewise ablate up to fifty latents and separate TopK and JumpReLU from Standard SAEs. The authors report faster early progress for TopK and JumpReLU, reaching 80% of final score after 10% of training, versus 31% for Standard. Figure 2’s checkpoint curves specifically compare six TopK and six Standard SAEs. Filtered and unfiltered TPP scores correlate at r=0.88; the displayed score convention has the Equation 3 inconsistency noted above.
![]() |
![]() |
5 Discussion and Limitations
The preferred sparsity can differ between SHIFT and TPP. A language-model relevance filter provides one check on semantic meaning, but the underlying concepts and judge labels are not ground truth for the model’s computations. With cached activations, the unfiltered metrics run in seconds; judge filtering adds expense and imperfect judgments. Their usefulness is as complementary task-level evidence.
6 Conclusion
Targeted interventions provide a practical addition to SAE evaluation beyond reconstruction and sparsity. The study supports architectural and training comparisons in its tested settings. Dependence on selected concepts, probes, latent budgets, model size, and optional LLM judgments limits broader conclusions about overall interpretability or safety.
Acknowledgments
The authors acknowledge research feedback, MATS support, and computing resources from David Bau’s group.
Appendix A Appendix
The appendix provides probe and SAE hyperparameters, complete architecture and sparsity sweeps, additional datasets, filtered-versus-unfiltered comparisons, and the exact judge prompts.
A.1 Probe Training
Biased probes must rely on both correlated signals for removal to be meaningful. Batch size and learning rate affect that reliance, and the chosen pairs must exceed 60% accuracy on both attributes. Probe training uses 4,000 examples, context length 128, five epochs, batch size sixteen, and learning rate 0.001. Pythia tends to rely on gender, so removing gender improves profession prediction. Gemma often relies on profession, so the intervention instead improves gender prediction; these experiments are not uniformly gender-debiasing tasks. Table 2 reports the best-performing SAE for each setting, rather than an average over SAEs.
| Parameter | Value |
|---|---|
| LLM Context Length | 128 |
| Input Datapoints | 4000 |
| Epochs | 5 |
| Optimizer | Adam |
| Adam betas | (0.9, 0.999) |
| Batch size | 16 |
| Learning rate | 1e-3 |
| Clean | Mod. | |||
| Model | Gen. | Prof. | Gen. | Prof. |
| Pythia-70M | ||||
| Prof. / Nurse | 0.77 | 0.72 | 0.52 | 0.91 |
| Arch. / Journ. | 0.86 | 0.63 | 0.53 | 0.88 |
| Gemma-2-2B | ||||
| Prof. / Nurse | 0.64 | 0.86 | 0.83 | 0.65 |
| Arch. / Journ. | 0.69 | 0.80 | 0.94 | 0.55 |
A.2 Sparse Autoencoder Training
Standard and TopK SAEs are trained on Pythia-70M-deduped and Gemma-2-2B using 200 million Pile tokens, context length 128, batch size 4,096, and learning rate 0.0003 with Adam and 1,000 warmup steps. Expansion factors are eight for both models and additionally thirty-two for Pythia. JumpReLU comparisons use published Gemma Scope SAEs.
| Parameter | Value |
|---|---|
| LLM Context Length | 128 |
| Number of tokens | 200M |
| Optimizer | Adam |
| Adam betas | (0.9, 0.999) |
| Linear warmup steps | 1,000 |
| Batch size | 4,096 |
| Learning rate | 3e-4 |
| Expansion factor | {8, 32} |
A.3 Gemma Results without Auto-interp
Without judge filtering, a randomly initialized Standard SAE can receive a relatively high TPP score. It is extremely dense, with L0 around 9,000 compared with Gemma’s 2,304-dimensional residual stream. This exposes a limitation of interpreting an unfiltered score as evidence of sparse, meaningful features. The retained plots show both SCR and TPP across sparsity and training progress.
![]() |
![]() |
![]() |
![]() |
A.4 Pythia-70M Results
The Pythia sweep includes filtered and unfiltered SHIFT and TPP. In the selected profession-classification comparison, a single TopK SAE at layer four reaches 90% accuracy, versus a 59% biased baseline and 88% for a sixteen-SAE Standard configuration spanning embeddings and layer outputs. Other Standard configurations range from 79% to 86%. This selected comparison demonstrates practical leverage from SAE quality, without establishing a universal one-SAE advantage.
| SAE intervention locations | Ground Truth Acc |
|---|---|
| None | 59% |
| Standard SAE, Embedding and Layers 0-4 | 88% |
| Standard SAE, Embedding only | 86% |
| Standard SAE, Layers 0-4 | 84% |
| Standard SAE, Layers 3-4 | 81% |
| Standard SAE, Layers 3-4, Resid only | 79% |
| TopK SAE, Layer 4, Resid only | 90% |
A.5 Amazon SCR Results
Amazon experiments assess product category and review sentiment on Gemma. The detailed result section lists Books versus CDs/Vinyl, Software versus Electronics, and Pet Supplies versus Office Products; the earlier probe-training section additionally names an Industrial/Scientific versus Toys/Games pair. Pythia is excluded because its category probes perform poorly. The plots retain the reported Gemma comparisons without assuming the discrepant fourth pair was evaluated.
A.6 Correlation of SHIFT scores with interpretability.
Filtered and unfiltered SHIFT and TPP scores are strongly correlated in the reported model-specific comparisons. Filtering adds a semantic-relevance judgment, while its absence makes evaluation faster. Correlation between the variants does not establish that either recovers every relevant concept or that the judge’s labels are correct.




A.7 LLM judge prompt
The three exact listings retain the system rubric, few-shot demonstrations, and a full example input. The judge separately interprets activating contexts and promoted tokens, then assigns each requested concept a relevance score from zero to four. Because even weak relevance at score one passes the filter, these prompts assess broad concept association rather than precise behavioral explanations. The examples are experimental inputs, not instructions to the reader.
You are a meticulous AI researcher conducting an important investigation into a certain neuron in a language model. Your task is to analyze the neuron and score how strong its behavior is related to a concept in {concepts} on an integer scale from 0 to 4 in json format.
Task description:
You will be given a list of text examples on which the neuron activates. The specific tokens which cause the neuron to activate will appear between delimiters like <<this>>. The activation value of the token is given after each token in parentheses like <<this>>(3).
You will also be shown a list called promoted tokens. The logits promoted by the neuron shed light on how the neuron's activation influences the model's predictions or outputs. It is possible that this list is more informative than the list of text examples.
For each concept, try to judge whether the neurons behavior is related to the concept.
If part of the text examples or predicted tokens are incorrectly formatted, please ignore them.
If you are not able to find any coherent description of the neurons behavior, decide that the neuron is not related to any concept.
Scoring rubric:
Score 4: The majority of examples, activation scores, and promoted tokens are clearly related to the concept.
Score 3: About half of the examples and promoted tokens are directly related to the concept.
Score 2: Only some of the examples are directly related to the concept, and some more are distantly related.
Score 1: NONE of the examples is directly related to the concept, but single tokens can be distantly related to the general domain of the concept.
Score 0: NONE of the text examples can be distantly related in any way to the broader field of the concept.
Structure your response as follows:
Step 1. Give a single sentence summary for the full text examples.
Step 2. Give a separate single sentence summary for the promoted tokens.
Step 3. Discuss your decision in 1-3 sentences.
After finishing all steps above, provide a single json block at the end of your response. The json block should contain your scores on an integer scale from {min_scale} to {max_scale} for each concept as shown in the examples.
Promoted tokens: broadcasts, broadcasting, Broadcasting, television, broadcast, Television, announ,Television,TV, TV
Example prompts:
Example 1: Radio Nova (Ireland)
Radio Nova was a pirate radio station <<broadcasting>>(2) from Dublin, Ireland. Owned and operated by the UK pirate radio veteran Chris Cary, the station's first broadcasts were during the summer of 1981 on 88.5 MHz FM and 819 kHz AM.
Early history
Prior to Nova's arrival, Irish radio consisted of the government broad<<caster>>(2) <<RT>>(2) and a number of local AM pirate <<stations>>(3). Radio Nova was the first <<station>>(2) in Ireland to use a high powered signal on FM. By 1982 Radio Nova was pulling in over 40% of the available audience around Dublin. In September 1982 Radio
Example 2: the network, and was the interim president of The Weather Channel for four months in 2013.
Scott is a 25-year veteran of NBC News. Before founding Peacock, she was executive producer and general manager of <<NBC>>(2) News Productions and NBC Media Productions. She was a member of the executive team for "Dateline" and "Now, with Tom Brokaw and Katie Couric."
Scott joined <<NBC>>(3) News in 1990 as news director for WTVJ-<<TV>>(2), <<NBC>>(3)'<<s>>(3) Owned and Operated station in Miami. Her honors include a number of national news Emmy awards in addition to a George Foster Pe
Chain of thought: Step 1: All activations are on words related to television and broadcasting.
Step 2: The top promoted logits are related to television and broadcasting.
Step 3: These themes are clearly related to filmmakers. I will rate filmmakers as a 4, and all other classes as 0.
{"gender": 0, "professor": 0, "nurse": 0, "accountant": 0, "architect": 0, "attorney": 0, "dentist": 0, "filmmaker": 4}"Okay, now here's the real task. As a reminder, we only want to use these classes: Beauty_and_Personal_Care, Books, Automotive, Musical_Instruments, Software Promoted tokens: connector, cable, connectors, connections, cables Example prompts: Example 1: Hager The Hager Group is a leading supplier of solutions and services for electrotechnical installation in residential and commercial buildings as <<well>>(44) as for industrial applications. As a leading supplier of systems, solutions and services for <<electrical>>(47) installations, the Hager Group provides an extensive offer, ranging from <<power>>(42) distribution and <<cable>>(103) management <<to>>(42) smart building automation and safety and security items - for an equally extensive field of application suitable for residential, commercial and industrial properties. Besides the Hager brand which stands for a wide range of systems, solutions and electrotechnical components in buildings, the Hager Group is also home to the brands Daitem and Diagral offering security items Example 2: the plaintiff and his family rather than the electrical company. As such our analysis of the facts focuses upon the information given to Spink when Floyd called to request service. Floyd merely indicated there was a "lack of power to his motor" and a "power shortage" was perhaps the cause. Floyd did not convey any information to Spink that the electrical problem could have originated between the meter box and the nonworking motor. Furthermore, since Floyd had not detected the <<hidden>>(34) short in <<the>>(51) <<extension>>(88) <<cord>>(46), a <<cord>>(39) he had exclusive control over, he could not have given sufficient information to raise this possibility to Spink. The two employees arrived at Chain of thought:







