Literature
Display
← All papers

Sparse Weight Decomposition for Efficient Circuit Extraction

Chuanhao Yan · Xuhan Huang · Yawen Duan · Zhenfei Yin · Hang Zhao · Bryan Dai · Jie Fu

Preprint posted · arXiv (preprint) ↗

Interpretability18 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete 48-page v1 dated 4 August 2026. It preserves all 37 figures and 11 tables, including the appendix diagnostics. Sparse factorization still requires optimization; the efficiency claim concerns calibration or optimizer-consumed tokens, not equivalent computation or the original dense model’s pretraining cost.

Complete original figure with every panel, label and legend.
Figure 1: The SWD pipeline. Step 1 (factorization). A dense linear projection with weight matrix 𝑾{\bm{W}} is reparameterized by two sparse factors, 𝑾≈𝑨​𝑩{\bm{W}}\approx{\bm{A}}{\bm{B}}. Each shared intermediate coordinate ii is a bottleneck unit: it reads from the input through 𝑨:,i{\bm{A}}_{:,i} and writes to the output through 𝑩i,:{\bm{B}}_{i,:}, thereby defining the fixed rank-one path 𝑨:,i𝑩i,:{\bm{A}}_{:,i}{\bm{B}}_{i,:}. In the graph representation, each nonzero scalar matrix entry corresponds to a directed edge whose weight is that entry, while a zero entry corresponds to no edge. Specifically, a nonzero entry 𝑨p​i{\bm{A}}_{pi} defines a read edge from input coordinate pp to unit ii, while a nonzero entry 𝑩i​q{\bm{B}}_{iq} defines a write edge from unit ii to output coordinate qq. For illustration, the diagram uses six bottleneck units. Step 2 (circuit extraction). These six units are scored and ranked by task attribution. The bar length represents the attribution score (longer is higher), and labels #1– #6 indicate the resulting ranking, with #1 ranked highest. The top-kk units are selected for the circuit; in this illustration, k=3k=3, so the three highest-ranked units are retained. The activation of each unselected unit ii is replaced by its mean μi\mu_{i}.
Code: github.com/veri-safe/SWD Model: huggingface.co/veri-safe/SWD Blog: huggingface.co/spaces/veri-safe/SWD-Blog

At a glance

Sparse Weight Decomposition (SWD) approximates each pretrained linear projection by two sparse factors. Their intermediate coordinates become individually addressable read–write paths for circuit ranking and ablation. Single-matrix experiments report high replacement fidelity with much less fitting data than learned replacements, and smaller task circuits in active-edge cost. Fixed-support finetuning extends the method to GPT-2’s transformer body; a weight-only variant omits calibration activations. These are local, protocol-dependent explanations, with explicit exceptions to fidelity matching and limited large-model coverage.

1 Introduction

Dense model weights distribute task behavior across many coordinates. SWD obtains sparse intermediate paths by repurposing Double Sparse Factorization rather than learning a separate activation dictionary. Unlike exact dense SVD components, each path has few nonzero incoming and outgoing weights. Experiments separate replacement fidelity from the cost of the subsequently selected task circuit, and test single matrices, GPT-2’s whole transformer body, exact dense controls, weight-only fitting and qualitative interventions.

2 Methodology

The method first constructs sparse factors, then applies a common task-based circuit extraction and evaluation protocol to their intermediate units.

2.1 Sparse Weight Decomposition

A bottleneck unit is a scalar intermediate activation coupled to sparse input and output directions. The factorization changes the model approximately, so replacement quality must be checked before interpreting its circuits.

Sparsity on the edges.

Sparsity is imposed on both read and write weights. Exact dense factorizations also expose intermediate units, but each normally touches almost every input/output coordinate. The controls test whether sparse connectivity reduces circuit cost beyond this change of basis.

Objective.

For W of shape input-width × output-width, SWD fits A and B under a fixed combined nonzero budget K. Calibration-weighted squared reconstruction error prioritizes activation directions the model encounters; setting the activation Gram matrix to identity yields the weight-only Frobenius objective. Larger reported sparsity means fewer surviving factor entries. The original derivation is retained below; its matrix quadratic form needs a trace to equal the scalar squared-error objective, although the printed expression omits it.

Equations
min𝑨,𝑩⁡𝔼𝒉∼𝒟cal​‖𝒉​𝑾−𝒉​𝑨​𝑩‖22s.t.‖𝑨‖0+‖𝑩‖0≤K,\min_{{\bm{A}},{\bm{B}}}\;\mathbb{E}_{{\bm{h}}\sim\mathcal{D}_{\mathrm{cal}}}\left\|{\bm{h}}{\bm{W}}-{\bm{h}}{\bm{A}}{\bm{B}}\right\|_{2}^{2}\quad\mathrm{s.t.}\quad\|{\bm{A}}\|_{0}+\|{\bm{B}}\|_{0}\leq K,
𝔼𝒉∼𝒟cal​‖𝒉​𝑾−𝒉​𝑨​𝑩‖22=𝔼𝒉∼𝒟cal​(𝑾−𝑨​𝑩)⊤​(𝒉⊤​𝒉)​(𝑾−𝑨​𝑩)\displaystyle\mathbb{E}_{{\bm{h}}\sim\mathcal{D}_{\mathrm{cal}}}\|{\bm{h}}{\bm{W}}-{\bm{h}}{\bm{A}}{\bm{B}}\|_{2}^{2}=\mathbb{E}_{{\bm{h}}\sim\mathcal{D}_{\mathrm{cal}}}({\bm{W}}-{\bm{A}}{\bm{B}})^{\top}({\bm{h}}^{\top}{\bm{h}})({\bm{W}}-{\bm{A}}{\bm{B}})
=(𝑾−𝑨​𝑩)⊤​𝔼𝒉∼𝒟cal​[𝒉⊤​𝒉]​(𝑾−𝑨​𝑩),\displaystyle=({\bm{W}}-{\bm{A}}{\bm{B}})^{\top}\mathbb{E}_{{\bm{h}}\sim\mathcal{D}_{\mathrm{cal}}}[{\bm{h}}^{\top}{\bm{h}}]({\bm{W}}-{\bm{A}}{\bm{B}}),
𝑾≈𝑾^=𝑨​𝑩{\bm{W}}\approx\widehat{{\bm{W}}}={\bm{A}}{\bm{B}}
𝑮=𝔼𝒉∼𝒟cal​[𝒉⊤​𝒉]{\bm{G}}=\mathbb{E}_{{\bm{h}}\sim\mathcal{D}_{\mathrm{cal}}}[{\bm{h}}^{\top}{\bm{h}}]

Solver.

The nonconvex sparsity-constrained problem is approximated with the DSF solver: alternate factor updates using regularized least squares, hard thresholding and ADMM dual updates. Local finalization freezes supports and A, then refits B. When all projections are replaced, a separate next-token finetuning stage updates existing nonzero values in both factors while keeping supports and other pretrained parameters fixed.

Takeaway. SWD turns a dense matrix into two sparse factors so that every bottleneck unit becomes a rank-one path with few read and write edges.
Selectable original takeaway
Takeaway. SWD turns a dense matrix into two sparse factors so that every bottleneck unit becomes a rank-one path with few read and write edges.

2.2 Circuit Extraction from Sparse Factors

Each intermediate scalar multiplies one write row, so the reconstructed output is an additive sum of read–write paths. Rank units by positive task-margin attribution on training examples. To test sufficiency, keep the selected paths and replace all others by their training-split mean activations; to test necessity, mean-ablate the selected paths and retain the remainder. Mean ablation preserves average activation levels while removing input dependence. Independent addressability does not imply functional independence across nonlinear computations.

Equations
zi(𝒉)=𝒉𝑨:,i,𝒉𝑾^=𝒉𝑨𝑩=∑i=0m−1zi(𝒉)𝑩i,:.z_{i}({\bm{h}})={\bm{h}}{\bm{A}}_{:,i},\qquad{\bm{h}}\widehat{{\bm{W}}}={\bm{h}}{\bm{A}}{\bm{B}}=\sum_{i=0}^{m-1}z_{i}({\bm{h}}){\bm{B}}_{i,:}.
zikeep​(𝒉,S)={zi​(𝒉),i∈S,μi,i∉S,ziabl​(𝒉,S)={μi,i∈S,zi​(𝒉),i∉S.z_{i}^{\mathrm{keep}}({\bm{h}};S)=\begin{cases}z_{i}({\bm{h}}),&i\in S,\\ \mu_{i},&i\notin S,\end{cases}\qquad z_{i}^{\mathrm{abl}}({\bm{h}};S)=\begin{cases}\mu_{i},&i\in S,\\ z_{i}({\bm{h}}),&i\notin S.\end{cases}
Takeaway. Each bottleneck is an independently ablatable path. We rank paths on circuit training examples, and then test sufficiency and necessity on the test set.
Selectable original takeaway
Takeaway. Each bottleneck is an independently ablatable path. We rank paths on circuit training examples, and then test sufficiency and necessity on the test set.

3 Experiments

Experiments compare fidelity before pruning and the minimum circuit cost needed to attain held-out sufficiency or necessity targets. Large-model studies replace one projection; whole-body replacement is evaluated on GPT-2 Small.

3.1 Baselines

Transcoders learn sparse activation features; VPD learns parameter components with an input-dependent causal-importance selector; sparse pretraining trains a separate sparse model. VPD-KL uses output-distribution KL, whereas VPD-Recon-CI changes the fidelity term to local reconstruction. VPD-KL fails the required fidelity match and is shown only for replacement quality. The sparse-pretraining comparison is confined to the whole-model experiment.

Property    SWD (ours) Transcoder VPD Sparse pretrain
Separate model training (additional optimization cost)    No; fit sparse factors post hoc Yes; train a replacement network for the pretrained model Yes; train a decomposition and input-dependent selector for the pretrained model Yes; train the complete sparse model from scratch
Representation and unit of analysis    Two sparse factors; one bottleneck unit is a rank-one read–write path A replacement network; one unit is a hidden feature A parameter decomposition; one unit is a rank-one parameter component A sparse pretrained model; one unit is a native hidden channel
Type of sparsity    Weight sparsity in the factor matrices Activation sparsity in hidden features Input-dependent component selection over parameter components Weight sparsity throughout the model; activation sparsity is disabled in our run
Optimization objective    Reconstruct target weights or their outputs on calibration data under fixed factor sparsity Reconstruct original MLP outputs while encouraging sparse feature activations Match model outputs or replaced-matrix outputs, with parameter reconstruction and sparse component use Minimize language-model loss while progressively sparsifying weights
Data required    None or a small amount of calibration data* Text used to train the replacement network Text used to train the decomposition and selector Full pretraining corpus
Applicable scope    MLP and attention matrices; from one matrix to all model projections MLP computations; one MLP at a time in our comparisons MLP and attention matrices; from one matrix up to full-model replacement Whole model during pretraining

*At matched replacement CE, SWD uses less than 1% of the data used by the corresponding trained baseline in our experiments. This greater data use does not consistently reduce circuit cost: SWD typically reaches the same sufficiency or necessity target with fewer active edges; see Sections 3.3 and 3.4.

Table 1: Comparison of the sparse intervention methods evaluated in this paper.

3.2 Evaluation Protocol

FineWeb-Edu supplies fitting data and a disjoint fidelity test split. Circuit tasks use separate train/test examples: train estimates positive first-order attribution and ablation means; held-out test evaluates fixed nested top-k sets. Sufficiency is retained task margin divided by the unpruned replacement’s margin; necessity is its margin minus the selected-unit-ablated margin. Cost is selected units or nonzero read/write edges of valid selected paths. Curves take the minimum cost among evaluated prefixes reaching each target. The usual CE tolerance is .001, but stated controls and some 27B curves exceed it; matched average CE also does not imply identical predictions. Intervention choice can affect circuit conclusions.

Equations
Suffτ​(S,𝒟)=Qτ​(Mkeep​(S),𝒟)Qτ​(Munpruned,𝒟),\mathrm{Suff}_{\tau}(S;\mathcal{D})=\frac{Q_{\tau}(M_{\mathrm{keep}}(S);\mathcal{D})}{Q_{\tau}(M_{\mathrm{unpruned}};\mathcal{D})},
NecDropτ​(S,𝒟)=Qτ​(Munpruned,𝒟)−Qτ​(Mabl​(S),𝒟).\mathrm{NecDrop}_{\tau}(S;\mathcal{D})=Q_{\tau}(M_{\mathrm{unpruned}};\mathcal{D})-Q_{\tau}(M_{\mathrm{abl}}(S);\mathcal{D}).
Cunit​(S)=|S|,Cedge​(S)=∑i∈Seff(‖𝐫i‖0+‖𝐰i‖0),C_{\mathrm{unit}}(S)=|S|,\qquad C_{\mathrm{edge}}(S)=\sum_{i\in S_{\mathrm{eff}}}\left(\|\mathbf{r}_{i}\|_{0}+\|\mathbf{w}_{i}\|_{0}\right),
Qτ​(M,𝒟)=𝔼𝒙∈𝒟​[gτ​(M,𝒙)]Q_{\tau}(M;\mathcal{D})=\mathbb{E}_{{\bm{x}}\in\mathcal{D}}[g_{\tau}(M,{\bm{x}})]

3.3 Single-Matrix Replacement

For GPT-2 layer-8 output projection and Qwen down projections, SWD reaches low error with fewer fitting tokens and generally needs fewer active circuit edges than the trained replacements. The main matched comparison uses sparsity .5; sparsity .75 usually trades fidelity for fewer edges. GPT-2 zero-ablation controls retain the cost advantage with rankings fixed. Qwen3.5-27B confirms feasibility for one matrix, with TC48k and VPD-Recon-CI outside the stated matching tolerance. Whole-MLP replacement gives more mixed results, including competitive Transcoder sufficiency and unit counts.

Complete original figure with every panel, label and legend.
Figure 2: GPT-2 single-matrix replacement quality. SWD reaches low CE delta with far less data; VPD-KL is included for replacement-quality context and excluded from circuit extraction evaluation. See Appendix C.1 for companion metrics and settings.
Complete original figure with every panel, label and legend.
Figure 3: GPT-2 single-matrix circuit results. SWD reaches the target thresholds with markedly fewer active edges than the Transcoder and VPD-Recon-CI baselines.
Complete original figure with every panel, label and legend.
Figure 4: Qwen single-matrix replacement quality. Left: Qwen2.5-3B. Right: Qwen3.5-27B. SWD reaches low CE delta with substantially fewer tokens than the trained baselines. Results for the other Qwen2.5 model sizes are reported in Appendix C.4.
Complete original figure with every panel, label and legend.
Figure 5: Qwen2.5 3B single-matrix circuit results. SWD reaches the same sufficiency and necessity thresholds at substantially fewer active edges. The 0.5B/1.5B results and their settings appear in Appendix C.4.
Complete original figure with every panel, label and legend.
Figure 6: Qwen3.5-27B single-matrix circuit results. The vertical axis reports the minimum active edges. Checkpoint settings and the selected-unit companion are reported in Appendix C.5.
Takeaway. SWD requires substantially less data than trained replacement baselines to reach matched replacement fidelity, while achieving better circuit sufficiency and necessity with fewer selected units and active edges.
Selectable original takeaway
Takeaway. SWD requires substantially less data than trained replacement baselines to reach matched replacement fidelity, while achieving better circuit sufficiency and necessity with fewer selected units and active edges.

3.4 Full-Model Replacement

Replacing all 48 attention/MLP matrices across GPT-2’s twelve blocks gives CE 3.90 before finetuning and 3.44 after fixed-support finetuning, versus 3.45 for a separate sparse-pretrained model at approximately 26.77M active body weights. SWD uses 4.19M calibration plus 16.38M finetuning token presentations, versus 2.884B sparse-pretraining presentations. This excludes inherited dense pretraining and is not a FLOP comparison. Circuit results are broadly comparable on GreaterThan/gendered pronouns; sparse pretraining has near-zero or negative necessity drop on docstring/IOI. Its circuit candidates and activation implementation also differ, as detailed in B.5.3.

Method Data used Active nonzero weights LM CE loss
SWD 4.194.19M calibration tokens 26.7726.77M 3.903.90
SWD-FT 4.194.19M calibration + 16.3816.38M fine-tuning tokens 26.7726.77M 3.443.44
Sparse pretrain 2.8842.884B sparse-pretraining tokens 26.7726.77M 3.453.45
Table 2: GPT-2 Small full-model replacement summary at approximately matched transformer-body active nonzeros. SWD-FT keeps the SWD support fixed and refits only nonzero factor entries, achieving CE comparable to that of the sparse-pretrained model.
Complete original figure with every panel, label and legend.
Figure 7: GPT-2 Small full-model replacement circuit results. SWD-FT remains effective across all four tasks. Sparse pretraining is comparable on greater-than and gendered-pronoun, but has near-zero or negative necessity drop on docstring and IOI.
Takeaway. Fixed-support fine-tuning restores replacement quality and matches sparse pretraining at an approximately equal active-weight budget and circuit cost-quality tradeoff with far fewer training tokens.
Selectable original takeaway
Takeaway. Fixed-support fine-tuning restores replacement quality and matches sparse pretraining at an approximately equal active-weight budget and circuit cost-quality tradeoff with far fewer training tokens.

3.5 Ablation: Exact Dense Reparameterizations

Full-rank SVD and seeded random-orthogonal factorizations reproduce the dense target exactly, yet need more read/write edges for similar circuit targets. This isolates sparse connectivity from factorization alone under the shared ranking/ablation protocol. Dense controls remain competitive in some selected-unit comparisons; the clearer advantage is active-edge cost.

Complete original figure with every panel, label and legend.
Figure 8: Exact dense reparameterization controls for the GPT-2 output-projection surface. The comparison with SWD tests whether sparse read/write structure reduces the number of active edges beyond factorization alone. See Appendix F for the control settings.
Takeaway. The exact SVD and Random-B factorizations require more active edges than SWD to preserve the same task behavior or produce the same ablation effect, isolating the contribution of sparse read/write structure.
Selectable original takeaway
Takeaway. The exact SVD and Random-B factorizations require more active edges than SWD to preserve the same task behavior or produce the same ablation effect, isolating the contribution of sparse read/write structure.

3.6 Zero-Data Factorization

Weight-only SWD sets the Gram matrix to identity and uses no calibration text to fit factors. It preserves the raw matrix more accurately: at sparsity .75 its relative Frobenius error is .366 versus .415 for calibrated SWD. Calibration instead better preserves held-out CE under aggressive sparsity. Task circuits remain useful, but their ranking, ablation statistics and evaluation still require task data. Tracking circuits across training checkpoints is proposed future work.

Complete original figure with every panel, label and legend.
Figure 9: Parameter-space and behavioral fidelity of zero-data and activation-calibrated SWD on the GPT-2 output-projection surface. Left: relative Frobenius reconstruction error. Right: held-out CE increase. Zero-data SWD better preserves the weight matrix, whereas activation calibration better preserves model behavior under aggressive sparsification. Appendix G.1 gives the factorization and evaluation settings.
Takeaway. Calibration data improves replacement fidelity, but useful circuit units can still be recovered directly from checkpoint weights without activation data.
Selectable original takeaway
Takeaway. Calibration data improves replacement fidelity, but useful circuit units can still be recovered directly from checkpoint weights without activation data.

3.7 Qualitative Results

The qualitative studies examine semantic hypotheses, a directional weight edit, one attention intervention and pair interactions. They illustrate possible uses of the representation rather than establish general monosemanticity or complete mechanistic understanding.

3.7.1 Semantic Audit of a Task Circuit

Rank 9,208 valid all-layer GPT-2 output-projection units by GreaterThan attribution. The six displayed units are the top two in layers 6, 8 and 10 and lie within the global top 512. Twenty strongest activation contexts per unit, drawn from 2,048 independent WikiText-2 tokens, support four number/quantity/measurement hypotheses and two contextual lexical/syntactic hypotheses. Appendix H specifies model-generated labeling.

Complete original figure with every panel, label and legend.
Figure 10: Semantic audit of high-attribution paths in a GreaterThan SWD circuit. Arrows show the directed computation within each factorized mlp.c_proj: the MLP input activation 𝒉ℓ{\bm{h}}_{\ell} is read by bottleneck unit ii to form zi=𝒉ℓ𝑨:,iz_{i}={\bm{h}}_{\ell}{\bm{A}}_{:,i}, and the unit writes the vector contribution zi𝑩i,:z_{i}{\bm{B}}_{i,:} toward the layer output. Each drawn arrow summarizes the corresponding sparse read or write connections, rather than a measured pairwise interaction between units. The ++ node denotes vector addition of the bottleneck-unit write contributions into the residual stream. We draw the two highest-ranked units in each displayed layer and omit the remaining selected units. Nodes are selected by task-related first-order attribution; their semantic labels come from recurring patterns in task-independent, high-activation contexts and are not used for selection. Four of the six displayed units have number-, quantity-, or measurement-related hypotheses.

3.7.2 Targeted Bottleneck-Unit Editing

All edits start from the same original dense layer-8 output matrix. Unit c205 supplies a read direction for a rank-one update toward “down” rather than “left” on “The opposite of up is”. Its largest positive setting changes the final margin by .216 with mean KL 4.02×10⁻⁵ on seven unrelated prompts. It improves measured locality over the selected random-unit and rank-4 LoRA controls; a target-conditioned dense oracle is stronger. This is one screened target and a small probe set.

Complete original figure with every panel, label and legend.
Figure 11: Target-margin change versus measured side effect for single-target edits applied to the same original dense weight matrix. The horizontal axis is mean final-token KL on seven unrelated factual prompts; the vertical axis is the change in the down versus left margin. Positive and negative edit settings form the upper and lower branches. At comparable margin change, farther left indicates lower measured side effect. The SWD curve uses the read direction of bottleneck unit c205; the dense rank-one oracle uses the target activation directly.

3.7.3 Mechanistic Diagnostics

Static query/key geometry selects candidates for prompt replay and intervention. In the highlighted layer-9/head-3 example, reconstructed attention has KL .0405 from the dense head; ablating q266 produces KL 2.496 from the reconstruction and maximum probability change .775. Separate diagnostics find mostly weak post-GELU contribution correlations but broader pre-GELU nonlinear interactions. Additive paths should therefore not be treated as mutually independent mechanisms.

Complete original figure with every panel, label and legend.
Figure 12: Prompt-level validation and intervention for layer 9, head 3. Top: dense attention, SWD-reconstructed attention, and their absolute difference; mean dense-to-reconstructed attention KL is 0.04050.0405. Bottom: SWD-reconstructed attention before and after ablating query bottleneck unit q266, followed by their elementwise absolute difference. Relative to the intact reconstruction, ablating q266 produces a mean attention KL of 2.4962.496 and a maximum absolute change of 0.7750.775 in any attention probability.
Takeaway. Across semantic auditing, targeted editing, and mechanistic diagnostics, SWD bottleneck units provide interpretable and manipulable units for analyzing model computation, while the observed interactions caution against treating every unit as independent or monosemantic.
Selectable original takeaway
Takeaway. Across semantic auditing, targeted editing, and mechanistic diagnostics, SWD bottleneck units provide interpretable and manipulable units for analyzing model computation, while the observed interactions caution against treating every unit as independent or monosemantic.

4 Related Work

Prior work supplies the circuit tests, learned-feature and parameter-space alternatives, sparse-model motivation and factorization solver used here.

Task-level circuit discovery and evaluation.

ACDC, attribution patching and edge pruning search existing computation graphs. SWD instead introduces a new basis of intervenable paths inside dense linear maps; these selection algorithms can then operate on that basis. Held-out behavior and intervention definitions matter more than agreement with one presumed canonical circuit.

Sparse activation features and learned replacement modules.

Sparse autoencoders learn activation dictionaries, while Transcoders approximate MLP input/output computations with sparse features. SWD fits sparse weight factors applicable to both attention and MLP projections. Its units are parameter-side rank-one paths, without assuming each has a single semantic meaning.

Parameter-space interpretability.

APD, SPD and VPD learn parameter components and sparse per-input use. NaNA interprets exact SVD components as detector–effector units. SWD constructs sparse read/write components independently of the downstream circuit task, then applies task-margin ranking.

Sparse models and post-training pruning.

Post-training pruning removes scalar weights; sparse pretraining changes connectivity while training a model. SWD adds explicit intermediate units to a pretrained checkpoint, enabling path-level selection and ablation. The whole-model and exact-factor controls probe related but distinct sources of circuit sparsity.

Matrix factorization and compression.

Matrix compression has long used low rank, nonnegativity and sparsity. SWD adopts Double Sparse Factorization and activation calibration, then evaluates the resulting units for circuit extraction. Its contribution is this interpretability application and evaluation, not a new DSF optimization algorithm.

5 Limitations

Cheaper extraction narrows one practical bottleneck, while replacement fidelity, non-uniqueness and verification remain unresolved.

Efficient extraction is not complete understanding.

Small circuits affecting held-out task margins do not provide a global account of the model or guarantee coverage of rare safety-relevant behavior. Reduced fitting data and circuit edges do not settle the broader limits of mechanistic interpretation.

SWD circuits are local and non-unique.

The approximate factors are not established as identifiable; units can depend on calibration, initialization, objective and sparsity, in addition to scaling/permutation symmetries. Average reconstruction/CE may hide input-specific drift. First-order ranking can miss important mechanisms, and ablation, pruning thresholds and graph granularity change conclusions. Treat results as task- and protocol-specific causal explanations, not unique ground truth.

Scaling verification remains open.

The 27B experiment covers one matrix, whole-model replacement covers GPT-2 Small, and the semantic audit shows six labeled units. Broader decompositions, prompts and adversarial edge cases need independent verification. SWD complements behavioral and representation-level analysis rather than certifies understanding or safety.

Takeaway. SWD makes local circuit hypotheses substantially cheaper to extract and test, but it does not eliminate MI’s non-uniqueness, distributional, verification, or scaling limitations.
Selectable original takeaway
Takeaway. SWD makes local circuit hypotheses substantially cheaper to extract and test, but it does not eliminate MI’s non-uniqueness, distributional, verification, or scaling limitations.

6 Conclusion

Sparse read/write factorization makes compact local circuit hypotheses available directly from pretrained weights. Fidelity checks, fixed-support finetuning and causal task tests make these paths useful analytical objects, while the controls attribute much of the edge-cost advantage to sparse connectivity. General interpretation remains limited by approximate replacements and interacting, non-unique units.

Appendix A Notation

The complete notation table remains below. Layer/head/unit indices are zero-based; displayed attribution ranks are one-based. A valid unit has at least one nonzero read and write edge. “Unit” encompasses SWD bottlenecks, Transcoder features, VPD components and sparse-pretrained latent channels.

Symbol Meaning
𝑾,𝑾^{\bm{W}},\widehat{{\bm{W}}} Dense target weight matrix and its SWD replacement, 𝑾^=𝑨​𝑩\widehat{{\bm{W}}}={\bm{A}}{\bm{B}}.
𝑨,𝑩{\bm{A}},{\bm{B}} Sparse read and write factors, 𝑨∈ℝdin×m{\bm{A}}\in\mathbb{R}^{d_{\mathrm{in}}\times m} and 𝑩∈ℝm×dout{\bm{B}}\in\mathbb{R}^{m\times d_{\mathrm{out}}}.
din,doutd_{\mathrm{in}},d_{\mathrm{out}} Input and output dimensions of the target matrix.
m,Cm,C Numbers of SWD bottleneck units and VPD component slots, respectively.
i,ki,k ii indexes a candidate unit; kk is the number of highest-ranked units retained in a top-kk circuit.
𝒙,𝒉ℓ,t​(𝒙){\bm{x}},{\bm{h}}_{\ell,t}({\bm{x}}) Input text example 𝒙{\bm{x}} and the input activation to target module ℓ\ell at token position tt.
𝑯{\bm{H}} Calibration-activation matrix whose rows are input-activation vectors 𝒉{\bm{h}}.
𝑮,𝑰{\bm{G}},{\bm{I}} Calibration Gram matrix 𝑮=𝔼⁡[𝒉⊤​𝒉]{\bm{G}}=\mathbb{E}[{\bm{h}}^{\top}{\bm{h}}] and the identity matrix used by the zero-data objective.
K,sK,s Total factor-entry budget K=‖𝑨‖0+‖𝑩‖0K=\|{\bm{A}}\|_{0}+\|{\bm{B}}\|_{0} and factor sparsity s=1−K/‖𝑾‖0s=1-K/\|{\bm{W}}\|_{0}.
𝒟cal\mathcal{D}_{\mathrm{cal}} Distribution of calibration activations used to fit SWD.
𝒟τ,train,𝒟τ,test\mathcal{D}_{\tau,\mathrm{train}},\mathcal{D}_{\tau,\mathrm{test}} Training and held-out evaluation splits for circuit task τ\tau.
zi,μi,aiz_{i},\mu_{i},a_{i} Activation of unit ii, its train-split mean used for ablation, and its task-attribution score. For SWD, zi=𝒉𝑨:,iz_{i}={\bm{h}}{\bm{A}}_{:,i}.
Sk,SeffS_{k},S_{\mathrm{eff}} SkS_{k} is the set of the kk highest-ranked units; SeffS_{\mathrm{eff}} contains the selected units with at least one nonzero read edge and one nonzero write edge.
𝐫i,𝐰i\mathbf{r}_{i},\mathbf{w}_{i} Method-specific read and write weight vectors of unit ii; for SWD these are 𝑨:,i{\bm{A}}_{:,i} and 𝑩i,:{\bm{B}}_{i,:}.
Cunit​(S),Cedge​(S)C_{\mathrm{unit}}(S),C_{\mathrm{edge}}(S) Selected-unit cost |S||S| and active-edge cost ∑i∈Seff(‖𝐫i‖0+‖𝐰i‖0)\sum_{i\in S_{\mathrm{eff}}}(\|\mathbf{r}_{i}\|_{0}+\|\mathbf{w}_{i}\|_{0}).
MunprunedM_{\mathrm{unpruned}} Replacement model before circuit pruning.
Mkeep​(S),Mabl​(S)M_{\mathrm{keep}}(S),M_{\mathrm{abl}}(S) Models that respectively retain only SS or mean-ablate SS.
gτ​(M,𝒙)g_{\tau}(M,{\bm{x}}) Per-example answer-versus-distractor logit margin for task τ\tau.
Qτ​(M,𝒟)Q_{\tau}(M;\mathcal{D}) Average task margin of model MM over split 𝒟\mathcal{D}.
Table 3: Notation used throughout the paper.

Appendix B Experimental Details

Fitting and fidelity evaluation use disjoint language-model data. Circuit rankings and ablation means are trained once and fixed before held-out circuit evaluation. Replacement surfaces, token accounting and baseline checkpoint choices remain explicit.

B.1 Models and Replacement Targets

Table 4 identifies each exact replacement surface: GPT-2 layer-8 output projection, middle Qwen2.5 down projections, Qwen3.5 layer-31 down projection, GPT-2 layer-8 complete MLP, or all 48 GPT-2 body matrices. Single-matrix comparisons share the same target within each model.

Setting Model Replacement surface
GPT-2 single matrix GPT-2 Small Layer 8 mlp.c_proj
Qwen2.5 scaling Qwen2.5 0.5B, 1.5B, 3B Middle-layer mlp.down_proj: layers 12, 14, and 18, respectively
Qwen3.5 single matrix Qwen3.5-27B Layer 31 mlp.down_proj
GPT-2 MLP replacement GPT-2 Small Layer 8 standard MLP, comprising mlp.c_fc and mlp.c_proj
GPT-2 full-model replacement GPT-2 Small attn.c_attn, attn.c_proj, mlp.c_fc, and mlp.c_proj in all 12 blocks
Zero-data GPT-2 Small Layer 8 mlp.c_proj, matching the GPT-2 single-matrix experiment
Qualitative audit GPT-2 Small mlp.c_proj in all 12 blocks; selected bottleneck units are visualized from layers 6, 8, and 10
Table 4: Models and replacement surfaces. Each comparison uses a common surface across the methods shown in that comparison.

B.2 SWD Factorization and Fixed-Support Fine-Tuning

Default inner width is the smaller matrix dimension. Square-factor density is .16 for square targets and .25 for rectangular ones; the other factor receives the remaining nonzero budget. Gram-diagonal rescaling precedes identity/pruned-target initialization. GPT-2 uses forty outer alternations, Qwen eight; each factor update has five ADMM iterations with ridge .01, initial support changes for two iterations and scheduled penalty. Twenty final fixed-support iterations refit one factor. Each calibration-curve point is independently fitted, not a successive checkpoint; Table 5 records exact data/CE settings.

Experiment CE-curve data (tokens) Fixed replacement used for circuit extraction
GPT-2 single matrix 1,024–4.19M s=0.5s{=}0.5: 16,384 tokens; CE delta 0.0008890.000889
s=0.75s{=}0.75: 16,384 tokens; CE delta 0.0082920.008292
Qwen2.5-0.5B 1,024–33.55M s=0.5s{=}0.5: 1,024 tokens; CE delta 0.0012220.001222
s=0.75s{=}0.75: 1.05M tokens; CE delta 0.0050100.005010
Qwen2.5-1.5B 1,024–33.55M s=0.5s{=}0.5: 1,024 tokens; CE delta 0.0007330.000733
s=0.75s{=}0.75: 1.05M tokens; CE delta 0.0012220.001222
Qwen2.5-3B 1,024–33.55M s=0.5s{=}0.5: 2,048 tokens; CE delta 0.0007330.000733
s=0.75s{=}0.75: 1.05M tokens; CE delta 0.0006110.000611
Qwen3.5-27B 1,024–16.78M s=0.5s{=}0.5: 2,048 tokens; CE delta −0.000427-0.000427
s=0.75s{=}0.75: 1.05M tokens; CE delta −0.000448-0.000448
GPT-2 MLP replacement 1,024–33.55M s=0.5s{=}0.5: 16,384 tokens; CE delta 0.0040030.004003
s=0.75s{=}0.75: 16,384 tokens; CE delta 0.0204910.020491
GPT-2 full model 4.19M (factorization) + 16.38M (fine-tuning) Final SWD-FT replacement (20.57M total tokens)
Zero-data SWD 0 Fixed zero-data factors at each evaluated sparsity
Table 5: Data used for SWD replacement-quality results and the fixed SWD replacements used for circuit extraction. CE delta is replacement CE minus dense-model CE. The third column identifies the fixed replacement whose units are subsequently scored and ablated; it does not count the task data used for circuit scoring and evaluation.
Equation
ρinit=min⁡{1,t/(T−3)}3\rho_{\mathrm{init}}=\min\{1,t/(T-3)\}^{3}

Circuit cost.

SWD cost counts nonzeros in a selected column of A plus its row of B. Whole-model finetuning freezes the 26,770,800-entry support and all other pretrained parameters. It uses 4.19M unique tokens replayed for 2,000 steps at batch eight, totaling 16.38M presentations. AdamW peaks at 5×10⁻⁷ with betas (.9,.95), epsilon .1 and decay .001; one-percent warmup precedes linear decay. Global factor-gradient RMS is normalized to one; training uses bfloat16, evaluation float32 and seed zero.

B.3 Unit Scoring and Circuit Selection

Score each candidate under the unpruned replacement by the training-set expectation of the positive part of activation times task-margin gradient. Means are also estimated on this split. One descending ranking defines every top-k prefix; neither rankings nor ablation means are refitted on test data. This common scoring procedure follows attribution patching.

Equations
μi=𝔼𝒙∈𝒟τ,train​[zi​(𝒙)].\mu_{i}=\mathbb{E}_{{\bm{x}}\in\mathcal{D}_{\tau,\mathrm{train}}}\left[z_{i}({\bm{x}})\right].
ai=𝔼𝒙∈𝒟τ,train​[max⁡(0,zi​(𝒙)​∂gτ∂zi​(𝒙))].a_{i}=\mathbb{E}_{{\bm{x}}\in\mathcal{D}_{\tau,\mathrm{train}}}\left[\max\!\left(0,\,z_{i}({\bm{x}})\frac{\partial g_{\tau}}{\partial z_{i}({\bm{x}})}\right)\right].
zi(𝒙)=𝒉ℓ,t(𝒙)𝑨:,iz_{i}({\bm{x}})={\bm{h}}_{\ell,t}({\bm{x}}){\bm{A}}_{:,i}

B.4 Data Sources and Splits

Language-model fitting/evaluation and circuit-task scoring/evaluation are separate datasets with distinct accounting.

B.4.1 Language-Model Data

FineWeb-Edu supplies fitting data and fixed disjoint test data shared within comparisons. “Data used” means calibration tokens for SWD and optimizer-replayed tokens for trained replacements. Weight-only factorization uses no text; this does not remove task-data requirements for circuit testing.

B.4.2 Circuit-Task Data

Public Edge-Pruning and auto-circuit examples supply train/test splits retained in Table 6, including repository revisions. GPT-2 uses GreaterThan, IOI, docstring and gendered pronouns. Qwen omits GreaterThan because numerical answers violate the single-token scoring protocol; 27B examples are additionally filtered for tokenization compatibility. Circuit examples are excluded from replacement-fitting token counts.

Task Public source GPT-2 Qwen2.5 Qwen3.5-27B
Greater-than Edge-Pruning, a918efd8 80,000 / 12,240 Not used Not used
IOI auto-circuit, 8fc5afd8 600 / 200 600 / 200 600 / 200
Docstring auto-circuit, 8fc5afd8 600 / 200 600 / 200 600 / 200
Gendered pronoun Edge-Pruning, a918efd8 3,024 / 378 3,024 / 378 2,887 / 370
Table 6: Public circuit-task sources and train/test example counts. The linked repository revisions fix the source data used in our experiments.

B.5 Baseline Configurations

Baseline data axes compare consumed tokens, not optimizer steps, FLOPs or wall time. The usual maximum pre-pruning CE difference is .001; explicit reference curves remain visible when that match fails.

B.5.1 Transcoder

Transcoders expose sparse hidden features for the same replacement surface. Width, feature sparsity and checkpoint choices determine the plotted configurations.

Circuit cost.

A selected feature costs its nonzero incoming encoder plus outgoing decoder weights.

GPT-2 single-matrix replacement.

GPT-2 layer-8 c_proj uses widths 3,072/12,288, L1 coefficient 5×10⁻⁵ and seed 42. Curves cover 1,024–4.19M training tokens. Both circuit checkpoints use 2.048M tokens, with CE deltas .000529/.000979.

Qwen2.5 down-projection scaling.

Qwen2.5 .5B/1.5B/3B targets layers 12/14/18. Widths remain 3,072/12,288 with the GPT-2 penalty/seed; trajectories reach 33.55M tokens. Circuit checkpoints generally use 25.17M tokens, except 1.5B TC3k at 33.55M. Their CE deltas are .001654/.002032, −.000057/−.000098 and .001516/.001229 respectively.

Qwen3.5-27B down projection.

Qwen3.5-27B layer-31 down projection uses widths 24,576/49,152, penalty 5×10⁻⁵ and seed 42. Circuit checkpoints consume 134.22M/67.11M tokens and have CE deltas .000398/.000695. TC48k is slightly outside the .001 difference tolerance relative to SWD.

GPT-2 MLP replacement.

The complete GPT-2 layer-8 MLP Transcoder has width 24,576, L1 coefficient 1.4×10⁻⁴ and seed 42. Its only circuit checkpoint uses 6.29M tokens with CE delta .003456; Appendix E preserves the full quality trajectory.

B.5.2 VPD-KL and VPD-Recon-CI

VPD-KL and VPD-Recon-CI share the parameter components and learned causal-importance selector. Their fidelity losses differ: final next-token KL versus local elementwise reconstruction MSE, respectively. Selector minimality is separately regularized. VPD-KL remains a quality reference when it cannot match CE.

Equation
ℒfidVPD​-​KL\displaystyle\mathcal{L}_{\mathrm{fid}}^{\mathrm{VPD\text{-}KL}} =𝔼𝒙,t[DKL(pdense,t(⋅∣𝒙)∥pVPD​-​KL,t(⋅∣𝒙))],\displaystyle=\mathbb{E}_{{\bm{x}},t}\!\left[D_{\mathrm{KL}}\!\left(p_{\mathrm{dense},t}(\cdot\mid{\bm{x}})\,\|\,p_{\mathrm{VPD\text{-}KL},t}(\cdot\mid{\bm{x}})\right)\right],
ℒfidVPD​-​Recon​-​CI\displaystyle\mathcal{L}_{\mathrm{fid}}^{\mathrm{VPD\text{-}Recon\text{-}CI}} =𝔼𝒙,t,j​[(y^ℓ,t,jCI​(𝒙)−yℓ,t,j​(𝒙))2].\displaystyle=\mathbb{E}_{{\bm{x}},t,j}\!\left[\left(\widehat{y}^{\mathrm{CI}}_{\ell,t,j}({\bm{x}})-y_{\ell,t,j}({\bm{x}})\right)^{2}\right].

Circuit cost.

Cost counts a selected component’s nonzero read/write weights. The causal-importance predictor is excluded from this structural edge count.

GPT-2 single-matrix replacement.

GPT-2 c_proj VPD-Recon-CI uses 3,072 components, a width-128 selector, seed zero and minimality coefficient 10⁻⁴. Its fixed 4.19M-token checkpoint has CE delta .000873 and is reused for fidelity and circuit evaluations.

Qwen2.5 down-projection scaling.

Qwen2.5 uses 12,288 components and local reconstruction on layers 12/14/18. Circuit checkpoints consume 16.78M/4.19M/16.78M tokens for .5B/1.5B/3B, with CE deltas .000729/.001186/.000906.

Qwen3.5-27B down projection.

The 27B VPD reference uses 12,288 components, width-12 selector, seed zero and minimality coefficient 3×10⁻⁴. Its 200M-token checkpoint has CE delta .001847, outside the stated tolerance relative to SWD; it remains a parameter-decomposition reference.

GPT-2 MLP replacement.

Complete-MLP VPD-Recon-CI uses 3,072 components; its circuit checkpoint consumes 4.19M tokens with CE delta .003522. The native-KL version uses 3,072 components/seed zero and extends to one billion tokens for replacement quality only.

B.5.3 Sparse-Pretrained Reference

The sparse-pretraining baseline is a separately trained GPT-2-shaped model, not a sparse replacement of the Hugging Face checkpoint inherited by SWD.

Circuit cost.

Only 36,864 post-GELU MLP channels across twelve layers are circuit candidates; attention and residual channels are excluded. A channel costs incoming/outgoing nonzeros plus its nonzero input-projection bias. This candidate definition is distinct from the full-body weight budget.

Equation
‖𝑾c​_​fc[:,j]‖0+‖𝑾c​_​proj[j,:]‖0+[bc​_​fc,j≠0].\left\|{\bm{W}}_{\mathrm{c\_fc}}[:,j]\right\|_{0}+\left\|{\bm{W}}_{\mathrm{c\_proj}}[j,:]\right\|_{0}+\mathbf{1}\!\left[b_{\mathrm{c\_fc},j}\neq 0\right].

GPT-2 full-model comparison.

The baseline uses GPT-2 dimensions, context 1,024, tied embedding/head weights, zero dropout and standard GELU, versus gelu_new in SWD’s dense source. Fidelity/circuit runs are separate, identically configured models evaluated after 2.883584B presentations, before their 4.999872512B-token schedule ends. Their CE is approximately 3.4504. At this point the 48 body matrices have 26,770,920 nonzeros, 120 more than SWD; this budget excludes biases, normalization, embeddings and LM head. Including embeddings, the baseline has 39,184,479 nonzero matrix entries.

Optimization and sparsification.

Seed-zero bfloat16 training uses AdamW with base rate 3×10⁻⁴, betas (.9,.95), epsilon .1 and decay .001. Density stays one for the first percent of training, decays exponentially to .2 by eighty percent and then remains fixed. Learning rate combines warmup/linear decay with inverse square-root density. Each update magnitude-projects all trainable matrices to their budgets while retaining an entry per output row; gradients receive global RMS normalization. The exact schedules remain below.

Equations
d⁡(n)={1,n<n0,0.2(n−n0)/(n1−n0),n0≤n<n1,0.2,n≥n1.d(n)=\begin{cases}1,&n<n_{0},\\ 0.2^{(n-n_{0})/(n_{1}-n_{0})},&n_{0}\leq n<n_{1},\\ 0.2,&n\geq n_{1}.\end{cases}
lr⁡(n)=3×10−4​h⁡(n)d⁡(n),h⁡(n)={n/n0,n<n0,1−(n−n0)/(N−n0),n≥n0.\operatorname{lr}(n)=3\times 10^{-4}\frac{h(n)}{\sqrt{d(n)}},\qquad h(n)=\begin{cases}n/n_{0},&n<n_{0},\\ 1-(n-n_{0})/(N-n_{0}),&n\geq n_{0}.\end{cases}

Appendix C Single-Matrix Replacement Results

The appendix retains companion fidelity metrics, unit-count circuits, zero-ablation controls and all model-size sweeps.

C.1 Replacement Quality beyond Cross-Entropy

Local activation reconstruction error and global next-token KL complement CE, distinguishing projection-level fit from propagated behavioral drift.

Metrics.

Relative MSE divides aggregated squared projection-output error by aggregated dense-output energy, rather than averaging per-example ratios. KL uses the dense next-token distribution as reference. SWD reaches low error with little calibration data; learned baselines require much more replay. No point on the extended GPT-2 VPD-KL trajectory through 1.6384B tokens satisfies the CE match. Broken logarithmic axes retain the long training tail.

Equations
relMSE=∥𝒀rep−𝒀dense∥F2∥𝒀dense∥F2.\mathrm{relMSE}=\frac{\lVert{\bm{Y}}_{\mathrm{rep}}-{\bm{Y}}_{\mathrm{dense}}\rVert_{F}^{2}}{\lVert{\bm{Y}}_{\mathrm{dense}}\rVert_{F}^{2}}.
KL=𝔼tKL(ptdense∥ptrep),\mathrm{KL}=\mathbb{E}_{t}\,\mathrm{KL}\!\big(p^{\mathrm{dense}}_{t}\,\|\,p^{\mathrm{rep}}_{t}\big),
Complete original figure with every panel, label and legend.
Figure 13: GPT-2 Small layer 8 mlp.c_proj single-matrix replacement quality for all methods. Left: activation relative MSE (local). Right: KL to the dense model (global). Lower is better; both panels use the same broken-x policy so the short-horizon comparison and the VPD-KL long tail are visually aligned. VPD-KL is a native-objective replacement-quality reference line and is not a circuit baseline.
Complete original figure with every panel, label and legend.
Figure 14: Qwen2.5 0.5B/1.5B/3B single-matrix mlp.down_proj activation relative MSE for all methods (lower is better). The panels use the same broken-axis style as the KL companion: 0.5B and 3B use a broken x-axis after the 16.8M-token shared horizon, and 1.5B also uses a y-axis break to keep the low-error region readable despite the VPD-KL long-tail spike. VPD-KL is included to contextualize replacement quality.
Complete original figure with every panel, label and legend.
Figure 15: Qwen2.5 0.5B/1.5B/3B single-matrix mlp.down_proj KL to the dense model for all methods (lower is better), with the same broken-axis convention as Figure 14.

C.2 GPT-2 Selected-Unit Circuit Results

Figure 16 uses selected units for the same GPT-2 output-projection circuits as Figure 3. Sparsity .5, TC3k/TC12k and VPD-Recon-CI form the matched comparison; sparsity .75 remains an explicitly unmatched control.

Complete original figure with every panel, label and legend.
Figure 16: Companion to Figure 3 with selected units as the cost axis. The x-axis is the target threshold and the y-axis is the minimum selected units needed to reach it (lower-right is better). Left: sufficiency. Right: necessity drop. The same SWD advantage holds on the selected-unit axis.

C.3 Robustness under Zero Ablation

Only the ablated activation value changes from its train mean to zero; checkpoints, scores, rankings, top-k sets and examples stay fixed. SWD’s edge-cost advantage persists on all four GPT-2 tasks. At GreaterThan sufficiency at least one, sparsity .75 uses 12,303 observed edges versus TC3k’s 184,320. This robustness check does not establish invariance to arbitrary intervention protocols.

Complete original figure with every panel, label and legend.
Figure 17: GPT-2 single-matrix circuit results under zero ablation, using minimum active edges as the cost. The checkpoints, unit rankings, and top-kk sets are identical to those in Figure 3; only the ablation value changes from the training-set mean to zero. Left: sufficiency. Right: necessity drop.
Complete original figure with every panel, label and legend.
Figure 18: The same fixed-ranking zero-ablation comparison using minimum selected units as the cost. Left: sufficiency. Right: necessity drop.

C.4 Qwen2.5 Circuit Results

Tables/figures preserve all Qwen2.5 model checkpoints and both cost axes. VPD-Recon-CI uses 12,288 components; VPD-KL appears only in fidelity plots. Unit-count companions separate the number of selected paths from their connectivity.

Complete original figure with every panel, label and legend.
Figure 19: Qwen2.5 single-matrix replacement quality across 0.5B, 1.5B, and 3B models. SWD reaches low CE delta with substantially fewer tokens than the trained baselines.
Model and target module Checkpoint for circuit extraction CE delta
Qwen2.5 0.5B, layer 12 mlp.down_proj 16.78M tokens used 0.000729
Qwen2.5 1.5B, layer 14 mlp.down_proj 4.19M tokens used 0.001186
Qwen2.5 3B, layer 18 mlp.down_proj 16.78M tokens used 0.000906
Table 7: Qwen2.5 VPD-Recon-CI settings. All rows use C=12,288C=12{,}288 components and replace one mlp.down_proj.
Complete original figure with every panel, label and legend.
Figure 20: Qwen2.5 0.5B single-matrix circuit results, with active edges as the cost axis (lower-right is better). Left: sufficiency. Right: necessity drop.
Complete original figure with every panel, label and legend.
Figure 21: Qwen2.5 1.5B single-matrix circuit results, with active edges as the cost axis (lower-right is better). Left: sufficiency. Right: necessity drop. The 3B results are in the main text (Figure 5).
Complete original figure with every panel, label and legend.
Figure 22: Qwen2.5 0.5B single-matrix circuit results, with selected units as the cost axis (lower-right is better). Companion to Figure 20. Left: sufficiency. Right: necessity drop.
Complete original figure with every panel, label and legend.
Figure 23: Qwen2.5 1.5B single-matrix circuit results, with selected units as the cost axis (lower-right is better). Companion to Figure 21. Left: sufficiency. Right: necessity drop.
Complete original figure with every panel, label and legend.
Figure 24: Qwen2.5 3B single-matrix circuit results, with selected units as the cost axis (lower-right is better). Companion to the 3B active-edge results in the main text (Figure 5). Left: sufficiency. Right: necessity drop.

C.5 Qwen3.5-27B Circuit Results

All 27B comparisons use layer-31 down projection and the same three task families. Relative to SWD at sparsity .5, CE differences are .000022 for sparsity .75, .000824 for TC24k, .001121 for TC48k and .002274 for VPD-Recon-CI. Thus the latter two are outside the .001 tolerance, with VPD kept as a contextual reference rather than a fully matched comparison.

Method Setting Checkpoint for circuit extraction CE delta
SWD s=0.5s=0.5 2,048 calibration tokens −0.000427-0.000427
SWD s=0.75s=0.75 1.05M calibration tokens −0.000448-0.000448
Transcoder width 24,576 134.22M training tokens 0.0003980.000398
Transcoder width 49,152 67.11M training tokens 0.0006950.000695
VPD-Recon-CI C=12,288C=12{,}288 200.00M training tokens 0.0018470.001847
Table 8: Qwen3.5-27B checkpoints used for circuit extraction. CE delta is measured against the same dense model and held-out language-model data.
Complete original figure with every panel, label and legend.
Figure 25: Companion to Figure 6 with selected units as the cost axis. The x-axis is the target threshold and the y-axis is the minimum selected units needed to reach it (lower-right is better). Left: sufficiency. Right: necessity drop.

Appendix D Full-Model Replacement

The quality trajectory starts after 4.19M calibration tokens at CE 3.90. Fixed-support finetuning reaches CE 3.44 at 20.6M total presentations, versus sparse pretraining’s 3.45 at 2.884B. Supports remain unchanged throughout SWD finetuning; its curve adds replay tokens to the calibration count.

Complete original figure with every panel, label and legend.
Figure 26: Held-out CE of GPT-2 Small full-model replacements versus data used in tokens (lower is better). The SWD-FT curve begins at the 4.19M-token factorization point and adds fixed-support fine-tuning tokens thereafter; the sparse-pretraining curve counts tokens consumed during sparse pretraining. The dotted horizontal line marks the dense model. At an approximately matched active-weight budget, factorizing and fine-tuning an existing dense checkpoint brings SWD-FT to CE 3.443.44 after 20.620.6M tokens, less than 1%1\% of the 2.8842.884B sparse-pretraining tokens used to reach CE 3.453.45.

Appendix E MLP Replacement

Whole-MLP SWD factors both projections while retaining the original intervening GELU and biases. Standard Transcoder replaces the complete input/output map. Quality sweeps share a 16.78M-token horizon, with VPD-KL extended to one billion and still excluded from circuits. The matched circuit checkpoints are SWD .5, TC24k at 6.29M tokens and VPD-Recon-CI at 4.19M. Outcomes are more mixed than single-matrix replacement: SWD is competitive in edge cost and often necessity, while Transcoder is competitive in several sufficiency/unit-count regions.

Equation
MLP^​(𝒉)=ϕ⁡(𝒉​𝑨fc​𝑩fc+𝒃fc)​𝑨proj​𝑩proj+𝒃proj.\widehat{\mathrm{MLP}}({\bm{h}})=\phi\!\left({\bm{h}}{\bm{A}}_{\mathrm{fc}}{\bm{B}}_{\mathrm{fc}}+{\bm{b}}_{\mathrm{fc}}\right){\bm{A}}_{\mathrm{proj}}{\bm{B}}_{\mathrm{proj}}+{\bm{b}}_{\mathrm{proj}}.
Method Replacement configuration Checkpoint for circuit extraction
SWD Sequential factorization of c_fc and c_proj; s∈{0.5,0.75}s\in\{0.5,0.75\} 16,384 tokens used
Transcoder Standard MLP Transcoder (Appendix B.5.1) 6.29M tokens used; CE delta 0.003456
VPD-Recon-CI Local MLP-reconstruction objective; C=3,072C=3{,}072 4.19M tokens used
VPD-KL Native KL-logit objective; C=3,072C=3{,}072, seed 0 Replacement only
Table 9: Fixed configurations for the GPT-2 Small layer 8 MLP-replacement comparison. SWD tokens are calibration tokens; Transcoder and VPD-Recon-CI tokens are optimizer replay. VPD-KL is excluded from circuit extraction because its replacement fidelity is not matched.
Complete original figure with every panel, label and legend.
Figure 27: GPT-2 Small layer 8 MLP replacement quality for all methods (lower is better in every panel). Left: CE delta versus the dense model. Middle: KL to the dense model. Right: activation relative MSE at the target module. The broken x-axis keeps the 1K–16.8M shared comparison region readable and shows only the VPD-KL tail beyond the break. The native-objective VPD-KL curve serves as a replacement-quality reference and is excluded from the circuit-pruning figures; it begins at its first 4,096-token checkpoint.
Complete original figure with every panel, label and legend.
Figure 28: GPT-2 layer 8 MLP-replacement circuit results for the setting in Table 9. For each curve, the x-axis is the target threshold and the y-axis is the minimum active edges needed to reach it (lower-right is better). Left: sufficiency. Right: necessity drop. Transcoder and VPD-Recon-CI use the fixed CE-matched 6.29M- and 4.19M-token checkpoints, respectively.
Complete original figure with every panel, label and legend.
Figure 29: Companion to Figure 28 with selected units as the cost axis (lower-right is better), separating selected-unit count from read/write connectivity.

Appendix F Exact-Dense Reparameterization Controls

Exact dense controls share the attribution, task data, mean ablation and curve construction with SWD. Their higher active-edge costs cannot be attributed to worse replacement fidelity.

Control configurations.

SVD uses all 768 singular directions of the 3,072×768 target; Random-B uses an orthogonal Gaussian-QR basis with A=WQᵀ. Ten random seeds provide a mean and one-standard-deviation band. Both cost k(3,072+768) edges for k units. SVD CE delta is 1.58×10⁻⁶; Random-B ranges −2.48×10⁻⁶ to 3.40×10⁻⁶. Dense controls can compete in IOI unit count and low docstring necessity, but SWD has the more consistent edge-cost advantage.

Equations
𝑾eff=𝑨svd​𝑩svd,𝑨svd=𝑼​𝚺,𝑩svd=𝑽⊤,{\bm{W}}_{\mathrm{eff}}={\bm{A}}_{\mathrm{svd}}{\bm{B}}_{\mathrm{svd}},\qquad{\bm{A}}_{\mathrm{svd}}={\bm{U}}{\bm{\Sigma}},\quad{\bm{B}}_{\mathrm{svd}}={\bm{V}}^{\top},
𝑩rand=𝑸,𝑨rand=𝑾eff​𝑸⊤,𝑾eff=𝑨rand​𝑩rand,{\bm{B}}_{\mathrm{rand}}={\bm{Q}},\qquad{\bm{A}}_{\mathrm{rand}}={\bm{W}}_{\mathrm{eff}}{\bm{Q}}^{\top},\qquad{\bm{W}}_{\mathrm{eff}}={\bm{A}}_{\mathrm{rand}}{\bm{B}}_{\mathrm{rand}},
Complete original figure with every panel, label and legend.
Figure 30: Companion to Figure 8 with selected units as the cost axis (lower-right is better). Left: sufficiency. Right: necessity drop. Dense controls become competitive with SWD at specific low thresholds or tasks, but do not generally reproduce its low-cost circuit.

Appendix G Zero-Data Analysis

Weight-only fitting isolates calibration’s contribution while retaining the solver, factor dimensions and budget schedule.

G.1 Factorization and Evaluation Setting

Set the activation Gram matrix to identity and factor GPT-2 layer-8 c_proj across sparsities .125 through .875. Neither supports nor surviving values use calibration activations. Compare each product with calibrated SWD at the same sparsity.

Equation
𝑯⊤​𝑯←𝑰,min𝑨,𝑩⁡‖𝑾−𝑨​𝑩‖F2s.t.‖𝑨‖0+‖𝑩‖0≤K.{\bm{H}}^{\top}{\bm{H}}\leftarrow{\bm{I}},\qquad\min_{{\bm{A}},{\bm{B}}}\|{\bm{W}}-{\bm{A}}{\bm{B}}\|_{F}^{2}\quad\mathrm{s.t.}\quad\|{\bm{A}}\|_{0}+\|{\bm{B}}\|_{0}\leq K.

G.2 Circuit Results

Train-split task data still determines rankings and mean-ablation values; test evaluates the frozen prefixes. At sparsity .375, docstring reaches .962 sufficiency with three units and GreaterThan .986 with 256. Gendered pronouns have fewer strong low-cost outcomes. These results support usable weight-derived paths without claiming calibration-free task evaluation.

Complete original figure with every panel, label and legend.
Figure 31: Zero-data identity-Gram SWD task-circuit results compared with fixed Transcoder reference settings. Each curve gives the minimum cost reaching the plotted threshold (lower is better). Top: sufficiency. Bottom: necessity drop. Left: active edges. Right: selected units.

Appendix H GreaterThan Case Study: Semantic Audit

The semantic audit ranks all 9,208 valid output-projection units using GreaterThan attribution, then examines independent WikiText-2 activation contexts. GPT-5.5 summarizes recurring patterns in each unit’s twenty strongest passages from 2,048 tokens; these labels do not affect ranking. The six plotted units have global ranks 32–129, with four quantitative and two punctuation/species-name hypotheses. Tables 10–11 preserve ranks and representative contexts. This small model-labeled sample does not establish general monosemanticity.

Layer Bottleneck unit Layer order Global rank Score Semantic hypothesis
6 c380 1 108 10.2 measure/unit
6 c723 2 129 9.7 punctuation/function
6 c558 3 152 9.3 mixed lexical
8 c264 1 76 11.3 quantity
8 c511 2 84 11.0 measure/place
8 c728 3 123 9.8 event/time
10 c757 1 32 14.4 numeric/year
10 c627 2 63 11.8 species/name
10 c481 3 85 10.9 numeric/year
Table 10: Top-three GreaterThan bottleneck units in each displayed layer. Layer order is used only for visualization; global rank is over all 9,2089{,}208 candidates. Semantic hypotheses are task-independent and do not affect ranking.
Bottleneck unit Semantic hypothesis Top-activating excerpts
L6 c380 measure/unit “carapace length of 80–85 millimetres”; “usually 23–38 cm”; “a carapace length of 15 mm”.
L6 c723 punctuation/function Repeated comma-token activations in clauses such as “higher than that of H. americanus, and the European species” and around numeric punctuation.
L8 c264 quantity “estimated that only 1 larva”; “not normally deeper than 50 m (160 ft)”; “published in 1758”.
L8 c511 measure/place “48 kilometres or 30 miles northwest of Gothenburg”; “for up to 12 months”; “body length up to 60 centimetres”.
L10 c757 numeric/year “published in 1758”; “Homarus Weber, 1795”; “Fabricius, 1775”.
L10 c627 species/name Repeated activations on species-name fragments in “H. americanus”, “H. gammarus”, and “Homarus americanus”.
Table 11: Representative task-independent top-activation contexts for the six bottleneck units rendered in Figure 10. Bold marks the token or phrase most relevant to each tentative semantic hypothesis.

Appendix I Targeted Editing of a Single SWD Bottleneck Unit

The editing study follows localized weight-editing and LoRA controls. All methods modify the same original dense layer-8 matrix, avoiding replacement-fidelity confounds. A sparsity-.75 factorization fitted on 131,072 WikiText-2 tokens supplies c205’s read vector; a rank-one write update is calibrated to a requested local answer-direction shift. The unit/target were screened from eight triples and top-24 candidate units. Random c540 is sampled from a middle activation range; the dense oracle directly uses the target activation. Rank-4 LoRA trains separately for each requested margin shift for 180 AdamW steps. At local shift three, the realized final margin changes only .215797; seven non-target prompts give KL 4.02375×10⁻⁵ and a separate 504-token sample 1.96415×10⁻⁵. The oracle is strongest; c205 improves the measured tradeoff over random/LoRA on this selected case.

Equations
𝒅=𝒖ans−𝒖foil,𝒅¯=𝒅∥𝒅∥2,{\bm{d}}={\bm{u}}_{\mathrm{ans}}-{\bm{u}}_{\mathrm{foil}},\qquad\overline{{\bm{d}}}=\frac{{\bm{d}}}{\lVert{\bm{d}}\rVert_{2}},
𝑾c​(δ)=𝑾+αc​(δ)​𝒂c​𝒅¯⊤,αc​(δ)=δ(𝒉⋆​𝒂c)​∥𝒅∥2.{\bm{W}}_{c}(\delta)={\bm{W}}+\alpha_{c}(\delta){\bm{a}}_{c}\overline{{\bm{d}}}^{\top},\qquad\alpha_{c}(\delta)=\frac{\delta}{({\bm{h}}^{\star}{\bm{a}}_{c})\lVert{\bm{d}}\rVert_{2}}.
𝒉⋆​(𝑾c​(δ)−𝑾)​𝒅=δ.{\bm{h}}^{\star}\bigl({\bm{W}}_{c}(\delta)-{\bm{W}}\bigr){\bm{d}}=\delta.
𝑩c,:′=𝑩c,:+αc(δ)𝒅¯⊤⟹𝑨𝑩′−𝑨𝑩=αc(δ)𝑨:,c𝒅¯⊤.{\bm{B}}^{\prime}_{c,:}={\bm{B}}_{c,:}+\alpha_{c}(\delta)\overline{{\bm{d}}}^{\top}\quad\Longrightarrow\quad{\bm{A}}{\bm{B}}^{\prime}-{\bm{A}}{\bm{B}}=\alpha_{c}(\delta){\bm{A}}_{:,c}\overline{{\bm{d}}}^{\top}.
δ∈{−2,−1,−0.5,0,0.5,1,2,3}.\delta\in\{-2,-1,-0.5,0,0.5,1,2,3\}.
𝒓dense=𝒉⋆⁣⊤∥𝒉⋆∥2,𝑾dense​(δ)=𝑾+αdense​(δ)​𝒓dense​𝒅¯⊤,{\bm{r}}_{\mathrm{dense}}=\frac{{\bm{h}}^{\star\top}}{\lVert{\bm{h}}^{\star}\rVert_{2}},\qquad{\bm{W}}_{\mathrm{dense}}(\delta)={\bm{W}}+\alpha_{\mathrm{dense}}(\delta){\bm{r}}_{\mathrm{dense}}\overline{{\bm{d}}}^{\top},
𝒂c=𝑨:,c∈ℝ3072{\bm{a}}_{c}={\bm{A}}_{:,c}\in\mathbb{R}^{3072}

Appendix J Attention Bottleneck Units: From Static QK Geometry to Prompt-Local Effects

Attention analysis combines QK-circuit geometry, sparse weight-space decomposition and feature-level attribution. It first checks reconstruction, then screens static pairs and tests prompt-specific contributions causally.

J.1 Attention Factorization Fidelity

Across GPT-2’s twelve layers, packed Q/K/V reconstruction relative SSE stays below four percent. Attention output-projection error reaches roughly six percent in middle layers. The replay case therefore uses the better-preserved Q/K/V factors and independently verifies its attention pattern.

J.2 Static QK Candidate Screen

A head’s bilinear query/key score expands into pair contributions: activation product times a static write-direction inner product scaled by √64. In each of 144 layer/head combinations, retain sixteen largest-norm query and key units, inspect all 256 pairs, and advance the five largest static interactions to prompt replay. Static geometry alone does not establish a contribution on a particular input.

Equations
𝒒th\displaystyle{\bm{q}}_{t}^{h} =∑aza​(𝒉t)​𝒃a,hQ,\displaystyle=\sum_{a}z_{a}({\bm{h}}_{t}){\bm{b}}^{\mathrm{Q}}_{a,h},
𝒌uh\displaystyle{\bm{k}}_{u}^{h} =∑bzb​(𝒉u)​𝒃b,hK.\displaystyle=\sum_{b}z_{b}({\bm{h}}_{u}){\bm{b}}^{\mathrm{K}}_{b,h}.
scoret​uh\displaystyle\operatorname{score}_{tu}^{h} =∑a,bza​(𝒉t)​zb​(𝒉u)​γa​bh,\displaystyle=\sum_{a,b}z_{a}({\bm{h}}_{t})z_{b}({\bm{h}}_{u})\gamma_{ab}^{h},
γa​bh\displaystyle\gamma_{ab}^{h} =⟨𝒃a,hQ,𝒃b,hK⟩64.\displaystyle=\frac{\langle{\bm{b}}^{\mathrm{Q}}_{a,h},{\bm{b}}^{\mathrm{K}}_{b,h}\rangle}{\sqrt{64}}.
Complete original figure with every panel, label and legend.
Figure 32: Attention reconstruction and static QK candidate screen. Top: activation-weighted relative reconstruction SSE for the packed Q/K/V and attention output matrices across GPT-2 Small. Bottom: the largest absolute static QK coefficient among the retained 16×1616\times 16 bottleneck-unit pairs in each layer and head.

J.3 Prompt Replay and Bottleneck-Unit Intervention

Five pairs per layer/head on three prompts yield 2,160 evaluations. The selected q266×k64 case is layer 9/head 3 on “When Mary gave John the book, he thanked”. Reconstructed attention has dense-reference KL .0405; the pair’s signed mean contribution is +4.26, suppressing the first key while favoring later keys. Removing q266 from the reconstructed query slice redirects attention to the first token, with KL 2.496 and maximum change .775. This verifies one selected prompt-local effect, not all attention units.

Equations
12​layers×12​heads×5​pairs×3​prompts=2,16012\ \text{layers}\times 12\ \text{heads}\times 5\ \text{pairs}\times 3\ \text{prompts}=2{,}160
z266​(𝒉t)​z64​(𝒉u)​γ266,64hz_{266}({\bm{h}}_{t})z_{64}({\bm{h}}_{u})\gamma_{266,64}^{h}
Complete original figure with every panel, label and legend.
Figure 33: Token-level pre-softmax score contribution of q266×\timesk64 in layer 9, head 3. Each cell shows z266​(𝒉t)​z64​(𝒉u)​γ266,64hz_{266}({\bm{h}}_{t})z_{64}({\bm{h}}_{u})\gamma_{266,64}^{h} for one causally valid query–key pair; the signed mean is +4.26+4.26.

Appendix K Interaction Patterns among MLP Bottleneck Units

The diagnostics distinguish correlation of additive post-GELU output paths from contextual non-additivity of pre-GELU input paths. Superposition and nonlinear interactions motivate checking combinations rather than assuming independent units.

K.1 c_proj : Linear Contributions Are Usually Weakly Correlated

For 768 layer-8 c_proj units at sparsity .75, 2,048 held-out tokens estimate read correlations, write-direction cosines and full residual-contribution correlations. The 96 highest-energy units appear in matrix views; distributions cover all units. The respective 95th-percentile magnitudes are .207, .071 and .0049, so overlapping reads/writes rarely coincide strongly. A small tail recurs around c373/c447/c481/c266; c373–c481 reaches .345 because opposed reads and opposed writes cancel their signs.

Equation
Var(zi)∥𝑩i,:∥22.\operatorname{Var}(z_{i})\lVert{\bm{B}}_{i,:}\rVert_{2}^{2}.
Complete original figure with every panel, label and legend.
Figure 34: Selection and overlap structure for layer-8 mlp.c_proj. Top: read variance versus squared write norm; the 96 bottleneck units with highest output-variance energy are highlighted. Bottom: read correlation, write-direction cosine, and complete contribution correlation for those units. Read and write structure is visible separately but largely disappears when combined into full residual contributions.
Complete original figure with every panel, label and legend.
Figure 35: Linear bottleneck-unit dependence in layer-8 mlp.c_proj. Top: off-diagonal pair distributions over all 768 units. Bottom: the 30 pairs with largest absolute contribution correlation. Most contribution correlations concentrate near zero, while a small tail is organized around recurring units; c373–c481 reaches 0.3450.345.

K.2 c_fc and GELU: Contextual Pair Interactions

The c_fc diagnostic replays 8,192 held-out tokens with lower-layer replacements installed. Select 128 high-energy paths from 1,536 units, yielding 8,128 pairs. A contextual second finite difference across GELU is projected through the original output matrix and normalized by individual marginal residual energies. Interaction ratios average .093, median .089, 95th percentile .132 and maximum .221; 2,130 pairs are at least .10. Recurring high-energy units and isolated strong pairs form a moderate interaction field, contrasting with mostly weak output-path correlation.

Equations
Ri=𝔼t[zt,i2]‖𝑩i,:𝑾proj‖22,R_{i}=\mathbb{E}_{t}[z_{t,i}^{2}]\left\lVert{\bm{B}}_{i,:}{\bm{W}}_{\mathrm{proj}}\right\rVert_{2}^{2},
𝒉t=𝒃fc+∑a𝒄t,a{\bm{h}}_{t}={\bm{b}}_{\mathrm{fc}}+\sum_{a}{\bm{c}}_{t,a}
𝚫t,i​j=\displaystyle\bm{\Delta}_{t,ij}={} GELU⁡(𝒉t)−GELU⁡(𝒉t−𝒄t,i)−GELU⁡(𝒉t−𝒄t,j)\displaystyle\operatorname{GELU}({\bm{h}}_{t})-\operatorname{GELU}({\bm{h}}_{t}-{\bm{c}}_{t,i})-\operatorname{GELU}({\bm{h}}_{t}-{\bm{c}}_{t,j})
+GELU⁡(𝒉t−𝒄t,i−𝒄t,j).\displaystyle+\operatorname{GELU}({\bm{h}}_{t}-{\bm{c}}_{t,i}-{\bm{c}}_{t,j}).
𝒎t,i=[GELU⁡(𝒉t)−GELU⁡(𝒉t−𝒄t,i)]​𝑾proj,Ei=∑t∥𝒎t,i∥22.{\bm{m}}_{t,i}=\left[\operatorname{GELU}({\bm{h}}_{t})-\operatorname{GELU}({\bm{h}}_{t}-{\bm{c}}_{t,i})\right]{\bm{W}}_{\mathrm{proj}},\qquad E_{i}=\sum_{t}\lVert{\bm{m}}_{t,i}\rVert_{2}^{2}.
Ri​j=(∑t∥𝒒t,i​j∥22)1/2(Ei​Ej)1/4.R_{ij}=\frac{\left(\sum_{t}\lVert{\bm{q}}_{t,ij}\rVert_{2}^{2}\right)^{1/2}}{(E_{i}E_{j})^{1/4}}.
Complete original figure with every panel, label and legend.
Figure 36: Contextual GELU-mediated interactions among 128 high-energy layer-8 mlp.c_fc bottleneck units. Top: distribution of residual-stream interaction ratios over all 8,128 unordered pairs. Bottom: the same ratios ordered by pre-GELU downstream energy. Interactions are broadly distributed, with stronger values concentrated around several recurring units and a smaller set of pair-specific cells.
Complete original figure with every panel, label and legend.
Figure 37: The 30 strongest contextual GELU-mediated pairs, ranked by residual-stream interaction ratio. Units c431, c928, c1140, and c1498 recur throughout the upper tail. The leading c431–c1498 pair reaches R=0.221R=0.221; its interaction residual RMS is 2.0962.096, compared with marginal residual RMS values 11.12711.127 and 8.0898.089 for the two units.
Equation
𝒒t,i​j=𝚫t,i​j​𝑾proj{\bm{q}}_{t,ij}=\bm{\Delta}_{t,ij}{\bm{W}}_{\mathrm{proj}}
Concept demonstration

Open image ↗