Literature
Display
← All papers

Sparse mixtures of linear transforms

Jack Lindsey · Brian Chen · Adam Pearce · Sasha Hydrie · Thomas Conerly

Published · Transformer Circuits Thread ↗

Interpretability3 min text · CondensedOriginal paper ↗
Contents

Overview

Transcoders can fragment a compact computation into many narrow lookup-table features. This preliminary report instead represents an MLP as a sparse MOLT, separating computational transforms from the features they act on. It reports improved reconstruction and local mechanistic faithfulness, while leaving interpretability and compositional benefits partly unresolved [2].

Implementation

Each transform has an activation gate and a low-rank linear map, factored into two matrices. The model sums active transforms applied to the residual stream and learns to reconstruct the original MLP output. A sparsity penalty on gate activity is scaled by each transform’s Frobenius norm. Transform ranks are tunable.

Equation

f(x)=∑t[ϕ(et⋅x−bt)∗(UtVtx)]f(x) = \sum_t [\phi(\mathbf{e_t} \cdot \mathbf{x} - b_t) * (U_t V_t \mathbf{x})]

Related Work

Mixture of Decoders also uses conditional linear maps but shares parameters between full-rank transforms [1]. MOLTs independently learn low-rank maps. Skip transcoders instead add a single always-available linear path to ordinary transcoder features [7]. Comparative advantages remain unestablished.

Intuition

A ReLU MLP is linear after fixing its active neurons, suggesting conditional transforms as natural computational units. Geometric operations such as rotating a numerical representation can implement many cases with one transform. Pairing a representation feature with a transform yields an implicit feature; chains of transforms could compactly express many compositions, but this is not yet demonstrated at scale.

Results

The following experiments separately assess reconstruction, local causal faithfulness, and how interpretable the transforms are. They use different models for some measurements and should not be read as a single unified evaluation.

Optimal allocation of ranks

The best tested allocation uses ranks 32, 64, 128, 256, and 512, with twice as many transforms whenever rank halves. This mixed-rank design outperforms uniform ranks at matched parameter count on Claude 3.5 Haiku; the broader allocation space remains unexplored.

ML performance

At matched parameters and sparsity, MOLTs reconstruct Haiku’s middle-layer MLP better than transcoders. In the reported sweep, the smallest MOLT runs outperform transcoder runs using 1024 times the FLOPs. On a separate 18-layer model, MOLT Jacobians also correlate better with the true layer’s Jacobian; this faithfulness test was not yet performed on Haiku. Apparent transcoder scaling saturation may partly reflect tuning.

Transform interpretability

Activation examples suggest interpretable gating conditions, from token patterns to abstract contexts. Higher-rank transforms activate more often without obviously losing gate interpretability. Their actual computational roles are harder to recover: raw feature-to-feature weights suffer interference. Attribution graphs offer more useful context.

Integrating MOLTs into attribution graphs

MOLTs can annotate the operations carried by an attribution-graph edge, while separate SAEs supply the graph’s representation features.

Method

Train dictionaries on residual streams, then decompose each feature-to-feature attribution into MOLT-mediated and attention-mediated contributions. Label each graph edge with its strongest transforms or attention heads, and separately trace the inputs activating each transform. Cross-layer dictionaries can improve the resulting graphs [8].

Qualitative findings

Examples link a heat concept to a Spanish output through a Spanish-context transform, and combine numerical inputs through addition-related transforms. Other transforms reflect generic words or duplicate information already present in their input features. These mixed qualitative cases do not establish consistently simple mechanisms.

Conclusion

MOLTs are promising replacements for MLPs, with stronger tested efficiency and local faithfulness. Composing features and transforms might explain dictionary reconstruction errors and rare concepts. However, the graphs are more cumbersome, some operations remain opaque, and strict superiority over transcoders is not established.

Concept demonstration

Open image ↗