Literature
Display
← All papers

A Toy Model of Mechanistic (Un)Faithfulness

Chris Olah

Published · Transformer Circuits Thread ↗

Interpretability3 min text · CondensedOriginal paper ↗
Contents

Overview

A sparse replacement can match a model’s outputs while implementing a different algorithm. This exploratory note demonstrates that failure in an absolute-value toy task and tests Jacobian matching as a possible remedy. The result is a warning about mechanistic faithfulness, not evidence that all transcoders fail.

Background: Sleepwalking into Mechanistic Faithfulness

Transcoders predict a layer’s output from its input, making feature interactions easier to trace [1]. Unlike a dictionary that only describes representations, they claim to describe computation. Interpretable features and low MSE therefore do not guarantee the original mechanism, although matching each intermediate layer constrains possible discrepancies.

The Absolute Value Transcoder Problem and Repeated Data

The ground-truth function independently takes each input coordinate’s absolute value, exactly implementable by two ReLU units per coordinate. A sufficiently large transcoder normally recovers these units. The experiment uses a tanh sparsity penalty to approximate L0 regularization more closely and reduce duplicate features.

Equation

yi=ReLU(xi)+ReLU(−xi)y_i = \text{ReLU}(x_i) + \text{ReLU}(-x_i)

Adding a Repeated Data Point

Inputs are sparse independent uniform coordinates, except that a chosen vector with three leading ones is repeated with controlled probability. The underlying absolute-value function remains unchanged.

Equation

p = [1,1,1,0,0,0,0,0,0,0,0,…]p ~=~ [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, \ldots]

Datapoint Features Emerge

Repetition makes a dedicated feature that recognizes and reproduces the special vector economical under sparsity pressure. The transcoder memorizes a pattern that the true function does not single out, directly demonstrating an alternative mechanism.

Equation

y += p⋅ReLU(λp⋅x−b)y ~+=~ p \cdot \text{ReLU}(\lambda p \cdot x - b)

When Do Datapoint Features Form?

Datapoint features require repeated examples and an incentive for sparse activity. The experimental scan finds their emergence governed approximately by the product of repetition frequency and sparsity strength.

Aside: How do we Detect a Datapoint Feature?

The memorization measure counts a feature’s excess contribution to the repeated vector beyond its contribution along the constituent coordinate directions. Summing across features supplies a useful but crude diagnostic.

Equations

y∣fi(x)=Wdownifi(x)y|_{f_i}(x) = W_\text{down}^i f_i(x)

m(fi,p)=ReLU(y∣fi(p)⋅p^−maxj<3y∣fi(p)⋅ejm(f_i, p) = \text{ReLU}(y|_{f_i}(p) \cdot \hat{p} - \text{max}_{j<3} y|_{f_i}(p) \cdot e_j

Investigating Datapoint Features

Feature contributions are compared with a library of expected coordinate-wise circuits and a memorization circuit. This allows the experiment to distinguish recovered mechanisms from superficially accurate outputs.

No Repeat Baseline

Without repeated data, learned features almost exactly implement the expected positive and negative coordinate ReLU circuits, leaving only a small unexplained residual.

Aside: How does this visualization work?

Each learned and ideal circuit is represented by its output contributions across the same sampled inputs. A sparse decomposition matches learned contribution vectors to ideal circuits and reports the remaining unexplained component.

Light Repetition

Repeating the special vector on just 5% of training examples produces a memorization feature. Other features rotate to avoid activating on that vector, so the changed mechanism is distributed across the dictionary.

Jacobian Regularization: A Potential Solution?

The original function has a diagonal Jacobian: each output depends locally only on its corresponding input. Memorization creates cross-coordinate dependencies. A layer’s input-dependent Jacobian captures aspects of its weights and activation gates while avoiding some arbitrary parameterizations.

Jacobian Matching

Add a penalty for the difference between true and transcoder Jacobians. Random-vector Jacobian products estimate the full squared matrix difference without explicitly constructing both matrices. In the toy task, this substantially reduces the memorization score.

Equations

∣∣JMLP−JTranscoder∣∣F2||J_\text{MLP} - J_\text{Transcoder}||_F^2

EvK∣∣JMLPv−JTranscoderv∣∣22{\Large E}_v K||J_\text{MLP}v - J_\text{Transcoder}v||_2^2

Jacobian Matching Can Solve Mechanistic Faithfulness (In This Example)

Jacobian regularization removes the dedicated memorization feature in the tested example. Some coordinate features remain rotated and leave a residual, so the experiment demonstrates improvement rather than exact general recovery.

Can the Transcoder Cheat?

A transcoder could manipulate gradients with large weights and rarely active, strongly negative biases. Tanh regularization reduces this loophole relative to ordinary L1. Penalizing individual features’ Jacobian contributions is proposed as another remedy but is not established here.

Superposition and Jacobian Matching

Exact Jacobian matching may itself be undesirable when superposition adds small interference directions. The intended target may be the meaningful large singular values rather than every tail component; this remains a hypothesis.

Conclusion

Faithfulness asks whether a replacement captures the true mechanism; simplicity asks whether it describes that mechanism compactly. Both matter, but faithfulness is essential for trusting causal explanations. The toy counterexample establishes a real failure mode and a promising corrective objective, without proving scalability to full language models.

Acknowledgments

The author thanks colleagues for feedback and discussion; the original source records the acknowledgments.

Concept demonstration

Open image ↗