Literature
Display
← All papers

Insights on Crosscoder Model Diffing

Siddharth Mishra-Sharma · Trenton Bricken · Jack Lindsey · Adam Jermyn · Jonathan Marcus · Kelley Rivoire · Christopher Olah · Thomas Henighan

Published · Transformer Circuits Thread ↗

Interpretability2 min text · CondensedOriginal paper ↗
Contents

Overview

This preliminary report studies crosscoders for comparing internal representations across related language models. Its results are exploratory; feature differences are not yet a complete account of behavioral differences.

Introduction and summary

Crosscoders jointly reconstruct activations from two models with a shared sparse dictionary. Decoder norms distinguish features shared across models from features concentrated in one. Real-model exclusive features are often denser and less interpretable than shared features, motivating a change to the training objective. The baseline penalizes each model’s decoder norm separately to encourage exclusivity [1]; stage-wise dictionary fine-tuning offers a complementary approach [2].

Equations

L=Ex[∣∣x−x^∣∣2+λ∑ifi(x)∣∣Wdec,i∣∣]L = E_x[||x - \hat{x}||^2 + \lambda \sum_i f_i(x) ||W_{dec,i}||]

L=Ex[∑m∣∣xm−x^m∣∣2+λ∑ifi(x)∑m∣∣Wdec,im∣∣]L = E_x[\sum_m ||x^m - \hat{x}^m||^2 + \lambda \sum_i f_i(x) \sum_m ||W^m_{dec,i}||]

Empirical observations when diffing real models

Exclusive features activate roughly an order of magnitude more frequently than shared features in the illustrated models and often combine unrelated contexts. Both compared models have unexpectedly similar numbers and kinds of exclusive features. Shared features with differently oriented decoders can also reveal representational changes, typically in narrower contexts.

A toy model of crosscoder diffing

Toy data reproduce the density gap when dictionary capacity is scarce: a shared feature reduces reconstruction error in both models, whereas an exclusive feature must explain more information to compete for a slot. With abundant capacity, the gap disappears. The toy model does not inherently reproduce symmetric exclusivity. Changing data sampling only partly affects symmetry; altered co-activation patterns offer another plausible explanation, still unproven.

A small variation can render interpretable model-exclusive features

Reserve 10,000 of roughly 250,000 features as explicitly shared, using tied decoders and only 0.1–0.2 times the normal sparsity penalty. These dense shared features absorb common variance, allowing remaining exclusive features to become sparser and more interpretable. Tests on a base versus helpful assistant and an assistant versus a sleeper agent isolate expected chatbot, tool-use, and deployment-trigger features. About 90% of exclusive features in the latter comparison relate to the expected sleeper behavior [8].

Equation

L=Ej[∑m∣∣xjm−x^jm∣∣2+λs∑i∈Sfi(xj)∣∣Wdec,i∣∣+λf∑i∈Ffi(xj)∑m∣∣Wdec,im∣∣]L = E_j[\sum_m ||x^m_j - \hat{x}^m_j||^2 + \lambda_s \sum_{i \in S} f_i(x_j)||W_{dec,i}|| + \lambda_f \sum_{i \in F} f_i(x_j) \sum_m ||W^m_{dec,i}||]

Conclusions and outlook

The modified loss improves the interpretability of exclusive features in the tested comparisons. Symmetry remains incompletely explained, and detecting a representation correlated with changed behavior does not establish the full causal mechanism. Broader models and causal validation are needed before using these differences as reliable safety evidence.

Concept demonstration

Open image ↗