Literature
Display
← All papers

Privileged Bases in the Transformer Residual Stream

Nelson Elhage · Robert Lasenby · Christopher Olah

· Transformer Circuits Thread ↗

Interpretability3 min text · CondensedOriginal paper ↗
Contents

At a glance

Residual-stream coordinates are often treated as arbitrary, yet trained transformers develop unusually large activations along particular coordinate axes. Experiments on a 200-million-parameter model show that this alignment survives removing LayerNorm’s basis dependence and is difficult to explain by floating-point precision. The authors suspect Adam’s coordinatewise gradient normalization, but do not directly establish it as the cause.

The longer story

Large residual-stream outliers raise two questions: what information they represent, and why they align with the stored coordinate axes. In a representation with no preferred orientation, a feature’s magnitude should usually spread across coordinates. Consistent axis-aligned outliers therefore indicate symmetry breaking somewhere in the architecture or training procedure. This investigation follows Dettmers’ outlier observations and the Transformer Circuits account of residual representations. [1] [2]

The experiments

The study uses Anthropic’s 200-million-parameter transformer. Outliers are frequent enough at this scale to compare controlled architectural and training changes, although their prevalence can differ with model size.

Measuring Outliers

An activation is counted as an outlier when its absolute value exceeds six. Across a batch of 128 sequences of 1,024 tokens, typical layers contain roughly 20–60 coordinates that exhibit an outlier, out of 1,280 hidden dimensions. The qualitative result persists across other thresholds.

Activation Kurtosis

The study averages per-token activation kurtosis across the batch. Randomly oriented high-dimensional feature directions motivate a Gaussian-like coordinate distribution with kurtosis near three; substantially heavier tails indicate basis alignment under this argument. Baseline activations have much higher kurtosis, while applying a fixed random orthogonal rotation brings it close to three. A value near three does not prove that no basis is privileged.

Hypotheses for the basis-alignment

The experiments separate architectural normalization, numerical precision, and optimizer dependence as possible sources of coordinate alignment.

LayerNorm

LayerNorm subtracts a coordinate mean and applies learned per-channel scaling. The experiment replaces it with root-mean-square normalization and one learned scalar, making the normalization invariant to orthogonal basis changes. Model performance remains similar and heavy tails persist, so these LayerNorm operations do not explain the observed outliers. [3]

Equation

RMSNorm(xi)=α⋅xiRMS(x)   where   RMS(x)=1d∑i=1dxi2\text{RMSNorm}(x_i) = \alpha\cdot\frac{x_i}{\text{RMS}(\bf{x})} ~~~\text{where}~~~\text{RMS}(\bf{x})=\sqrt{\frac{1}{d}\sum_{i=1}^dx_i^2}

Finite-Precision

The precision hypothesis proposes that separating features of different magnitudes into different coordinates avoids losing small terms during floating-point addition. The following tests probe both forward execution and training. [4]

Verifying the model is basis-independent

For the modified normalization, the authors rotate every residual-reading and residual-writing matrix consistently. The resulting loss changes are tiny compared with the baseline test loss of about 3.02 nats and with ordinary initialization variation. This supports effective rotation invariance of the forward computation at the tested precision.

Optimizing in an arbitrary basis

Training-time experiments insert either one shared rotation or independent rotations around each residual read and write. Both variants achieve similar loss to baseline, while residual-stream kurtosis approaches three. With a shared rotation, heavy tails reappear in the internal computation basis; with independent rotations they are much weaker. Alignment therefore follows the shared computation coordinates more than the residual communication coordinates. These results weigh against precision as the main explanation, without making that exclusion mathematically conclusive.

Conclusion

After the normalization and precision tests, Adam remains a strong candidate because its separate moment estimates for each parameter coordinate privilege the stored basis. This is a provisional inference, not an optimizer-ablation result. Suggested follow-ups are training with SGD and expressing Adam’s moments in a basis different from the weights.

Loss curves

Training curves differ slightly early on but converge after roughly a billion tokens to differences the authors regard as noise. The figure supplies the loss comparison for the rotation and normalization experiments.

Author Contributions

Nelson Elhage designed and ran the experiments and proposed kurtosis as a diagnostic. Chris Olah developed the accompanying theoretical argument and contributed text; Robert Lasenby helped edit and analyze the work. Shan Carter produced the illustrations with Elhage and Olah.

Concept demonstration

Open image ↗