Overview
A sparse reconstruction toy model can memorize individual training examples in superposition or learn reusable input features. Varying the dataset size moves between these regimes and produces a peak in test error between them: data double descent. The result offers a concrete mechanism in this controlled model, not a general explanation of memorization in language models. [1]
Experiment Setup
Inputs contain 10,000 nonnegative features, each independently zero with probability 0.999, then are normalized to unit length. A tied-weight linear downprojection and ReLU reconstruction minimize mean squared error on a finite dataset. Training uses 50,000 full-batch AdamW updates, 2,500 warmup steps, a peak learning rate of 0.001, cosine decay, and usually weight decay 0.01. Long optimization and seed selection matter especially with two hidden dimensions; results become less optimization-sensitive above three dimensions.
Equations
"Datapoint Features" vs "Generalizing Features"
With abundant data, sparse input features organize into geometric patterns such as pentagons in a two-dimensional hidden space. With few examples, the feature vectors look disordered while the hidden representations of training examples form clean polygons. Memorizing a small set of complete examples can be easier than reconstructing thousands of reusable features. Treating these examples as special-purpose features is a useful interpretation of this toy result.


How Do Models Change with Dataset Size?
A dimensionality statistic measures how much hidden-space capacity is assigned to a feature or training example. At small dataset sizes, examples occupy approximately equal shares of a polygon; at large sizes, about five reusable features dominate while others are largely ignored. Between these limits the geometry is less understood. Vector and parameter norms rise during increasingly demanding memorization, then fall as the solution changes toward generalizing features. Exact polygon memorization arguments assume ideal precision and suitably normalized, orthogonal examples.
Equation


Superposition Double Descent
Averaging four independently sampled datasets at each size reveals a test-loss peak near the transition between memorized-example and reusable-feature representations. No label noise is added: reconstruction has identical inputs and targets, but a narrow linear bottleneck still loses information. This connects the observed double descent to a change in representational strategy, without proving that the same mechanism explains other settings. [9]
The Effect of m on Double Descent
Sweeping both dataset size and hidden dimension finds regions where adding data or increasing width worsens test error. Four models are averaged per configuration, and widths above three reduce optimization instability. The phenomenon depends on regularization and training duration: stronger weight decay can remove the peak, while longer training can increase it.

Conclusion
The central finding is that both generalization and memorization can use superposition, with reusable features and individual examples as different represented objects. Open problems include mixed solutions, the poorly understood transition regime, and whether comparable mechanisms occur in realistic networks.
Comments & Replications
The following comments and replications are contributions by external researchers and the original authors. They provide supporting experiments and unresolved differences rather than additional controlled results from one unified study.
Replication
Adam Jermyn reproduces the qualitative three-regime geometry, but the transition sizes and peak errors differ substantially. His generalizing regime begins near 20,000 examples versus about 5,000 in the original, and intermediate losses can reach roughly 5,000 versus about 1.01. Parameter-norm trends agree qualitatively. The authors acknowledge the unresolved likely setup difference; the replication supports the regime transition, not agreement on its precise location or magnitude.


Replication
Marius Hobbhahn reports similar double descent after reducing the feature count and changing sparsity by factors of ten. He also observes it under different loss functions and with an added hidden ReLU. These external experiments broaden the toy settings exhibiting the pattern.
Extension: What controls the generalization scale?
Jermyn tests what determines the onset of generalization. Changing weight decay has little effect on that onset, whereas rescaling dataset size by feature frequency approximately aligns the transition: a feature has appeared roughly ten times when generalization begins. Alignment is less clean at larger hidden dimensions, so this is a suggestive empirical rule rather than a complete theory.




Extension: Repeated Data Points
Repeating individual examples sufficiently often produces mixed representations: a memorized example can displace one of the reusable pentagon features, and several repeated examples can displace all five. A rough comparison of reconstruction benefits predicts a repetition threshold proportional to dataset size divided by feature count. Noisy experiments broadly support that scaling in the large-data regime. Training trajectories can temporarily memorize and then discard an example, or briefly learn and then forget reusable features.





Data Dimensionality of MNIST
An exploratory 512-hidden-unit MNIST model uses maximal data dimensionality, optimizing over projection directions to detect unusually special examples. A few training examples have values almost one hundred times typical values and look atypical, suggesting special-case representations. This is not conclusive evidence of memorization or a validated anomaly detector. Hobbhahn reports a qualitative replication and further practical sensitivity of the statistic.
Equation

Optimization Failures in 2D
Some two-dimensional low-data runs fail to form a single circle. The authors interpret the result as two groups on different-radius circles that do not merge, with large angular gaps needed to separate inner-circle points. This is an optimization failure mode of the setup.

Author Contributions
Tom Henighan led the experiments and drafting, with collaborators contributing the original hypothesis, optimization, geometry analysis, replication, interpretation, and illustrations.
Acknowledgements
The authors acknowledge external readers for comments on early drafts and their colleagues for supporting the work.
Effect of Weight Decay and Epochs on Double Descent
For four-dimensional models, increasing weight decay progressively suppresses the double-descent peak and a value of 1.0 removes it in the shown experiment. With weight decay fixed at 0.01, increasing the number of epochs makes the peak larger. All runs use a 5% learning-rate warmup followed by cosine decay.

