Literature
Display
← All papers

Scaling Laws for Neural Language Models

Jared Kaplan · Sam McCandlish · Tom Henighan · Tom B. Brown · Benjamin Chess · Rewon Child · Scott Gray · Alec Radford · Jeffrey Wu · Dario Amodei

Preprint posted · arXiv (preprint) ↗

Language models10 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition. Treat training like an engineering learning curve: how much does prediction improve when you buy more parameters, examples or computation? Cross-entropy measures how surprised the model is by real next tokens; lower loss means it assigns them higher probabilities. A power law predicts fractional improvements rather than a fixed gain in accuracy.

Reading the math. N counts model parameters, D counts training tokens and C measures computation. In L(N) ∝ N−α, doubling N multiplies the fitted loss by 2−α, while other bottlenecks are absent. A small α means diminishing returns. Its value is measured from experiments, not a universal constant of language.

Across the tested Transformer language models, cross-entropy loss follows approximate power laws in model size, data, and training compute. The paper combines these fits to estimate efficient training allocations and argues for larger models stopped before convergence. These are empirical laws for the studied data, models, and optimization regime; the paper itself identifies extrapolation limits and incompatible far-future predictions.

1 Introduction

The study measures predictive loss rather than an omnibus intelligence score. It varies architecture, model size, available text, and training budget over wide ranges, seeking equations that can forecast held-out language-model performance. Figure 1 summarizes the separately model-, data-, and compute-limited regimes.

Original figure
Figure 1:Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute11 1 Here we display predicted compute when using a sufficiently small batch size. See Figure13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

1.1 Summary

The following findings summarize the measured trends before the methods, fits, and caveats are developed.

Performance depends strongly on scale, weakly on model shape:

Non-embedding parameter count, data size, and training compute explain much more variation than moderate changes in Transformer depth or width at fixed size.

Smooth power laws:

Approximate power laws fit broad observed ranges when the other resources are sufficient. Their continuation to arbitrarily low loss is impossible because language has residual uncertainty.

Universality of overfitting:

Jointly increasing model and dataset size controls overfitting. Under the fitted finite-data relationship, preserving the relative penalty requires dataset size to grow approximately as parameter count to the power 0.74.

Universality of training:

After early training transients, learning curves admit a common parameterized form. Extrapolation is useful within that regime, not a guarantee about initial optimization or very distant scales.

Transfer improves with test performance:

On the tested alternative text distributions, loss largely tracks in-distribution loss with an offset. The evidence concerns predictive text likelihood, not every downstream transfer task.

Sample efficiency:

Larger models can reach a fixed loss with fewer data examples and fewer update steps. This sample efficiency does not mean each step is cheaper; the later analysis trades those costs against model size.

Refer to caption
Figure 2:We show a series of language model training runs, with models ranging in size from10310^{3} to10910^{9} parameters (excluding embeddings).
Original figure
Figure 3:As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps. We illustrate this for a billion-fold increase in compute. For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required.

Convergence is inefficient:

Within the paper’s fitted training regime, compute-efficient runs use larger models and stop before full convergence. The reported data-growth exponent is a conclusion of this historical fit, not a universal training prescription.

Optimal batch size:

Critical batch size is related primarily to achieved loss and gradient noise scale. Beyond it, extra parallel examples buy diminishing reductions in update count.

1.2 Summary of Scaling Laws

Separate fits describe loss as a function of parameters, data, and batch-adjusted compute. Joint models then combine finite-data penalties or unfinished optimization with the parameter-limited loss. The equations retain their original constants and exponents; different fitting subsets yield slightly different values, and tokenization-dependent constants have no universal meaning.

Equations 1.1–1.4
L⁡(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013​(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\penalty\ \penalty\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\penalty\ \text{(non-embedding parameters)} (1.1)
L⁡(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013​(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\penalty\ \penalty\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\penalty\ \text{(tokens)} (1.2)
L⁡(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108​(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\penalty\ \penalty\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\penalty\ \text{(PF-days)} (1.3)
Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
Original figure
Refer to caption
Figure 4: Left: The early-stopped test lossL⁡(N,D)L(N,D) varies predictably with the dataset sizeDD and model sizeNN according to Equation (1.5). Right: After an initial transient period, learning curves for all model sizesNN can be fit with Equation (1.6), which is parameterized in terms ofSminS_{\rm min}, the number of steps when training at large batch size (details in Section5.1).
Equations 1.5–1.8
L⁡(N,D)=[(NcN)αNαD+DcD]αDL(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L⁡(N,S)=(NcN)αN+(ScSmin​(S))αSL(N,S)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S_{\rm min}(S)}\right)^{\alpha_{S}} (1.6)
N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅SN\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)
αCmin=1/(1/αS+1/αB+1/αN)\alpha_{C}^{\rm min}=1/\left(1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}\right) (1.8)

1.3 Notation

Loss is measured in nats per token. N excludes token and positional embeddings; D counts tokens; B is batch size in tokens; S counts updates. Training compute is approximated by 6NBS. The adjusted quantities C_min and S_min describe different limiting batch regimes and should not be confused with directly observed FLOPs or steps.

2 Background and Methods

Decoder-only Transformers predict tokens in 1,024-token contexts using WebText2 and a 50,257-token byte-pair vocabulary. Held-out cross-entropy is the main metric. LSTMs and parameter-sharing Universal Transformers provide architectural comparisons.

2.1 Parameter and Compute Scaling of Transformers

Parameter and FLOP estimates omit embeddings, biases, nonlinearities, and smaller terms. The approximation of six non-embedding parameter operations per training token is appropriate when context-dependent attention work is relatively small. Very long contexts can violate that assumption.

Operation Parameters FLOPs per Token
Embed (nvocab+nctx)​dmodel\left(n_{{\rm vocab}}+n_{{\rm ctx}}\right)d_{{\rm model}} 4​dmodel4d_{{\rm model}}
Attention: QKV nlayer​dmodel​3​dattnn_{{\rm layer}}d_{{\rm model}}3d_{{\rm attn}} 2​nlayer​dmodel​3​dattn2n_{{\rm layer}}d_{{\rm model}}3d_{{\rm attn}}
Attention: Mask — 2​nlayer​nctx​dattn2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm attn}}
Attention: Project nlayer​dattn​dmodeln_{{\rm layer}}d_{{\rm attn}}d_{{\rm model}} 2​nlayer​dattn​dembd2n_{{\rm layer}}d_{{\rm attn}}d_{{\rm embd}}
Feedforward nlayer​2​dmodel​dffn_{{\rm layer}}2d_{{\rm model}}d_{{\rm ff}} 2​nlayer​2​dmodel​dff2n_{{\rm layer}}2d_{{\rm model}}d_{{\rm ff}}
De-embed — 2​dmodel​nvocab2d_{{\rm model}}n_{{\rm vocab}}
Total (Non-Embedding) N=2​dmodel​nlayer​(2​dattn+dff)N=2d_{{\rm model}}n_{{\rm layer}}\left(2d_{{\rm attn}}+d_{{\rm ff}}\right) Cforward=2​N+2​nlayer​nctx​dattnC_{\mathrm{forward}}=2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm attn}}
Table 1:Parameter counts and compute (forward pass) estimates for a Transformer model. Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted.
Equations 2.1–2.2
N\displaystyle N ≈2​dmodel​nlayer​(2​dattn+dff)\displaystyle\approx 2d_{{\rm model}}n_{{\rm layer}}\left(2d_{{\rm attn}}+d_{{\rm ff}}\right)
=12nlayerdmodel2 with the standarddattn=dff/4=dmodel\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1)
Cforward≈2​N+2​nlayer​nctx​dmodelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2)

2.2 Training Procedures

Most runs use Adam, 250,000 updates, and batches of 512 sequences, with warmup and cosine decay. Models above one billion parameters use Adafactor because of memory constraints. Learning schedules are studied separately, but optimizer and hyperparameter coverage remain limited.

2.3 Datasets

WebText2 extends the Reddit-linked WebText collection through October 2018, producing 20.3 million documents and about 22.9 billion tokens. The study holds out 660 million tokens and evaluates additional book, web, and Wikipedia distributions. Reddit-link selection is a sampling heuristic, not representative coverage of all language.

3 Empirical Results and Basic Power Laws

Experiments vary non-embedding model sizes from 768 to 1.5 billion parameters, datasets from 22 million to 23 billion tokens, model shape, context, and batch size. The fitted domain is defined by these experiments.

3.1 Approximate Transformer Shape and Hyperparameter Independence

At fixed non-embedding parameter count, broad but non-extreme changes in depth, width, heads, and feed-forward dimension alter loss by only a few percent. This is approximate architectural insensitivity within the investigated family, not equivalence between arbitrary architectures.

Original figure
Figure 5:Performance depends very mildly on model shape when the total number of non-embedding parametersNN is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit toL⁡(N)L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an(nlayer,dmodel)=(6,4288)(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the(48,1600)(48,1600) model used in[RWC+19].

3.2 Performance with Non-Embedding Parameter Count N

Excluding embedding parameters aligns models of different shapes into a cleaner loss–size trend. One-layer models and extreme aspect ratios deviate. Near-converged training on the full corpus supplies the parameter-scaling fit; incomplete convergence must be distinguished from insufficient capacity.

Original figure
Original figure
Figure 6: Left: When we include embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters. Right: When we exclude embedding parameters, the performance of models with different depths converge to a single trend. Only models with fewer than 2 layers or with extreme depth-to-width ratios deviate significantly from the trend.
Equation 3.1
L⁡(N)≈(NcN)αNL(N)\approx\left(\frac{N_{c}}{N}\right)^{\alpha_{N}} (3.1)

3.2.1 Comparing to LSTMs and Universal Transformers

LSTMs compare well on early context tokens but benefit less from later context than Transformers. Universal Transformers reuse parameters and gain slightly per parameter, while losing slightly per unit of compute. The appropriate comparison changes with the resource being constrained.

Original figure
Figure 7:

3.2.2 Generalization Among Data Distributions

Other text distributions improve roughly in parallel with WebText2 loss, across both model sizes and training progress. In the measured settings, held-out training-distribution performance is more predictive of this transfer than depth or proximity to convergence.

Original figure
Original figure
Figure 8: Left: Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution. Right: Generalization performance depends only on training distribution performance, and not on the phase of training. We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains.

3.3 Performance with Dataset Size and Compute

Data-limited fits use fixed dataset subsets with early stopping. Compute-limited comparisons search model training curves at equal estimated budgets; the initial fixed-batch analysis is subsequently adjusted for batch inefficiency. These are distinct experimental regimes rather than interchangeable curves.

Equations 3.2–3.3
L⁡(D)≈(DcD)αDL(D)\approx\left(\frac{D_{c}}{D}\right)^{\alpha_{D}} (3.2)
L⁡(C)≈(CcC)αCL(C)\approx\left(\frac{C_{c}}{C}\right)^{\alpha_{C}} (3.3)

4 Charting the Infinite Data Limit and Overfitting

The joint model-and-data fit estimates how finite data limits otherwise larger models. Early stopping and fixed regularization are part of the setup.

4.1 Proposed L ⁡ ( N , D ) Equation

The proposed formula matches separate infinite-data and infinite-model limits and permits loss rescaling under tokenization changes. Its analytic expansion in inverse dataset size is more speculative. Empirical fit quality is the principal justification; it is not derived as a general theorem about neural networks.

Original figure
Original figure
Figure 9: The early-stopped test lossL⁡(N,D)L(N,D) depends predictably on the dataset sizeDD and model sizeNN according to Equation (1.5). Left: For largeDD, performance is a straight power law inNN. For a smaller fixedDD, performance stops improving asNN increases and the model begins to overfit. (The reverse is also true, see Figure4.) Right: The extent of overfitting depends predominantly on the ratioNαNαD/DN^{\frac{\alpha_{N}}{\alpha_{D}}}/D, as predicted in equation (4.3). The line is our fit to that equation.
Equation 4.1
L⁡(N,D)=[(NcN)αNαD+DcD]αDL(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (4.1)

4.2 Results

The formula fits most tested sizes but is weaker on the smallest dataset, where an epoch is only forty updates. A practical overfitting threshold is inferred from approximately 0.02 loss variation between seeds. Dropout was fixed rather than optimized across all dataset and model sizes, limiting universality.

Parameter αN\alpha_{N} αD\alpha_{D} NcN_{c} DcD_{c}
Value 0.0760.076 0.1030.103 6.4×10136.4\times 10^{13} 1.8×10131.8\times 10^{13}
Table 2:Fits toL⁡(N,D)L(N,D)
Equations 4.2–4.4
δ​L​(N,D)≡L⁡(N,D)L⁡(N,∞)−1\delta L(N,D)\equiv\frac{L(N,D)}{L(N,\infty)}-1 (4.2)
δ​L≈(1+(NNc)αNαD​DcD)αD−1\delta L\approx\left(1+\left(\frac{N}{N_{c}}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}\frac{D_{c}}{D}\right)^{\alpha_{D}}-1 (4.3)
D≳(5×103)​N0.74D\gtrsim(5\times 10^{3})\,N^{0.74} (4.4)

5 Scaling Laws with Model Size and Training Time

A batch-adjusted training-step variable helps compare learning curves that were not trained at their own optimal batch sizes.

5.1 Adjustment for Training at B crit ​ ( L )

Small batches minimize examples and compute needed for a target loss; very large batches minimize serial updates. The critical batch lies near the tradeoff. The paper fits its dependence on loss and uses that empirical relation to estimate minimum-compute and minimum-step curves from fixed-batch runs.

Original figure
Figure 10:The critical batch sizeBcritB_{\rm crit} follows a power law in the loss as performance increase, and does not depend directly on the model size. We find that the critical batch size approximately doubles for every13%13\% decrease in loss.BcritB_{\rm crit} is measured empirically from the data shown in Figure18, but it is also roughly predicted by the gradient noise scale, as in[MKAT18].
Equations 5.1–5.5
(SSmin−1)​(EEmin−1)=1\left(\frac{S}{S_{\rm min}}-1\right)\left(\frac{E}{E_{\rm min}}-1\right)=1 (5.1)
Bcrit​(L)≡EminSminB_{\rm crit}(L)\equiv\frac{E_{\rm min}}{S_{\rm min}} (5.2)
Bcrit​(L)≈B∗L1/αBB_{\rm crit}(L)\approx\frac{B_{*}}{L^{1/\alpha_{B}}} (5.3)
Smin​(S)≡S1+Bcrit​(L)/B(minimum steps, at​B≫Bcrit)S_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit}) (5.4)
Cmin​(C)≡C1+B/Bcrit​(L)(minimum compute, at​B≪Bcrit)C_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit}) (5.5)

5.2 Results for L ⁡ ( N , S min ) and Performance with Model Size and Compute

After warmup, loss is fitted as a parameter-limited term plus a decaying optimization term. At fixed compute, making the model larger eventually leaves too few updates, yielding an intermediate best size. Fit deviations at early steps are expected because the power-law regime has not yet begun.

Refer to caption
Refer to caption
Figure 11: When we hold either total compute or number of training steps fixed, performance followsL⁡(N,S)L(N,S) from Equation (5.6). Each value of compute budget has an associated optimal model size that maximizes performance. Mediocre fits at smallSS are unsurprising, as the power-law equation for the learning curves breaks down very early in training.
Equation 5.6
L⁡(N,Smin)=(NcN)αN+(ScSmin)αSL(N,S_{\rm min})=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S_{\rm min}}\right)^{\alpha_{S}} (5.6)
Parameter αN\alpha_{N} αS\alpha_{S} NcN_{c} ScS_{c}
Value 0.0770.077 0.760.76 6.5×10136.5\times 10^{13} 2.1×1032.1\times 10^{3}
Table 3:Fits toL⁡(N,S)L(N,S)

5.3 Lower Bound on Early Stopping Step

Comparing finite-data and infinite-data loss suggests a lower bound on when early stopping should occur. The derivation treats their curves as similar until overfitting matters; the actual finite-data curve can improve more slowly. It is therefore a bound and rough estimate, not an exact stopping rule.

Equation 5.7
Sstop​(N,D)≳Sc[L⁡(N,D)−L⁡(N,∞)]1/αSS_{\rm stop}(N,D)\gtrsim\frac{S_{c}}{\left[L(N,D)-L(N,\infty)\right]^{1/\alpha_{S}}} (5.7)
Original figure

Figure 12:Left: Given a fixed compute budget, a particular model size is optimal, though somewhat larger or smaller models can be trained with minimal additional compute.Right: Models larger than the compute-efficient size require fewer steps to train, allowing for potentially faster training if sufficient additional parallelism is possible. Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve, after initial transient effects.

6 Optimal Allocation of the Compute Budget

The compute frontier corrects the fixed-batch experiments using measured critical-batch behavior, then balances parameter count against data processed and serial training time. Adjusted estimates should be distinguished from direct runs at every predicted optimal setting.

Original figure
Figure 13: When adjusting performance to simulate training far below the critical batch size, we find a somewhat altered power law forL⁡(Cmin)L(C_{\rm min}) when compared with the fully empirical results. The conspicuous lump at10−510^{-5} PF-days marks the transition from 1-layer to 2-layer networks; we exclude 1-layer networks in the power-law fits. It is theL⁡(Cmin)L(C_{\rm min}) trend that we expect to provide a reliable extrapolation for larger compute.

6.1 Optimal Performance and Allocations

The empirical frontier gives approximate exponents 0.73 for model size, 0.24 for batch size, and 0.03 for minimum steps as compute grows. Thus most extra budget goes into larger models within this fit. The tiny step exponent is uncertain enough to be compatible with little or no growth.

Equations 6.1–6.2
N⁡(Cmin)∝(Cmin)0.73.N(C_{\rm min})\propto(C_{\rm min})^{0.73}. (6.1)
Smin∝(Cmin)0.03,S_{\rm min}\propto(C_{\rm min})^{0.03}, (6.2)
Original figure
Original figure
Figure 14: Left: Each value of the compute budgetCminC_{\rm min} has an associated optimal model sizeNN. Optimal model size grows very rapidly withCminC_{\rm min}, increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x. Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes.

6.2 Predictions from L ⁡ ( N , S min )

Minimizing the fitted learning-curve equation predicts loss and model-size exponents close to those measured on the adjusted frontier. This internal agreement supports the framework over its experimental range; it does not independently verify arbitrary extrapolations.

Equations 6.3–6.5
L⁡(Cmin)=(CcminCmin)αCminL(C_{\rm min})=\left(\frac{C_{c}^{\rm min}}{C_{\rm min}}\right)^{\alpha_{C}^{\rm min}} (6.3)
αCmin≡11/αS+1/αB+1/αN≈0.054\alpha_{C}^{\rm min}\equiv\frac{1}{1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}}\approx 0.054 (6.4)
N⁡(Cmin)∝(Cmin)αCmin/αN≈(Cmin)0.71N(C_{\rm min})\propto(C_{\rm min})^{\alpha_{C}^{\rm min}/\alpha_{N}}\approx(C_{\rm min})^{0.71} (6.5)

6.3 Contradictions and a Conjecture

Far beyond the measured regime, the predicted data requirement conflicts with the predicted compute-efficient loss. At least one relation must change before their intersection. The suggested trillion-parameter/trillion-token scale and natural-language entropy interpretation are explicitly uncertain conjectures, sensitive to fitted exponents, rather than established limits.

Original figure
Figure 15:Far beyond the model sizes we study empirically, we find a contradiction between our equations forL⁡(Cmin)L(C_{\rm min}) andL⁡(D)L(D) due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.
Equations 6.6–6.8
D∝N0.74∝Cmin0.54D\propto N^{0.74}\propto C_{\rm min}^{0.54} (6.6)
D⁡(Cmin)=2​Cmin6​N​(Cmin)≈(4×1010​tokens)​(Cmin/PF​-​Day)0.26D(C_{\rm min})=\frac{2C_{\rm min}}{6N(C_{\rm min})}\approx\left(4\times 10^{10}\ {\rm tokens}\right)(C_{\rm min}/\mathrm{PF}{\text{-}}\mathrm{Day})^{0.26} (6.7)
C∗∼104​PF​-​DaysN∗∼1012​parameters,D∗∼1012​tokens,L∗∼1.7​nats/tokenC^{*}\sim 10^{4}\penalty\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\penalty\ \text{parameters},\quad D^{*}\sim 10^{12}\penalty\ \text{tokens},\quad L^{*}\sim 1.7\penalty\ \text{nats/token} (6.8)

7 Related Work

The paper connects its results to prior learning curves, dataset scaling, efficient architectures, overparameterization, and noisy optimization models. Earlier studies found different data–model relationships, reinforcing that training setup and the quantity held fixed matter.

8 Discussion

The main value is a predictive empirical framework for allocation within a studied regime. The authors leave open which trends generalize to other modalities and whether lower text loss translates into meaningful task improvements. Diminishing returns, hardware parallelism, and omitted inference costs also matter when applying the findings.

Acknowledgements

The authors acknowledge colleagues for discussions and comments on drafts.

Appendix A Summary of Power Laws

The six tables collect the fitted regimes, constants, and compute-efficient allocations. They retain which variables are fixed, unlimited, or optimized; constants differ among separate and joint fits and depend on tokenization.

Parameters Data Compute Batch Size Equation
NN ∞\infty ∞\infty Fixed L⁡(N)=(Nc/N)αNL\left(N\right)=\left(N_{{\rm c}}/N\right)^{\alpha_{N}}
∞\infty DD Early Stop Fixed L⁡(D)=(Dc/D)αDL\left(D\right)=\left(D_{{\rm c}}/D\right)^{\alpha_{D}}
Optimal ∞\infty CC Fixed L⁡(C)=(Cc/C)αCL\left(C\right)=\left(C_{{\rm c}}/C\right)^{\alpha_{C}} (naive)
NoptN_{{\rm opt}} DoptD_{{\rm opt}} CminC_{{\rm min}} B≪BcritB\ll B_{{\rm crit}} L⁡(Cmin)=(Ccmin/Cmin)αCminL\left(C_{{\rm min}}\right)=\left(C_{{\rm c}}^{{\rm min}}/C_{{\rm min}}\right)^{\alpha_{C}^{{\rm min}}}
NN DD Early Stop Fixed L⁡(N,D)=[(NcN)αNαD+DcD]αDL\left(N,D\right)=\left[\left(\frac{N_{{\rm c}}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}
NN ∞\infty SS steps BB L⁡(N,S)=(NcN)αN+(ScSmin​(S,B))αSL\left(N,S\right)=\left(\frac{N_{{\rm c}}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{{\rm c}}}{S_{{\rm min}}\left(S,B\right)}\right)^{\alpha_{S}}
Table 4:
Power Law Scale (tokenization-dependent)
αN=0.076\alpha_{N}=0.076 Nc=8.8×1013N_{{\rm c}}=8.8\times 10^{13} params (non-embed)
αD=0.095\alpha_{D}=0.095 Dc=5.4×1013D_{{\rm c}}=5.4\times 10^{13} tokens
αC=0.057\alpha_{C}=0.057 Cc=1.6×107C_{{\rm c}}=1.6\times 10^{7} PF-days
αCmin=0.050\alpha_{C}^{{\rm min}}=0.050 Ccmin=3.1×108C_{{\rm c}}^{{\rm min}}=3.1\times 10^{8} PF-days
αB=0.21\alpha_{B}=0.21 B∗=2.1×108B_{\ast}=2.1\times 10^{8} tokens
αS=0.76\alpha_{S}=0.76 Sc=2.1×103S_{{\rm c}}=2.1\times 10^{3} steps
Table 5:
Compute-Efficient Value Power Law Scale
Nopt=Ne⋅CminpNN_{{\rm opt}}=N_{e}\cdot C_{{\rm min}}^{p_{N}} pN=0.73p_{N}=0.73 Ne=1.3⋅109N_{e}=1.3\cdot 10^{9} params
B≪Bcrit=B∗L1/αB=Be​CminpBB\ll B_{{\rm crit}}=\frac{B_{\ast}}{L^{1/\alpha_{B}}}=B_{e}C_{{\rm min}}^{p_{B}} pB=0.24p_{B}=0.24 Be=2.0⋅106B_{e}=2.0\cdot 10^{6} tokens
Smin=Se⋅CminpSS_{{\rm min}}=S_{e}\cdot C_{{\rm min}}^{p_{S}} (lower bound) pS=0.03p_{S}=0.03 Se=5.4⋅103S_{e}=5.4\cdot 10^{3} steps
Dopt=De⋅CminpDD_{{\rm opt}}=D_{e}\cdot C_{{\rm min}}^{p_{D}} (1 epoch) pD=0.27p_{D}=0.27 De=2⋅1010D_{e}=2\cdot 10^{10} tokens
Table 6:

Appendix B Empirical Model of Compute-Efficient Frontier

This appendix derives an efficient frontier from the fitted learning curve and batch-size relation. Its notation suppresses the batch-adjustment label, so symbols should be interpreted with that convention.

B.1 Defining Equations

Substitute compute divided by model size and batch size for the update count, then differentiate loss with respect to model size at fixed compute. The resulting optimum balances the capacity-limited and unfinished-training terms. The retained equations show the assumptions and algebra.

Equations B.1–B.4
L⁡(N,S)=(NcN)αN+(ScS)αS.L\left(N,S\right)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S}\right)^{\alpha_{S}}. (B.1)
B⁡(L)=B∗L1/αB.B\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}}. (B.2)
L⁡(N,C)=(NcN)αN+(6​B∗​Sc​NL1/αB​C)αS.L\left(N,C\right)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}}. (B.3)
0\displaystyle 0 =∂L∂N|C\displaystyle=\frac{\partial L}{\partial N}\big|_{C}
=−αNN​(NcN)αN+αSN​(6​B∗​Sc​NL1/αB​C)αS​(1−5​NL​∂L∂N|C)\displaystyle=-\frac{\alpha_{N}}{N}\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\frac{\alpha_{S}}{N}\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}}\left(1-5\frac{N}{L}\cancel{\frac{\partial L}{\partial N}\big|_{C}}\right)
⟹αNαS​(NcN)αN\displaystyle\implies\frac{\alpha_{N}}{\alpha_{S}}\left(\frac{N_{c}}{N}\right)^{\alpha_{N}} =(6​B∗​Sc​NL1/αB​C)αS\displaystyle=\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}} (B.4)

B.2 Efficient Training

The fitted optimum stops at a loss about 10% above the same model’s asymptotic loss. Eliminating either model size or loss yields the predicted power laws for efficient compute, parameters, and updates.

Equations B.5–B.10
L⁡(Neff​(C),C)=(1+αNαS)​L​(Neff,∞),L\left(N_{{\rm eff}}\left(C\right),C\right)=\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)L\left(N_{{\rm eff}},\infty\right), (B.5)
L⁡(C)\displaystyle L\left(C\right) =(CcC)αC\displaystyle=\left(\frac{C_{c}}{C}\right)^{\alpha_{C}} (B.6)
αC\displaystyle\alpha_{C} =1/(1/αS+1/αB+1/αN)≈0.052\displaystyle=1/\left(1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}\right)\approx 0.052 (B.7)
Cc\displaystyle C_{c} =6​Nc​B∗​Sc​(1+αNαS)1/αS+1/αN​(αSαN)1/αS.\displaystyle=6N_{c}B_{\ast}S_{c}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{1/\alpha_{S}+1/\alpha_{N}}\left(\frac{\alpha_{S}}{\alpha_{N}}\right)^{1/\alpha_{S}}. (B.8)
N⁡(C)Nc=(CCc)αC/αN​(1+αNαS)1/αN\displaystyle\frac{N\left(C\right)}{N_{c}}=\left(\frac{C}{C_{c}}\right)^{\alpha_{C}/\alpha_{N}}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{1/\alpha_{N}} (B.9)
S⁡(C)\displaystyle S\left(C\right) =Cc6​Nc​B∗(1+αNαS)−1/αN(CCc)αC/αS\displaystyle=\frac{C_{c}}{6N_{c}B_{\ast}}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{-1/\alpha_{N}}\left(\frac{C}{C_{c}}\right)^{\alpha_{C}/\alpha_{S}} (B.10)

B.3 Comparison to Inefficient

Against a model trained to within 2% of its converged loss, the fitted 10%-gap strategy uses about 2.7 times as many parameters, 7.7 times fewer updates, and 65% less training compute at equal target loss. These are consequences of the empirical model, not measurements guaranteed for every workload.

Equations B.11–B.14
L⁡(N,C)=(1+f)​L​(N,∞).L\left(N,C\right)=\left(1+f\right)L\left(N,\infty\right). (B.11)
NfNf′\displaystyle\frac{N_{f}}{N_{f^{\prime}}} =(1+f1+f′)1/αN≈2.7\displaystyle=\left(\frac{1+f}{1+f^{\prime}}\right)^{1/\alpha_{N}}\approx 2.7 (B.12)
SfSf′\displaystyle\frac{S_{f}}{S_{f^{\prime}}} =(1+1f1+1f′)1/αS≈0.13\displaystyle=\left(\frac{1+\frac{1}{f}}{1+\frac{1}{f^{\prime}}}\right)^{1/\alpha_{S}}\approx 0.13 (B.13)
CfCf′\displaystyle\frac{C_{f}}{C_{f^{\prime}}} =NfNf′​SfSf′≈0.35\displaystyle=\frac{N_{f}}{N_{f^{\prime}}}\frac{S_{f}}{S_{f^{\prime}}}\approx 0.35 (B.14)

B.4 Suboptimal Model Sizes

The efficient optimum is fairly broad: sizes from about 0.6 to 2.2 times optimal need only around 20% extra training compute in the fit. Smaller models may reduce inference costs; larger ones may shorten serial training if parallel hardware is available. The paper’s unresolved figure placeholders refer to this tradeoff, illustrated in Figure 12.

Equations B.15–B.17
C(N,L)=(6B∗ScNL1/αB)(L−(NcN)αN)−1/αS.C\left(N,L\right)=\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}}\right)\left(L-\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}\right)^{-1/\alpha_{S}}. (B.15)
C⁡(N,Neff)C⁡(Neff,Neff)=NNeff[1+αSαN(1−(NeffN)αN)]−1/αS.\frac{C\left(N,N_{{\rm eff}}\right)}{C\left(N_{{\rm eff}},N_{{\rm eff}}\right)}=\frac{N}{N_{{\rm eff}}}\left[1+\frac{\alpha_{S}}{\alpha_{N}}\left(1-\left(\frac{N_{{\rm eff}}}{N}\right)^{\alpha_{N}}\right)\right]^{-1/\alpha_{S}}. (B.16)
S⁡(N,Neff)S⁡(Neff,Neff)=[1+αSαN(1−(NeffN)αN)]−1/αS.\frac{S\left(N,N_{{\rm eff}}\right)}{S\left(N_{{\rm eff}},N_{{\rm eff}}\right)}=\left[1+\frac{\alpha_{S}}{\alpha_{N}}\left(1-\left(\frac{N_{{\rm eff}}}{N}\right)^{\alpha_{N}}\right)\right]^{-1/\alpha_{S}}. (B.17)

Appendix C Caveats

The authors lack a complete theory and caution against extrapolating critical batch size, small-data behavior, or long-context compute estimates. Regularization, augmentation, initialization, momentum, and short-run learning rates were not comprehensively tuned. These limitations directly affect the allocation exponents and are essential to interpreting the headline recommendation.

Appendix D Supplemental Figures

The supplemental figures retain training-versus-test behavior and parameter-sharing comparisons before the more detailed diagnostics below.

Original figure
Refer to caption
Figure 16:Left: We characterize the step on which early stopping occurs, as a function of the extent of overfitting. The red line indicates a lower bound for early stopping that is derived in Section5.3.Right: We display train and test loss for a series of 300M parameter models trained on different sized dataset sub-samples. The test loss typically follows that of a run done with unrestricted data until diverging. Note that the degree of overfitting (as compared to the infinite data limit) is significantly overestimated byLtest−LtrainL_{\rm test}-L_{\rm train} (denoted by a black bar for each run).
Original figure
Original figure
Figure 17:We compare recurrent Transformers[DGV+18], which re-use parameters, to standard Transformers. Recurrent Transformers perform slightly better when comparing models with equal parameter count, but slightly worse when accounting for reuse and comparing per FLOP.

D.1 Early Stopping and Test vs Train

The early-stopping plot compares the derived lower bound with measured stopping times. The ordinary train–test gap can overstate the finite-data penalty relative to a model trained on effectively unlimited data.

D.2 Universal Transformers

Parameter reuse modestly improves loss per parameter but modestly worsens loss per FLOP in the tested Universal Transformers. Parameter efficiency and compute efficiency are different objectives.

D.3 Batch Size

Batch-size sweeps at fixed target losses supply the fitted critical-batch measurements used in the main allocation analysis. The two model sizes provide a check on the proposed loss-based dependence.

Refer to caption
Refer to caption
Figure 18:These figures demonstrate fits to Equation (5.1) for a large number of values of the lossLL, and for two different Transformer model sizes. These fits were used to measureBcrit​(L)B_{\rm crit}(L) for Figure10.

D.4 Sample Efficiency vs Model Size

At a fixed target loss, larger models need fewer serial updates and data examples. The curves illustrate substantial sample-efficiency gains while leaving the higher cost per model evaluation visible in the separate compute analysis.

Refer to caption
Refer to caption
Figure 19:The number of minimum serial steps needed to reach any fixed value of the test loss decreases precipitously with model size. Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to a very large one.
Refer to caption
Refer to caption
Figure 20: This figure provides information about the performance per token as a function of model size and training time.Left: Loss per token as a function of its positionTT in the 1024-token context. Loss scales predictably as a power-law inTT.Right: Test loss per token as a function of training step.

D.5 Context Dependence

Larger models improve prediction across context positions and exploit patterns with less preceding text. During training, short-range structure appears before longer-range dependencies. Models trained with eight-token contexts can outperform longer-context models on early tokens, showing that average loss reflects a capacity-allocation tradeoff.

Original figure
Figure 21:In addition to the averaged loss, individual tokens within the 1024-token context also improve smoothly as model size increases. Training runs with shorter contextnctx=8n_{\rm ctx}=8 (dashed lines) perform better on early tokens, since they can allocate all of their capacity to them.
Original figure
Original figure
Figure 22: We test a variety of learning rate schedules including cosine decay, linear decay, as well as other faster/slower decays schedules on a 3 million parameter model, shown on the left. For these experiments we do not decay to zero, since we find that this tends to give a fixed improvement close to the end of training. We find that, as long as the learning rate is not too small and does not decay too quickly, performance does not depend strongly on learning rate. Run-to-run variation is at the level of 0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level.

D.6 Learning Rate Schedules and Error Analysis

Several schedules yield similar final losses when warmup, sufficient learning, and late decay are present. Random-seed variation limits small comparisons. A practical learning-rate formula decreases with model size but is explicitly invalid beyond its fitted range.

Equation D.1
LR⁡(N)≈0.003239+−0.0001395​log⁡(N)\mathrm{LR}(N)\approx 0.003239+-0.0001395\log(N) (D.1)

D.7 Fit Details and Power Law Quality

Power laws fit the observed trends better than the tested logarithmic alternatives. Fits exclude one-layer models and, for parameter scaling, very large models that have not converged. Those exclusions define the fitting procedure rather than exceptions hidden from the reader.

Original figure
Figure 23:The trend for performance as a function of parameter count,L⁡(N)L(N), is fit better by a power law than by other functions such as a logarithm at a qualitative level.
Original figure
Figure 24:We show evaluations on a series of datasets for models with approximately 1.5 Billion parameters. We observe no effect of depth on generalization; generalization performance depends primarily on training distribution performance. The 12-layer model overfit the Internet Books dataset and we show the early-stopped performance; we have not seen this surprising result in other experiments.

D.8 Generalization and Architecture

At roughly fixed parameter count, depth does not materially alter cross-distribution performance after conditioning on in-distribution loss. One Internet Books overfitting case is noted as unusual. The paper’s concluding figure and table indexes contain no additional experimental material.

Concept demonstration

Open image ↗