Literature
Display
← All papers

Intrinsic Entropy of Context Length Scaling in LLMs

Jingzhe Shi · Qinwei Ma · Hongyi Liu · Hang Zhao · Jenq-Neng Hwang · Lei Li

· ICLR 2026 ↗

Language models15 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

The paper explains context-length effects through two components: irreducible uncertainty with limited context and the trained model’s gap from that ideal predictor. It proposes intrinsic entropy of hidden representations as a proxy for information available from context. Experiments find entropy–loss correlations and intermediate best context lengths in selected training and retrieval settings. The framework is conditional on representation and scaling assumptions; it does not establish that every model or task has a finite optimal context length.

1 Introduction

More context can reveal useful evidence while also making prediction harder for a finite model trained on finite data. The authors connect these competing effects to latent-space entropy and dimension, extending earlier data/model scaling arguments Bahri et al., 2024. The contribution is an explanatory framework and a collection of measurements, rather than a new long-context architecture or an algorithm that automatically selects the best context for each input.

Refer to caption
Figure 1: Left: Total loss is decomposed into Bayes Risk (decreasing with context length) and Approximation Loss (increasing with context length), so a critical point can emerge in some scenarios. Middle: Validation Loss Gap (Val Loss - minD(Val Loss) vs. Context Length), measured on subsets of OpenWebText, where we subtract the minimum loss within each dataset-size curve (please refer to Figure 15 for the original figure). For each training dataset size, there exists an optimal context length that minimizes pretraining validation loss, and this optimum increases with dataset size. Right: Error rate of Qwen series models on the CWE task from RulerBench when a certain amount of context is masked. Critical points are visible. More results on other RulerBench subsets are shown in Figure 8.

2 Assumptions, Deductions and Observations for Language Modeling

2.1 Preliminaries

2.1.1 Preliminary: Loss Decomposition

Expected next-token cross-entropy decomposes into the entropy of the ideal conditional predictor for visible length l plus the expected KL divergence from that predictor to the learned model. The first term is Bayes risk; the second is called approximation loss here and includes all learned-model mismatch. Bayes risk depends on the data distribution and visible information, not the particular model parameters. The exact decomposition should be distinguished from later assumptions about how each term changes with length.

Equation 1
H⁡(P,Ql)=\displaystyle H(P,Q_{l})= RB​a​y​e​s+LA​p​p​r​o​x\displaystyle R_{Bayes}+L_{Approx} (1)
=\displaystyle= H⁡(P,Pl)+DK​L​(Pl,Ql)\displaystyle H(P,P_{l})+D_{KL}(P_{l},Q_{l})

2.1.2 Preliminary: Intrinsic Space

Intrinsic space is a latent representation of contexts from which a decoder can predict the next token. In experiments, the authors use hidden features of trained networks as a proxy for this ideal space. Its geometry and entropy depend on the representation, model, and task; a hidden-vector entropy estimate is not automatically a model-independent information quantity.

2.1.3 Preliminary: Outlines

The analysis first relates ideal prediction loss to intrinsic entropy, then relates finite-data approximation to intrinsic dimension. Their proposed trade-off motivates context-length experiments during training and fixed-model inference. Appendix D formalizes the assumptions and gives geometric bounds.

2.2 Bayes Risk with context length: an Intrinsic Entropy perspective

2.2.1 Bayes Risk and Entropy in Intrinsic Space: derived from First Principles

The framework assumes that intrinsic entropy increases with context and approaches a finite limit. It further assumes that the reduction in next-token uncertainty is affine in intrinsic entropy, with a positive coefficient below one. Under that assumption, Bayes risk equals a constant minus a positive multiple of intrinsic entropy. The linear relationship is therefore an explicit premise of the theory, with experiments supplying evidence for its usefulness. Hidden states may also retain information needed for later tokens, so representation entropy is distinguished from next-token information. Continuous derivatives of integer context length use a smooth-extension convention.

Equation 2
RB​a​y​e​s\displaystyle R_{Bayes} =H⁡(P,Pl)\displaystyle=H(P,P_{l}) (2)
=−k∗S(Pl)+Const\displaystyle=-k*S(P_{l})+Const

2.2.2 Bayes Risk and Intrinsic Entropy: Experiment Measurement

The authors use pretrained language-model cross-entropy as a proxy for ideal Bayes risk and fit a decaying power law C₀ + C/lᵞ. A good fit describes the measured models and corpora; it does not separately measure their remaining approximation loss or prove that they attain Bayes risk.

Equation 3
H⁡(P,Pl)≈C0+C/lγH(P,P_{l})\approx C_{0}+C/l^{\gamma} (3)

Experimentally measure Intrinsic Entropy using Gaussian-KDE

For each context length, at least 10,000 final-token, final-layer vectors are collected and Gaussian KDE estimates their representation entropy. On OpenWebText, Figure 2 reports loss–entropy correlations of −0.9888 for Llama-3.1-8B, −0.9960 for Qwen3-8B-Base, and −0.9967 for RecurrentGemma-9B after excluding three high-loss short-context points. The fitted slopes differ across models. These are strong within-experiment correlations; high-dimensional density estimation, feature scaling, and the selected exclusions limit interpreting them as exact or universal entropy laws.

Equation
RB​a​y​e​s≈−k∗S(Pl)+Const,R_{Bayes}\approx-k*S(P_{l})+Const,
Refer to caption
Figure 2: Cross Entropy loss vs. Gaussian-KDE measured Intrinsic Entropy (in nats) for three Language Models on a subset of OpenWebText: Llama-3.1-8B (left, k=−0.0038k=-0.0038, R=−0.9888R=-0.9888), Qwen3-8B-Base (middle, k=−0.0026k=-0.0026, R=−0.9960R=-0.9960), and RecurrentGemma-9B (right, k=−0.0174k=-0.0174, R=−0.9967R=-0.9967, with 3 outlier points at high CE loss excluded from regression). The linear relationship between CE loss and Intrinsic Entropy holds across different model architectures.

2.3 Approximation Loss with Context Length: an Intrinsic Dimension Perspective

Approximation Loss in the Training Scenario

The proposed finite-data model is C₀ + A(l)/Dᵅ⁽ˡ⁾, with exponent approximately inversely related to intrinsic dimension. If longer context raises effective dimension, finite data can make approximation harder. Appendix D derives nearest-neighbor upper bounds under smooth-density, finite-negative-moment, and Lipschitz-decoding conditions, building on Sharma and Kaplan, 2022. An upper bound is not an exact loss law for a trained transformer, and length dependence of A(l) or other training effects must also be considered.

Equation 4
LA​p​p​r​o​x\displaystyle L_{Approx} =C0+A⁡(l)/Dα⁡(l),\displaystyle=C_{0}+A(l)/D^{\alpha(l)}, (4)
∂α∂l\displaystyle\frac{\partial\alpha}{\partial l} <0.\displaystyle<0.

Approximation Loss in the Inference Scenario

For a fixed pretrained model, the paper similarly proposes that longer visible context makes the ideal predictor harder to approximate. Harder tasks may increase this gap further. This is a modeling explanation for the observed failures, not a general theorem that every fixed model’s approximation error must grow monotonically with context.

3 Deduction: Optimal Context Length

A finite optimum can occur when the decreasing Bayes-risk slope and increasing approximation-loss slope cross. The authors argue that more training data or stronger models can move that balance to longer contexts, while tasks needing distant evidence can also benefit from longer inputs. Opposing monotonic trends alone do not guarantee a crossing, uniqueness, or its direction of movement; those conclusions require additional shape assumptions. The empirical optima below are measured within the tested context ranges.

Equation 5
Loss​(l,θt,θm)\displaystyle\text{Loss}(l,\theta_{t},\theta_{m}) =RB​a​y​e​s​(l,θt)+LA​p​p​r​o​x​(l,θm),\displaystyle=R_{Bayes}(l,\theta_{t})+L_{Approx}(l,\theta_{m}), (5)
Original figure
Figure 3: Our modified Position-Weighted Ruler-QA1 (Hsieh et al., 2024) dataset. Multiple paragraphs are concatenated together with context length close to M​a​x​C​t​lMaxCtl, and a question queries the ‘golden paragraph’ (i.e. the doc paragraph with answer to that query). In the original Ruler-QA1 dataset each doc has equal probability of being queried (i.e. γ=0\gamma=0); while in our experiments shown in Figure 4, we measure LLM performance on a set of tasks with different hyper-parameter γ\gamma, each with different probability of querying far-away contexts.
Refer to caption
Figure 4: Measured results on Position-Weighted Ruler-QA1 dataset. Left: QA accuracy vs. number of tokens input to the Language Model, for different tasks with different γ\gamma values. We observe that: (1) each curve shows a trend to increase and then decrease with context length; and (2) the critic point corresponds to a smaller optimal context length for tasks with larger γ\gamma (i.e. tasks requiring less long context abilities). Right: Intrinsic Entropy measured on samples truncated to certain context lengths. The Intrinsic Entropy shows increment of intrinsic information when increasing context length, and resembles acc-ctl curves for larger γ\gamma.

3.1 Experimental Measurement of Optimal Context Length for Training

The training experiment uses a GPT-2-style network in nanoGPT Karpathy, 2022, with twelve heads and width 768 but only six transformer layers instead of twelve. Thus the named GPT-2-124M configuration is modified. The authors train with 200M, 250M, 300M, 350M, 500M, and 750M tokens and vary context length. Smaller-data curves exhibit minima followed by modest loss increases; the largest-data curve continues improving near the tested endpoint. Figure 1 subtracts each curve’s minimum for visibility, while Figure 15 retains absolute validation loss. Results support dataset-dependent context choices for this setup, rather than a universal cutoff.

3.2 Experimental Measurement of Optimal Context Length on Downstream Tasks

Fixed-model tests vary how much of a long input remains visible. Position-Weighted Ruler-QA1 modifies Hsieh et al., 2024 by sampling the answer-bearing paragraph with weight proportional to (1 − distance/maximum-length)ᵞ. Larger γ concentrates questions near the input end, while γ = 0 is uniform. In the tested Llama setup, more distant-information tasks generally favor longer visible contexts. Additional Qwen3 tasks show peaks for several question-answering and word-extraction settings, but variable tracking keeps improving across the tested range.

4 Proof of Concept with Synthetic Data

4.1 List of Points to prove

Synthetic parity tasks provide an analytically computable Bayes loss, allowing a clearer check of whether learned representations track information than is possible for natural language. The tests target loss–entropy linearity and agreement between measured hidden-space proxies.

4.2 Construction of Synthetic Data: the ‘position weighted multitask sparse parity’ dataset

Each task selects two fixed binary context positions and predicts their XOR. If either required bit is hidden, the best predictor incurs log 2 loss; otherwise the answer is known. Weighting task frequencies toward nearer positions produces a decreasing Bayes-loss curve resembling the fitted context power law. Figure 5 also illustrates why a single answer-token perplexity need not change equally when either of two relevant facts is removed. The main text alternates between 100 and 200 tasks: two tasks per maximum position from 11 through 60 specifies 100, while the task vocabulary is described as size 200. This reporting ambiguity is retained explicitly.

Original figure
Figure 5: Left: An example of the ‘two needles in a haystack’ task, similar to those in (Levy et al., 2024). The text part is the input to the Language Model, with key information and question visualized in blue; the figure part shows perplexity of the answer token ⟨8⟩\langle 8\rangle of LLaMa-3.1-8B (horizontal) vs. number of masked leftmost tokens (vertical). Although seeing both pieces of information are necessary to answer the question, perplexity rises dramatically only when the first piece of information is masked. Right: An example of our synthetic data. Each sub-task corresponds to 2 context bits of fixed position. At each time, exactly one sub-task is activate, and the ground truth output is calculated by taking XOR over the 2 context bits of the activate task. As shown in the example,

the answer for Subtask 1,2,3 is 0⊕0=0,0⊕1=10\oplus 0=0,0\oplus 1=1 and 1⊕1=01\oplus 1=0 respectively, but since the thrid bit is 11 for control bits, only Subtask 3 is activated and the final answer is 00. However, for a model of context length 77, it cannot see the 9​t​h9th bit required by subtask 3, making it unable to predict the answer correctly.

Equation
RB​a​y​e​s​(c​t​l)\displaystyle R_{Bayes}(ctl)
=\displaystyle= M​i​n​C​E​L​o​s​s​(c​t​l)\displaystyle MinCELoss(ctl)
=\displaystyle= (∑t​a​s​k​s.t.m​a​x​(b​i​t1,b​i​t2)>c​t​lf​r​e​q​(t​a​s​k)​log⁡2)/∑t​a​s​kf​r​e​q​(t​a​s​k)\displaystyle(\sum_{task\,s.t.\,max(bit_{1},bit_{2})>ctl}freq(task)\log 2)/\sum_{task}freq(task)
≈\displaystyle\approx A+B/(c​t​l+C)α\displaystyle A+B/(ctl+C)^{\alpha}

4.3 Transformer-based Synthetic Model with Entropy Measurements

The synthetic transformer has three causal layers, width 208, feed-forward width 832, and RoPE base 4,000. Its sequence contains sixty context positions and a task token. Half of training examples are unmasked; the remainder hide a variable suffix using the source’s mask convention. The model is trained to approximate the known Bayes predictor. Eigenvalue spectra spread with greater visible context, and sums of log eigenvalues correlate strongly with measured and theoretical cross-entropy, especially when at least seventy dimensions are included.

Refer to caption
Figure 6: Transformer and Rope-based model for the synthetic task. Here, we use one task token to encode the task information.
Refer to caption
Figure 7: Eigen value and CE results measured on trained model for Synthetic Dataset in this section. Left: Eigen Value vs. Index of Eigen Value; Right: Correlation between Cross Entropy Loss and Measured Entropy. We see good linear relationship between CE Losses and Measured Intrinsic Entropy from Lower figures.

4.4 Entropy in Intrinsic Space: Synthetic Dataset Validation

The synthetic model’s loss is close to its calculable optimum over the reported lengths, making this a useful controlled validation of the representation measurements. Both eigenvalue and density-based proxies show the expected association. The task frequencies and architecture are constructed to expose that structure, so the experiment does not independently establish the same assumptions for arbitrary natural-language problems.

5 Conclusion and Discussions

5.1 Conclusion

The work combines a Bayes-risk/approximation-error decomposition, representation-entropy correlations, and context sweeps showing intermediate optima in selected regimes. Its practical implication is to evaluate usable context jointly with data, model, and task, rather than assume that a larger accepted input always improves accuracy.

5.2 Limitations and Future Work

The authors acknowledge that the theory depends on its assumptions and that intrinsic geometry may depend on the model and prediction task. Further limits include approximating Bayes risk with imperfect models, high-dimensional entropy estimation, conditional geometric bounds, incomplete uncertainty reporting, and a limited set of training architectures and downstream tasks. The study motivates deeper theory and more general measurements; it does not provide an accuracy-preserving truncation rule for arbitrary prompts.

Appendix A Downstream Task

The appendix separates next-token prediction from downstream accuracy. It examines when a fixed model benefits from additional visible documents and whether representation entropy remains informative about that benefit.

A.1 Optimal Context Length on Ruler-QA: a Case Study

Ruler-QA1 inserts a SQuAD answer-bearing paragraph among other paragraphs, then varies the suffix visible to Llama-3.1-8B. The reported uniform-query curve peaks around 6k tokens before declining. Position weighting changes how often distant paragraphs matter. Appendix A mentions both 16k and 8k maximum-length constructions, while the main diagram and result plot use about 8k; the precise case-study settings should be checked in the released code before replication.

A.2 Intrinsic Entropy: a proxy of information learned by Language Model

KDE entropy is measured from late-token hidden states of truncated document collections. It generally rises with visible length, but it is independent of the query-position parameter γ, whereas accuracy depends on γ. It is therefore not a universal linear predictor of QA success. The authors interpret late-context accuracy deterioration as failure to approximate the task’s ideal predictor; the entropy correlation is strongest before that deterioration.

A.3 More experiments on RULER benchmark

Qwen3 non-thinking chat models are tested on document QA, frequent-word extraction, common-word extraction, and variable tracking. Maximum length is 16k for common-word extraction and 8k for the other tasks. Figure 8 includes the four task plots; the additional Figure A.3-2 compares Qwen3-4B common-word extraction at 8k and 16k. Several models peak before full visibility, while variable tracking improves throughout. Larger models generally handle longer inputs better, and the 16k common-word task makes deterioration more apparent. These exceptions matter: a finite optimum is a measured phenomenon in some settings, not every curve.

Refer to caption
Figure 8: Acc vs. Visible Context Length of Qwen-3 series models (non-thinking chat models) on 4 representative subsets of the RULER dataset: qa_1 (document qa, upper-left), fwe (frequent word extraction, upper-right), cwe (common words extraction, lower-left), and vt (variable tracking, lower-right), for a fixed max context length and a varying visible fraction of the input context. Most models show an optimal context length for qa_1, fwe and cwe subtask, while the vt subtask shows increased performance with respect to context length. Moreover, larger model tends to perform better and have a larger optimal context length, represented by the performance comparison between Qwen3-4B and Qwen3-8B on cwe subtask (lower-left).
Original figure
Figure A.3-2. Acc vs. Visible Context Length of Qwen-3 4B, on cwe (common words extraction) subtask, with different Max context length (left: M​a​x​C​t​l=8​kMaxCtl=8k, right: M​a​x​C​t​l=16​kMaxCtl=16k). As shown, though optimal context length is hard to observe for the task requiring 8​k8k as max context length, it is easy to observe for that requiring 16​k16k as max context length.

Appendix B Definition and Properties of Cross Entropy loss

B.1 Definition of Cross Entropy Loss discussed in this work

The relevant cross-entropy averages negative log next-token probability over contexts and next tokens from the data distribution. It is the teacher-forced conditional prediction loss, rather than the cross-entropy of an entire generated context sequence. The appendix retains the full derivation connecting these definitions.

Equations 6
Ho​r​g(P,Q)=∑x\displaystyle H_{org}(P,Q)=\sum_{x} −P⁡(x)​log⁡Q​(x)\displaystyle-P(x)\log Q(x)
=∑x\displaystyle=\sum_{x} −P(x−∞:0)P(x0|x−∞:0)\displaystyle-P(x_{-\infty:0})P(x_{0}|x_{-\infty:0})
∗log{Q(x0|x−∞:0)Q(x−∞:0)},\displaystyle*\log\{Q(x_{0}|x_{-\infty:0})Q(x_{-\infty:0})\},
He​x​p​(P,Q)\displaystyle H_{exp}(P,Q)
=\displaystyle= ∑x−P(x−∞:1)log{Q(x0|x−∞:0)𝐏(𝐱−∞:𝟎)}\displaystyle\sum_{x}-P(x_{-\infty:1})\log\mathbf{\{}Q(x_{0}|x_{-\infty:0})\mathbf{P(x_{-\infty:0})}\mathbf{\}}
=\displaystyle= Const+Ex−∞:0∑x0−P(x0|x−∞:0)logQ(x0|x−∞:0).\displaystyle\text{Const}+E_{x_{-\infty:0}}\sum_{x_{0}}-P(x_{0}|x_{-\infty:0})\log Q(x_{0}|x_{-\infty:0}).
H⁡(P,Q)\displaystyle H(P,Q) (6)
=\displaystyle= He​x​p​(P,Q)\displaystyle H_{exp}(P,Q)
=\displaystyle= Ex−∞:0[∑x0−P(x0|x−∞:0)logQ(x0|x−∞:0)]\displaystyle\text{E}_{x_{-\infty:0}}[\sum_{x_{0}}-P(x_{0}|x_{-\infty:0})\log Q(x_{0}|x_{-\infty:0})]

B.2 Cross Entropy Loss for Language Model with Context Length l

When a model sees only the last l tokens, marginalizing the unavailable earlier context gives the true conditional distribution Pₗ. Expected cross-entropy then splits exactly into its conditional entropy plus expected KL(Pₗ ‖ Qₗ). Setting Qₗ = Pₗ identifies the limited-context Bayes risk and isolates the information lost by truncation.

Equations 7
H⁡(P,Ql)\displaystyle H(P,Q_{l})
=\displaystyle= Ex−∞:0[∑x0−P(x0|x−∞:0)logQl(x0|x−l:0)]\displaystyle\text{E}_{x_{-\infty:0}}[\sum_{x_{0}}-P(x_{0}|x_{-\infty:0})\log Q_{l}(x_{0}|x_{-l:0})]
=\displaystyle= −∑x−∞:1P(x−∞:1)logQl(x0|x−l:0)\displaystyle-\sum_{x_{-\infty:1}}P(x_{-\infty:1})\log Q_{l}(x_{0}|x_{-l:0})
=\displaystyle= −∑x−∞:−l∑x−l:1P(x−∞:−l,x−l:1)logQl(x0|x−l:0)\displaystyle-\sum_{x_{-\infty:-l}}\sum_{x_{-l:1}}P(x_{-\infty:-l},x_{-l:1})\log Q_{l}(x_{0}|x_{-l:0})
=\displaystyle= −∑x−l:1P(x−l:1)logQl(x0|x−l:0)\displaystyle-\sum_{x_{-l:1}}P(x_{-l:1})\log Q_{l}(x_{0}|x_{-l:0})
=\displaystyle= Ex−l:0[∑x0−P(x0|x−l:0)logQl(x0|x−l:0)]\displaystyle E_{x_{-l:0}}[\sum_{x_{0}}-P(x_{0}|x_{-l:0})\log Q_{l}(x_{0}|x_{-l:0})]
=\displaystyle= H⁡(Pl,Ql).\displaystyle H(P_{l},Q_{l}).
DK​L​(P,Ql)\displaystyle D_{KL}(P,Q_{l}) =−H⁡(P)+H⁡(P,Ql)\displaystyle=-H(P)+H(P,Q_{l})
=−H⁡(P)+H⁡(Pl,Ql)\displaystyle=-H(P)+H(P_{l},Q_{l})
=−H⁡(P)+H⁡(Pl)+DK​L​(Pl,Ql).\displaystyle=-H(P)+H(P_{l})+D_{KL}(P_{l},Q_{l}).
DK​L​(P,Pl)=−H⁡(P)+H⁡(Pl,Pl)=−H⁡(P)+H⁡(Pl).D_{KL}(P,P_{l})=-H(P)+H(P_{l},P_{l})=-H(P)+H(P_{l}). (7)

Appendix C Experimentally measure next-token-prediction Information Entropy S n ​ t ​ p

C.1 PCA-based Information Entropy Estimation

The representation can encode information relevant to several future predictions, whereas the current output is a coarse-grained readout. The authors use correlations between cross-entropy and entropy proxies in different PCA subspaces to motivate a linear relationship. Correlation alone does not identify a unique predictive subspace or prove the assumed information mapping.

C.2 Gaussian-KDE based Information Entropy Estimation

Figure 9 compares Gaussian-KDE entropy with PCA-based entropy, threshold-based dimension, and cross-entropy. It uses 10,000 samples and reported automatic bandwidth 0.997756. The estimators show similar trends on this experiment; the paper does not establish estimator accuracy across arbitrary hidden-vector distributions or bandwidth choices.

Refer to caption
Figure 9: Gaussian-KDE measured Entropy (10000 samples, auto bandwidth = 0.997756) vs. PCA-measured Entropy (left), Measrued ID (middle) and Cross Entropy Loss (right).

C.3 Synthetic dataset: Entropy in Intrinsic Space, and Entropy for output layer

In an idealized task-factorized representation, storing T independent binary task answers gives T log 2 entropy, while one binary output has log 2 entropy. This construction motivates proportional information scales. It relies on the assumed factorization and task structure, rather than proving that all learned latent dimensions encode equal independent information.

C.4 Details for synthetic data

The synthetic entropy experiment uses 10 million training examples, one million validation examples, and 125 epochs, with early-stopping patience 25 that is not triggered. Table 1 compares nine lengths from 17 to 50. Measured losses range from 0.4648 to 0.0613 and differ from calculated minima by at most about 0.0005 in the table. This directly supports near-Bayes behavior for that synthetic model over those lengths.

Equation
M​i​n​C​E​L​o​s​s​(c​t​l)=(∑task s.t. m​a​x​(b​i​t1,b​i​t2)>c​t​lf​r​e​q​(t​a​s​k)∗log⁡2)/∑t​a​s​kf​r​e​q​(t​a​s​k)MinCELoss(ctl)=(\sum_{\text{task s.t. $max(bit_{1},bit_{2})>ctl$}}freq(task)*\log 2)/{\sum_{task}freq(task)}
Context Length Model CE Loss Minimum CE Loss Calculated
17 0.4648 0.4643
20 0.3988 0.3988
23 0.3438 0.3437
25 0.3119 0.3116
28 0.2687 0.2686
30 0.2429 0.2429
35 0.1867 0.1864
40 0.1390 0.1387
50 0.0613 0.0612
Table 1: Comparison between trained model and Bayes Model (minimum CE Loss) for Synthetic Data

Appendix D Definitions of Intrinsic Space and Derived Properties

This appendix makes latent-space properties explicit and derives conditional relationships. The distinction is between defining an ideal representation with these properties and showing how well actual network features satisfy them.

D.1 Formal Definitions of Intrinsic Space

Setup.

A finite vocabulary generates the set of finite contexts, and the data distribution defines its support. A mapping sends contexts of bounded length to a latent space.

Definition.

Predictive consistency requires a decoder that recovers the true next-token conditional distribution. Separate assumptions specify uniform predictive information per effective dimension or an affine entropy–prediction relationship. The geometric results also require a density with bounded gradient and a finite integral of q to the power 1 − ε. Table 2 maps each theorem to its assumptions. A finite vocabulary and a neural feature map do not by themselves establish the smooth-density or information-linearity conditions.

Equations 8–14
Φ:𝒳≤t→𝒵,\Phi:\mathcal{X}_{\leq t}\to\mathcal{Z}, (8)
π⁡(Φ⁡(x<t))=p⁡(xt∣x<t),\pi(\Phi(x_{<t}))=p(x_{t}\mid x_{<t}), (9)
DK​L​(P,Pl)=s⋅(dim(∞)−dim(l))D_{KL}(P,P_{l})=s\cdot\left(\dim(\infty)-\dim(l)\right) (10)
|−sH[qt(Z)]+b−H[p(⋅∣x<t)]|≤εt.\Bigl|\,-s\;H\bigl[q_{t}(Z)\bigr]\;+\;b\;-\;H\bigl[p(\,\cdot\mid x_{<t})\bigr]\;\Bigr|\;\leq\;\varepsilon_{t}. (11)
−s​H​[qt​(Z)]+b=H⁡[p⁡(xt∣x<t)]∀t.-s\;H\bigl[q_{t}(Z)\bigr]\;+\;b=\;H\bigl[p(x_{t}\mid x_{<t})\bigr]\quad\forall\,t. (12)
‖∇q​(z)‖≤L\|\nabla q(z)\|\leq L (13)
∫𝒵q​(z) 1−ϵ​𝑑z:=Cϵ<∞.\int_{\mathcal{Z}}q(z)^{\,1-\epsilon}\,dz:=C_{\epsilon}<\infty. (14)
Key Properties Assumed Derived Results
(Bahri et al., 2024)Theorem 2 Bounded, Uniform Distribution, Data Scaling for Approx. Loss
Lipschitz Differentiable
Theorem 1, 2 in Appendix D.2 (Bounded,) Finite Negative Moment, Data Scaling for Approx. Loss
(corr. to Section 2.3) Lipschitz Differentiable
Theorem 3 in Appendix D.3 Predictive Consistency, Bayes Risk for Ntp of varied Ctl
(corr. to Section 2.2.1) Uniform Information Gain (Intrinsic Dimension perspective)
Theorem 4 in Appendix D.4 Predictive Consistency, Bayes Risk for Ntp of varied Ctl
(corr. to Section 2.2.1) Linear Entropy Relationship (Information Entropy perspective)
Table 2: In this Section (Appendix D) we formulate results from previous sections with Theorems derived with defined assumptions and properties of intrinsic space in this section. Ntp refers to Next-token-prediction, Ctl refers to Context Length. We derive data scaling for approximation loss with weaker assumptions compared to (Bahri et al., 2024), please refer to Theorem 1, 2 in Appendix D.2 for more details.

D.2 Derivation for Data Scaling for Approximation Loss

Theorem 1 bounds expected capped nearest-neighbor distance among D independent latent samples. Its proof uses a local density lower bound to control empty-ball probability, splits moderate/high-density from low-density regions, bounds the first with a tail integral, and controls the second using the finite negative moment. The resulting rate is D⁻¹ᐟᵈ when ε is large enough; otherwise a slower (log D/D)ᵋᐟ⁽ᵈ⁺¹⁾ term controls the bound. Every displayed proof equation is retained. The result concerns an expected geometric distance under stated regularity conditions, not measured transformer generalization by itself.

Equations 15–22
RM​(Zi)=min⁡{M,minj≠i⁡‖Zi−Zj‖},M>0.R_{M}(Z_{i})\;=\;\min\!\bigl\{\,M,\;\min_{j\neq i}\|Z_{i}-Z_{j}\|\,\bigr\},\qquad M>0. (15)
𝔼𝒵D​[RM​(Z1)]≤{C(logD)ϵ/(d+1)D−ϵ/(d+1),if ​ϵ<d+1d,CD−1/d,if ​ϵ≥d+1d.\mathbb{E}_{\mathcal{Z}_{D}}\!\bigl[R_{M}(Z_{1})\bigr]\;\leq\;\begin{cases}C\,(\log D)^{\epsilon/(d+1)}\,D^{-\epsilon/(d+1)},&\text{if }\displaystyle\epsilon<\frac{d+1}{d},\\[6.0pt] C\,D^{-1/d},&\text{if }\displaystyle\epsilon\geq\frac{d+1}{d}.\end{cases} (16)
𝔼𝒵D[RM(Z1)]≤CD−c/d.\mathbb{E}_{\mathcal{Z}_{D}}\!\bigl[R_{M}(Z_{1})\bigr]\;\leq\;C\;D^{-c/d}. (17)
vd:=vol​(B1​(0)),cd:=vd2,r0​(z):=q⁡(z)2​L.v_{d}:=\text{vol}(B_{1}(0)),\qquad c_{d}:=\frac{v_{d}}{2},\qquad r_{0}(z):=\frac{q(z)}{2L}.
μr​(z):=∫Br​(z)q⁡(u)​𝑑u≥12​q​(z)​vd​rd=cd​q​(z)​rd.\mu_{r}(z)\;:=\;\int_{B_{r}(z)}q(u)\,du\;\geq\;\frac{1}{2}\,q(z)\,v_{d}r^{d}=\;c_{d}q(z)r^{d}.
Pr⁡(R⁡(z)>r|Z1=z)=(1−μr​(z))D−1≤exp⁡[−cd​(D−1)​q​(z)​rd].\Pr\!\bigl(R(z)>r\,\big|\,Z_{1}=z\bigr)=\bigl(1-\mu_{r}(z)\bigr)^{D-1}\;\leq\;\exp\!\bigl[-c_{d}(D-1)q(z)\,r^{d}\bigr]. (18)
λD:=(2d+1​Ld​d​log⁡Dcd​D)1d+1(D≥2),ℋD:={z:q⁡(z)≥λD},ℒD:=ℝd∖ℋD.\lambda_{D}\;:=\;\Bigl(\frac{2^{d+1}L^{d}d\log D}{c_{d}D}\Bigr)^{\!\frac{1}{d+1}}\quad(D\geq 2),\qquad\mathcal{H}_{D}:=\bigl\{z:q(z)\geq\lambda_{D}\bigr\},\quad\mathcal{L}_{D}:=\mathbb{R}^{d}\!\setminus\mathcal{H}_{D}.
ρ⁡(z,D):=(2​d​log⁡Dcd​D​q​(z))1/d.\rho(z,D)\;:=\;\Bigl(\frac{2d\log D}{c_{d}D\,q(z)}\Bigr)^{\!1/d}\;.
Pr⁡(R⁡(z)>ρ⁡(z,D)|Z1=z)≤exp⁡[−2​d​log⁡D]=D−2​d.\Pr\!\bigl(R(z)>\rho(z,D)\,\big|\,Z_{1}=z\bigr)\;\leq\;\exp[-2d\log D]=D^{-2d}.
𝔼[RM(Z1)𝟏{R>ρ}|Z1=z]≤MD−2​d.\mathbb{E}\bigl[R_{M}(Z_{1})\mathbf{1}_{\{R>\rho\}}\,\big|\,Z_{1}=z\bigr]\;\leq\;MD^{-2d}. (19)
𝔼⁡[R⁡(Z1)∧ρ⁡(z,D)|Z1=z]\displaystyle\mathbb{E}\!\bigl[R(Z_{1})\wedge\rho(z,D)\,\big|\,Z_{1}=z\bigr] =∫0ρPr⁡(R>r|Z1=z)​𝑑r\displaystyle=\int_{0}^{\rho}\Pr(R>r\,|\,Z_{1}=z)\,dr
≤∫0ρexp⁡[−cd​(D−1)​q​(z)​rd]​dr.\displaystyle\leq\int_{0}^{\rho}\exp\!\bigl[-c_{d}(D-1)q(z)r^{d}\bigr]\,dr.
∫0ρexp[−cd(D−1)q(z)rd]dr=Γ⁡(1+1/d)d1/d​cd1/d(Dq(z))−1/d.\int_{0}^{\rho}\!\!\exp[-c_{d}(D-1)q(z)r^{d}]\,dr=\frac{\Gamma(1+1/d)}{d^{1/d}c_{d}^{1/d}}\bigl(Dq(z)\bigr)^{-1/d}.
𝔼[R(Z1)∧ρ(z,D)|Z1=z]≤Cd,L(Dq(z))−1/d.\mathbb{E}\bigl[R(Z_{1})\wedge\rho(z,D)\,\big|\,Z_{1}=z\bigr]\;\leq\;C_{d,L}\,\bigl(Dq(z)\bigr)^{-1/d}. (20)
𝔼[RM(Z1)𝟏ℋD(Z1)]≤C1D−1/d+MD−2​d,C1:=Cd,L(𝔼[q(Z)−1/d])1/d<∞.\mathbb{E}\bigl[R_{M}(Z_{1})\mathbf{1}_{\mathcal{H}_{D}}(Z_{1})\bigr]\leq C_{1}D^{-1/d}\;+\;MD^{-2d},\quad C_{1}:=C_{d,L}\bigl(\mathbb{E}[q(Z)^{-1/d}]\bigr)^{1/d}<\infty. (21)
Pr⁡(Z∈ℒD)=∫q<λDq⁡(z)​𝑑z≤λDϵ​∫ℝdq​(z)1−ϵ​𝑑z=Cϵ​λDϵ.\Pr\bigl(Z\in\mathcal{L}_{D}\bigr)=\int_{q<\lambda_{D}}\!\!q(z)\,dz\leq\lambda_{D}^{\epsilon}\int_{\mathbb{R}^{d}}\!\!q(z)^{1-\epsilon}\,dz=C_{\epsilon}\lambda_{D}^{\epsilon}.
𝔼[RM(Z1)𝟏ℒD(Z1)]≤MCϵλDϵ=MCϵ(logD)ϵ/(d+1)D−ϵ/(d+1).\mathbb{E}\!\bigl[R_{M}(Z_{1})\mathbf{1}_{\mathcal{L}_{D}}(Z_{1})\bigr]\leq MC_{\epsilon}\lambda_{D}^{\epsilon}=MC_{\epsilon}\bigl(\log D\bigr)^{\epsilon/(d+1)}D^{-\epsilon/(d+1)}. (22)
𝔼[RM(Z1)]≤C1D−1/d+MCϵ(logD)ϵ/(d+1)D−ϵ/(d+1)+o(min(D−1/d,D−ϵ/(d+1))).\mathbb{E}\bigl[R_{M}(Z_{1})\bigr]\;\leq\;C_{1}D^{-1/d}\;+\;MC_{\epsilon}(\log D)^{\epsilon/(d+1)}D^{-\epsilon/(d+1)}+o(\min(D^{-1/d},D^{-\epsilon/(d+1)})).

How large can ϵ be, for unbounded and bounded 𝒵 ?

A bounded support with density bounded away from zero satisfies stronger negative-moment conditions and permits the faster rate. Typical unbounded densities permit a smaller range of ε, giving a slower bound. This discussion explains the sensitivity to low-density tails; bounded support, compactness, finite measure, and a positive density lower bound should not be treated as interchangeable without further conditions.

Equation 23
{D−ϵ/(d+1)for every admissible ​1/d<ϵ<1,D−1/donly if the support is compact and ϵ>1+1/d.\begin{cases}D^{-\epsilon/(d+1)}&\text{for every admissible }1/d<\epsilon<1,\\ D^{-1/d}&\text{only if the support is compact and $\epsilon>1+1/d$.}\end{cases} (23)

From Nearest-Neighbour Distance to Approximation Loss

Theorem 2 combines the geometric result with Lipschitz control of decoder KL loss and bounded nearest-neighbor distance. It bounds expected nearest-neighbor predictive mismatch. The motivating connection to learned-model approximation additionally uses interpolation/Lipschitz assumptions from earlier work Bahri et al., 2024. It supplies a conditional scaling upper bound, rather than an equality or convergence guarantee for the training algorithm used in the experiments.

Equations 24–26
minj≠il(g(Zi),g(Zj))≤CD−c/d.\min_{j\neq i}l(g(Z_{i}),g(Z_{j}))\leq C\;D^{-c/d}. (24)
minj≠i⁡l⁡(g⁡(Zi),g⁡(Zj))\displaystyle\min_{j\neq i}l(g(Z_{i}),g(Z_{j})) ≤minj≠i⁡Ll⋅‖Zi−Zj‖\displaystyle\leq\min_{j\neq i}\;L_{l}\;\cdot\;||Z_{i}-Z_{j}|| (25)
=Ll⋅min⁡{M,minj≠i⁡‖Zi−Zj‖}\displaystyle=L_{l}\cdot\min\{M,\min_{j\neq i}||Z_{i}-Z_{j}||\}
=Ll⋅RM​(Zi)​ by the definition of RM​(Zi) in Theorem 1\displaystyle=L_{l}\cdot R_{M}(Z_{i})\text{ by the definition of $R_{M}(Z_{i})$ in Theorem \ref{thm:nnd}}
𝔼𝒵D[minj≠il(g(Zi),g(Zj))]≤LlCD−c/d\mathbb{E}_{\mathcal{Z}_{D}}[\min_{j\neq i}l(g(Z_{i}),g(Z_{j}))]\leq L_{l}\;C\;D^{-c/d} (26)

Meaning of a finite ϵ -negative moment

The finite negative moment constrains low-density behavior. For 0 < ε < 1, it is algebraically related to finite Rényi entropy of order 1 − ε. The source also gives a low-density-volume interpretation. These formulas are preserved as the appendix presents them; the usable nearest-neighbor result still depends on its full smoothness, sampling, and support assumptions.

Equations 27–29
Leb​(Et)≤t−ϵ​∫q1−ϵ=Cϵ​t−ϵ.\text{Leb}(E_{t})\leq t^{-\epsilon}\!\int q^{1-\epsilon}=C_{\epsilon}\,t^{-\epsilon}. (27)
Hα(q)=−1α−1log∫qα.H_{\alpha}(q)=-\frac{1}{\alpha-1}\log\int q^{\alpha}. (28)
∫q1−ϵ=e−(1−ϵ)​H1−ϵ​(q),\int q^{1-\epsilon}=e^{-(1-\epsilon)H_{1-\epsilon}(q)}, (29)

D.3 Derivation for Bayes Risk with Intrinsic Dimension Assumption

Under predictive consistency and uniform information gain, missing latent dimensions contribute a fixed amount to predictive KL divergence. Adding that divergence to irreducible full-context entropy yields Bayes risk affine in intrinsic dimension. A binary-integer example illustrates progressively revealing bits. This theorem derives the stated consequence of the information-per-dimension assumption, rather than establishing the assumption from language alone.

Equations 30–31
H(P,Pl)=−s⋅dim(l)+ConstH(P,P_{l})=-s\cdot dim(l)+Const (30)
H⁡(P,Pl)\displaystyle H(P,P_{l}) =H⁡(P)+DK​L​(P,Pl)\displaystyle=H(P)+D_{KL}(P,P_{l}) (31)
=H⁡(P)+s⋅(d​i​m​(∞)−d​i​m​(l))\displaystyle=H(P)+s\cdot(dim(\infty)-dim(l))
=−s⋅dim(l)+Const\displaystyle=-s\cdot dim(l)+Const

D.4 Derivation for Bayes Risk with Information Entropy Assumption

Under predictive consistency and the zero-tolerance affine entropy assumption, conditional entropy directly becomes a constant minus s times latent entropy. The proof substitutes that assumed relationship into the conditional-loss identity. Nonzero tolerances or imperfect learned features would need additional error accounting.

Equations 32–33
H(P,Pl)=−s⋅H[qt(Z)]+ConstH(P,P_{l})=-s\cdot H[q_{t}(Z)]+Const (32)
H⁡(P,Pl)\displaystyle H(P,P_{l}) =H⁡(Pl)​ (from Appendix B.2)\displaystyle=H(P_{l})\text{ (from Appendix \ref{app: cross entropy loss for lm with context length l})} (33)
=H⁡[p⁡(xt|x<t)]\displaystyle=H[p(x_{t}|x<t)]
=−s⋅H[qt(Z)]+b\displaystyle=-s\cdot H[q_{t}(Z)]+b
=−s⋅H[qt(Z)]+Const\displaystyle=-s\cdot H[q_{t}(Z)]+Const

Appendix E More experiments of LLaMa on another dataset

The authors collect web text containing words they expect Llama’s pretraining filters to remove, then observe a similar context-loss power-law fit for Llama models. This is a supplementary corpus check. Such keyword selection does not independently verify that every passage or overlapping information was absent from pretraining, so the claimed unseen status remains an inference.

Refer to caption
Figure 10: Cross Entropy Loss vs. Context Length, with log scale. We see that y=C0+C/xγy=C_{0}+C/x^{\gamma} fits this curve well.

Appendix F Optimal Context Length for Two-needle-in-Haystack training: study on Synthetic Dataset

A longer synthetic parity setup uses more than 500 context bits, although most tasks depend on the first hundred. MLPs train at fixed dataset sizes until overfitting, and the best validation checkpoint is compared across context lengths. Many finite-data curves have an intermediate minimum that shifts outward with more examples. The experiment demonstrates the trade-off in this constructed problem, while some tested curves keep improving over the displayed range.

Refer to caption
Figure 11: Validation set Cross Entropy Loss of the output token (resembling the ‘answer’ token of two-needle-in-haystack tasks) vs. context length, for MLPs trained with different training dataset sizes.

Appendix G Experiments on Other Language Models

Qwen3-8B-Base closely follows the loss–entropy trend. RecurrentGemma-9B has three short-context points with much higher loss; those are excluded from its linear regression and interpreted as poor approximation to the ideal predictor. The exclusion is part of the reported analysis, and the resulting correlation should not be described as fitting every tested context or architecture equally well.

Appendix H Eigenvalue-based Intrinsic Entropy Estimation

An alternative estimator uses hidden-feature covariance eigenvalues instead of kernel density estimation. Agreement between the two is examined empirically.

H.1 Bayes Risk vs. Context Length on OpenWebText

Llama-3.1 models on OpenWebText exhibit decreasing cross-entropy fitted by C₀ + C/lᵞ. Figure 12 retains both model curves and fits. The plotted quantity is actual model loss used as a Bayes-risk proxy, not an independently known entropy of natural language.

Original figure
Figure 12: Bayes Risk vs. Context Length: Bayes Risk is approximated by Cross Entropy loss measured with LLaMa-3.1 series on OpenWebText, for different context length.

H.2 Eigenvalue-based Entropy Measurement

For at least 10,000 final-token vectors, PCA provides relative eigenvalues. The authors sum their logarithms as an entropy proxy, motivated by latent volume and Gaussian differential entropy. Figure 13 shows spectra and cross-length relative changes. This statistic depends on normalization, retained dimension, and covariance shape; constant scale conventions can shift it, and its interpretation as information requires the stated representation assumptions.

Refer to caption
Figure 13: Left: Relative Eigen Value Measured for the last token, for LLaMa-3.1-8B on a subset of OpenWebText. Right: relative increment of relative eigenvalues (for different context lengths measured). We can see that the relative eigenvalues approximately increase at a same scale.
Equation
S=\displaystyle S= log⁡Ω\displaystyle\log\Omega
=\displaystyle= log⁡V/hd​i​m​(V)​ where V is the volume in intrinsic space\displaystyle\log V/h^{dim(V)}\text{ where $V$ is the volume in intrinsic space}
=\displaystyle= ∑ilog⁡r​e​l​_​e​i​g​v​a​li/h\displaystyle\sum_{i}\log rel\_eigval_{i}/h
=\displaystyle= ∑ilog⁡r​e​l​_​e​i​g​v​a​li+C​o​n​s​t\displaystyle\sum_{i}\log rel\_eigval_{i}+Const

H.3 CE Loss vs. Eigenvalue-based Intrinsic Entropy

Figure 14 compares loss with log-eigenvalue sums over several leading subspaces and provides a correlation matrix. The reported correlations exceed 0.97 across many choices. This supports a stable empirical trend within the measured model/corpus, while highly correlated subspace statistics do not uniquely determine which features causally carry next-token information.

Original figure
Figure 14: Upper-left, Upper-right, Bottom-left: Cross Entropy Loss vs. measured Intrinsic Entropy with NN first Eigen Values: ∑i≤Nlog\sum_{i\leq N}\log rel_eig_val; Bottom-right: correlation between minus CE loss and ∑i≤Nlog\sum_{i\leq N}\log rel_eig_val. All experiments are for LLaMa-3.1-8B on a subset of OpenWebText. From the first three figure, we see CE loss is linear with the Entropy of certain subspaces. From the bottom-right figure, we see that Entropy measured in different subspaces are highly correlated (c​o​r​r>0.97corr>0.97), which are also highly correlated with the CE loss for Next Token Prediction.
Equation
RB​a​y​e​s≈−k∗S(Pl)+Const,R_{Bayes}\approx-k*S(P_{l})+Const,

Appendix I Experiment Settings

I.1 Natural Language Data

I.1.1 Optimal Context Length Experiments

Training uses AdamW with learning rate 6 × 10⁻⁴, weight decay 0.1, and 1,000 warmup iterations. For a fixed token budget, context variants use the same iteration count, approximately token count divided by 0.1M. Long OpenWebText documents are selected before splitting; validation contains 134M tokens. The paper reports roughly 300 GPU-hours on eight AMD MI250X devices for the context sweep. Figure 15 preserves absolute losses, including cases where changes between long contexts are small; repeated-seed uncertainty is not shown.

Original figure
Figure 15: Openwebtext subset, Validation Loss vs. Context Length, for different dataset sizes. Different curves represent different amount of training data used. A more readable figure can be found in Figure 1, where the minimum validation loss reachable for each training dataset size is subtracted.

I.1.2 Intrinsic Dimension Experiments

The representation measurements use 10,000 sufficiently long OpenWebText samples, final-token features from Llama-3.1-8B’s last layer, and about 100 reported GPU-hours on MI250X hardware. These details describe the empirical proxy, not a direct measurement of a unique language data manifold.

Appendix J Related Work

J.1 Enlarging Context Length for LMs

Related approaches extend usable context through rotary position embeddings, longer-context training, cache compression, linear attention, recurrent or state-space models, and staged short-to-long training Su et al., 2023; Gu and Dao, 2024. Those methods change representation or computation, whereas this paper studies the observed relation between context and prediction quality.

J.2 Irrelevant Long Context hurts performance of LMs

Earlier two-needle and retrieval studies show that irrelevant surrounding material can hurt performance Levy et al., 2024. The present analysis also considers contexts containing additional useful evidence, so relevance alone does not guarantee that a particular trained model will exploit the extra input.

J.3 Long Context in another field: Time Series Forecasting

Prior time-series work finds an optimal look-back horizon that grows with dataset size Shi et al., 2024. This motivates the language experiments, but the analogy is not itself evidence that the same quantitative law holds across domains.

J.4 Related Theories for Scaling Laws

The discussion situates the framework alongside empirical scaling laws, manifold-dimension explanations, feature-quantization models, and intrinsic-dimension accounts of adaptation. Its added focus is context length and the information represented by hidden states, with model/data scaling providing supporting assumptions.

Appendix K Intrinsic Dimension perspective: measurements in Intrinsic Space

This appendix develops an alternative dimension-based account and connects it to the entropy measurements through eigenvalue spectra.

K.1 Bayes Risk from an Intrinsic Dimension perspective: Assumptions

Assumptions posit a finite limiting effective dimension, growth with context, and constant predictive information per dimension. They imply an affine decrease of Bayes risk with effective dimension. This is stronger than simply observing that more context can provide more information; its relevance is assessed through the following measurements.

Equation 34
RB​a​y​e​s\displaystyle R_{Bayes} =H⁡(P,Pl)\displaystyle=H(P,P_{l}) (34)
=−s∗dim(l)+Const\displaystyle=-s*dim(l)+Const

K.2 Experimentally measure Intrinsic Dimension using PCA

PCA dimension is estimated by counting eigenvalues above a relative threshold. Thresholds from 0.002 to 0.25 produce different numerical dimensions but broadly linear relations with loss. Figure 16 retains spectra under two axis scalings and all threshold/loss plots. This is a threshold-dependent effective dimension, not a threshold-free topological dimension of language.

Refer to caption
Figure 16: Left figures: Relative Eigen Value for LLaMa-3.1-8B on a subset of OpenWebText, presented in different x-axis scales, with different context length visible to Language Model. Gray lines represent different thresholds we take to measure the intrinsic dimension of the current model. Right figures: Cross Entropy Loss vs. Measured Intrinsic Dimension. Each line represents a certain threshold used to measure ID in the intrinsic space of the used LLM. Different Measurements would give ID values that are linear w.r.t. each other, and they are all linear w.r.t. CE loss.
Equation
RB​a​y​e​s≈−s∗dim(l)+Const,R_{Bayes}\approx-s*dim(l)+Const,

K.3 MLP-based Synthetic Dataset: Intrinsic Dimension Experiments

An MLP separates context encoding from task selection, allowing PCA on the context feature. The constructed tasks assign one information unit per visible subtask, and fitted context–dimension and context–loss power laws report R² = 0.9997. Measured representation spectra show a cutoff related to visible tasks despite a fixed internal width of eighty. Figures 17–18 preserve the architecture, every spectrum, and threshold intervals. The extremely close fitted relationships reflect this deliberately structured test rather than a general estimate for natural language.

Refer to caption
Figure 17: Model trained on the proposed synthetic dataset; ⊕\oplus represents feature concatenation. Only the first ll bits are used as input to context MLP when the context length is set to ll. We conduct PCA on Context Feature to analyze the intrinsic dimension of input context bits for various context lengths.
Refer to caption
Figure 18: Relative eigen value vs. index, for models trained on different context length. Vertical lines: xl=I​D​(l)x_{l}=ID(l) and xl′=I​D​(l+1)x_{l}^{\prime}=ID(l+1). For example, a context length 2727 has 1616 subtasks visible, corresponding to 1616 bits in Intrinsic Space. Assuming 11 dimension in Intrinsic Space represents 11 bit, the leftmost purple rectangle drawn means a range of yt​h​r​e​s​h​o​l​dy_{threshold} that would provide an accurate estimation of I​D​(27)=16ID(27)=16 for context length 2727. There exists y∗y^{*} that would provide an estimation of I​DID for all context lengths, as shown in the figure.

K.4 Bridging the gap between Intrinsic Dimension explanation and Intrinsic Entropy explanation

If high-index eigenvalues share approximately the same exponential decay rate and differ mainly by a context-dependent scale, log-eigenvalue entropy and threshold-counted dimension both become affine functions of that scale’s logarithm. This explains why their measured relationships with cross-entropy can align, with slope tied to subspace dimension and decay rate. The derivation applies to the fitted spectral regime, not every possible representation distribution.

Equations 35–36
releigvall,i​d​x=releigvall, 0∗exp{−αl∗idx}, for certain context lengthreleigval_{l,\ idx}=releigval_{l,\ 0}*\exp\{-\alpha_{l}*idx\}\text{, for certain context length}
Ss​u​b​s​p​a​c​e​(l)\displaystyle S_{subspace}(l) =∑i​d​x∈{dimension of subspace}logreleigval∞,0γ(l)exp(−α∗idx)\displaystyle=\sum_{idx\in\{\text{dimension of subspace}\}}\log releigval_{\infty,0}\gamma(l)\exp(-\alpha*idx) (35)
=m​log⁡γ⁡(l)+C​o​n​s​t\displaystyle=m\log\gamma(l)+Const
releigval∞,0∗γ(l)∗exp{−α∗dim(l,thres)}=thres,releigval_{\infty,0}*\gamma(l)*\exp\{-\alpha*dim(l,thres)\}=thres,
LC​E=−Ss​u​b​s​p​a​c​e(l)+Const=−mα∗dim(l,thres)+Const(thres).L_{CE}=-S_{subspace}(l)+Const=-m\alpha*dim(l,thres)+Const(thres). (36)

Appendix L Disclosure of LLM Usage

The authors disclose using language models only to polish the writing.

Concept demonstration

Open image ↗