Literature
Display
← All papers

YaRN: Efficient Context Window Extension of Large Language Models

Bowen Peng · Jeffrey Quesnelle · Honglu Fan · Enrico Shippole

Preprint first posted · ICLR 2024 ↗

Language models9 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete twenty-page arXiv v3, revised 6 February 2026, including both appendices. The work appeared at ICLR 2024. The authors provide code and model resources. All original figures, tables, formal definitions and displayed formulas remain.

At a glance

Intuition. RoPE contains several position “clocks”: fast clocks distinguish nearby tokens, while slow clocks describe longer spans. Uniformly slowing every clock can blur local distinctions. YaRN treats frequency bands differently, rather like adjusting an equalizer’s bands separately.

Reading the math. The extension factor s compares the target window with the original window. High-frequency rotations are retained, low-frequency rotations are stretched, and intermediate frequencies blend those treatments. A second adjustment rescales attention scores, controlling how concentrated their softmax weights become. This temperature change modifies the attention distribution, not the text length. The choices are supported by the reported training and evaluations, rather than a universal long-context guarantee.

YaRN combines frequency-dependent RoPE interpolation with attention-temperature scaling to extend pretrained language models’ context. Experiments adapt Llama 2 to 128k-token evaluation using at most 64k-token training examples. Dynamic scaling also offers a narrower, inference-only extension. The headline token and training-step savings depend on the comparison protocol; the results do not establish universal preservation of short-context quality.

1 Introduction

Longer inputs expose rotary models to relative phases absent during pretraining. YaRN builds on Position Interpolation, NTK-aware scaling, NTK-by-parts and dynamic scaling, documenting the relationship to previously unpublished community work. Figure 1 preserves that lineage. The central contribution combines different treatment of frequency bands with a simple attention-logit adjustment.

Refer to caption
Figure 1: An outline of the relationship between different interpolation methods.

2 Background and Related Work

The notation separates a change to token positions from a change to the frequencies used to rotate query/key pairs.

2.1 Rotary Position Embeddings

For an even-dimensional attention head, pair real coordinates into complex coordinates. RoPE multiplies each pair by a position-dependent phase; query/key inner products then depend on relative displacement. Equations 1–7 retain the real/complex correspondence and the general form using position map g and frequency map h. The original frequency base is 10,000.

Equations 1–7
𝒒m=fq​(𝒙m,m)∈ℝ|D|,𝒌n=fk​(𝒙n,n)∈ℝ|D|.\displaystyle{\bm{q}}_{m}=f_{q}({\bm{x}}_{m},m)\in\mathbb{R}^{|D|},~{\bm{k}}_{n}=f_{k}({\bm{x}}_{n},n)\in\mathbb{R}^{|D|}. (1)
softmax​(𝒒mT​𝒌n|D|),\text{softmax}(\dfrac{{\bm{q}}_{m}^{T}{\bm{k}}_{n}}{\sqrt{|D|}}), (2)
ℝ|D|≅ℂ|D|/2\mathbb{R}^{|D|}\cong\mathbb{C}^{|D|/2}
((𝒙m)1,⋯,(𝒙m)|D|)↦((𝒙m)1+i​(𝒙m)2,⋯,((𝒙m)|D|−1+i​(𝒙m)|D|)),\displaystyle\big(({\bm{x}}_{m})_{1},\cdots,({\bm{x}}_{m})_{|D|}\big)\mapsto\big(({\bm{x}}_{m})_{1}+i({\bm{x}}_{m})_{2},\cdots,(({\bm{x}}_{m})_{|D|-1}+i({\bm{x}}_{m})_{|D|})\big), (3)
((qm)1,⋯,(qm)|D|)↦((qm)1+i​(qm)2,⋯,((qm)|D|−1+i​(qm)|D|)).\displaystyle\big((\textbf{q}_{m})_{1},\cdots,(\textbf{q}_{m})_{|D|}\big)\mapsto\big((\textbf{q}_{m})_{1}+i(\textbf{q}_{m})_{2},\cdots,((\textbf{q}_{m})_{|D|-1}+i(\textbf{q}_{m})_{|D|})\big). (4)
𝑾q,𝑾k:ℝ|D|→ℝ|D|.{\bm{W}}_{q},{\bm{W}}_{k}:\mathbb{R}^{|D|}\rightarrow\mathbb{R}^{|D|}.
f𝑾​(𝒙m,m,𝜽)=ei​m​𝜽​𝑾​𝒙m,f_{{\bm{W}}}({\bm{x}}_{m},m,\bm{\theta})=e^{im\bm{\theta}}{\bm{W}}{\bm{x}}_{m}, (5)
fq=f𝑾q,fk=f𝑾k.f_{q}=f_{{\bm{W}}_{q}},~f_{k}=f_{{\bm{W}}_{k}}. (6)
f𝑾′​(𝒙m,m,𝜽)=f𝑾​(𝒙m,g⁡(m),𝒉⁡(𝜽)),\displaystyle f^{\prime}_{{\bm{W}}}({\bm{x}}_{m},m,\bm{\theta})=f_{{\bm{W}}}({\bm{x}}_{m},g(m),{\bm{h}}(\bm{\theta})), (7)

2.2 Additional notations

Let s=L′/L be the target-to-pretrained context ratio. A frequency’s wavelength is 2π/θ, and L divided by that wavelength counts its rotations across the trained window. This count guides which bands should be interpolated.

Equation 8
λd=2​πθd=2​π​b2​d|D|.\displaystyle\lambda_{d}=\dfrac{2\pi}{\theta_{d}}=2\pi b^{\frac{2d}{|D|}}. (8)

2.3 Related work

PI compresses positions into the trained interval. Equation 9 prints g(m)=s·m, inconsistent with that description and with the correct inverse mapping mL/L′ in Appendix A.1; both original formulas are preserved. The paper excludes ReRoPE and LM-Infinite from its direct comparison because they modify attention and did not fit its FlashAttention-2 implementation assumptions.

Equation 9
g⁡(m)=s⋅m,𝒉⁡(𝜽)=𝜽,g(m)=s\cdot m,~{\bm{h}}(\bm{\theta})=\bm{\theta}, (9)

3 Methodology

Uniform interpolation crowds together high-frequency phases that distinguish nearby positions. YaRN aims to preserve this local information while reducing extrapolation in lower-frequency bands. The NTK terminology motivates the design; these transformations are not proved to preserve a transformer’s full neural tangent kernel.

3.1 Loss of High Frequency information - "NTK-aware" interpolation

NTK-aware interpolation changes the RoPE base so that lower frequencies are stretched more than higher frequencies. It can extend an unfine-tuned model more effectively than uniform PI in the reported tests, but its nominal scale need not equal its usable context length. Some coordinates still extrapolate, and selecting a useful base remains empirical.

3.2 Loss of Relative Local Distances - "NTK-by-parts" interpolation

High-frequency coordinates complete many rotations during pretraining and can encode local relative distance; very low frequencies see less than one rotation and may act more like absolute-position features. NTK-by-parts leaves sufficiently high frequencies unchanged, fully interpolates low frequencies, and linearly blends the two treatments in between. Definition 1 retains the precise ramp and frequency map. For the tested LLaMA models, the chosen rotation-count thresholds are α=1 and β=32; these are empirical settings.

Equations 10–11
r⁡(d)=Lλd=L2​π​b2​d|D|.r(d)=\dfrac{L}{\lambda_{d}}=\dfrac{L}{2\pi b^{\frac{2d}{|D|}}}. (10)
γ⁡(r)={0,if ​r<α1,if ​r>βr−αβ−α,otherwise.\displaystyle\gamma(r)=\begin{cases}0,&\text{if }r<\alpha\\ 1,&\text{if }r>\beta\\ \dfrac{r-\alpha}{\beta-\alpha},&\text{otherwise}.\end{cases} (11)
Definition 1

The "NTK-by-parts" interpolation is a modification of RoPE using Eq. 7 with the following functions‡‡ ‡ The interpolation by linear ramp on hh may have alternatives, such as a harmonic mean over θd/s\theta_{d}/s and θd\theta_{d} converted from a linear interpolation on wavelengths. The choice of hh here was for the simplicity of implementation, but both would work..

Equations 12–13
g⁡(m)\displaystyle g(m) =m\displaystyle=m (12)
h⁡(θd)\displaystyle h(\theta_{d}) =(1−γ⁡(r⁡(d)))​θds+γ⁡(r⁡(d))​θd.\displaystyle=\Big(1-\gamma\big(r(d)\big)\Big)\frac{\theta_{d}}{s}+\gamma\big(r(d)\big)\theta_{d}. (13)

3.3 YaRN

Divide attention logits by temperature t, equivalently scaling both queries and keys by √(1/t). Combining this with NTK-by-parts gives YaRN (Definition 2). The recommended multiplier is 1+.1 ln s, fitted on unfine-tuned LLaMA 7B/13B/33B/65B and then used for Llama 2. Scaling cached rotary embeddings makes the fixed-scale implementation inexpensive, but the empirical formula is not a universal optimal-temperature law.

Equation 14
softmax​(𝒒mT​𝒌nt​|D|).\text{softmax}\left(\dfrac{{\bm{q}}_{m}^{T}{\bm{k}}_{n}}{t\sqrt{|D|}}\right). (14)
Definition 2

By the "YaRN method", we refer to a combination of the attention scaling in Eq. 14 and the "NTK-by-parts" interpolation introduced in Section 3.2.

Equation 15
1t=0.1​ln⁡(s)+1.\displaystyle\sqrt{\frac{1}{t}}=0.1\ln({s})+1. (15)

3.4 Dynamic Scaling - "Dynamic NTK" interpolation

Dynamic scaling uses s=max(1,current sequence length/L) instead of a fixed target ratio, increasing the extension gradually as generation continues. This avoids the abrupt out-of-range behavior of a fixed scale in the displayed tests. Because changing frequencies changes every key’s rotary phase, a correct implementation must retain pre-RoPE keys or otherwise recompute their rotations; blindly reusing already-rotated cached keys is inconsistent.

4 Experiments

The experiments separate fixed-scale fine-tuning, ablations on the original LLaMA model, and inference-only dynamic scaling.

4.1 Training

Llama 2 7B/13B start with 4k context. Train the s=16 models for 400 steps on contiguous 64k-token PG-19 chunks, then initialize s=32 models from those checkpoints and train 200 more steps, still on 64k chunks. Global batch is 64. AdamW uses learning rate 2×10⁻⁵, no weight decay, betas (.9,.95), and twenty warmup steps. FSDP and FlashAttention-2 support training. The ablation instead extends original 2k-context LLaMA 7B to 32k for 400 steps.

Refer to caption
Refer to caption
Figure 2: Training loss curves for the LLaMA 7B model extended to 32k context size using different interpolation techniques. The graph on the right is zoomed in.

4.2 Long Sequence Language Modeling

Evaluate ten randomly selected Proof-pile test documents with at least 128k tokens, using truncation lengths from 2k to 128k and a sliding-window stride of 256. The 128k YaRN checkpoints reach perplexity 2.37 (7B) and 2.24 (13B) at the full window. The 64k-target checkpoints exceed 10 at 128k. Thus the successful 64k-training-to-128k-evaluation result includes a changed target scale and 200 additional steps; it is not unchanged extrapolation by the 64k model. Figure 3 and Appendix B preserve the ablations and longer-window curves.

Model Extension Fine- Training Extension Evaluation Context Window Size
Size Method tuned Steps Scale ss 8192 16384 32768 65536 131072
7B YaRN ✓ 400 4k ×\times 16 3.51 2.99 2.65 2.42 >101>10^{1}
7B YaRN ✓ 400+200 4k ×\times 32 3.56 3.04 2.70 2.45 2.37
13B YaRN ✓ 400 4k ×\times 16 3.25 2.79 2.50 2.29 >101>10^{1}
13B YaRN ✓ 400+200 4k ×\times 32 3.29 2.83 2.53 2.31 2.24
Table 1: Sliding window perplexity (S=256S=256) of ten 128k Proof-pile documents over Llama 2 models extended via YaRN. We show successful context size extrapolation and transfer learning from 64k to 128k given only 64k context as training data.
Refer to caption
Refer to caption
Figure 3: Sliding window perplexity (S=256S=256) of ten 128k Proof-pile documents and passkey retrieval accuracy at different prompt lengths for finetuned LLaMA 7B models fine-tuned to 32k context for 400 steps using different interpolation techniques. YaRN outperforms other interpolation methods given the same training budget.

4.3 Passkey Retrieval

Hide a five-digit passkey at a uniformly random location in distracting text, following Landmark Attention. The 32k ablation uses fifty trials per prompt length from 2k to 32k. YaRN retains strong retrieval across the plotted interval. This synthetic retrieval test does not establish comparable performance on long-document reasoning.

4.4 Standardized Benchmarks

Evaluate ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot) and TruthfulQA (zero-shot). Results are mixed: in the LLaMA 7B ablation, NTK-by-parts beats YaRN on ARC and MMLU, while YaRN wins HellaSwag and TruthfulQA. Against the original Llama 2 13B, the 128k checkpoint’s MMLU score falls from 55.8 to 51.9. The source’s “minimal degradation” should therefore be read task by task, rather than as lossless preservation.

Extension Fine- Extension ARC-c Hellaswag MMLU TruthfulQA
Method tuned Scale ss
None ✗ - 51.0 77.8 35.7 34.3
PI ✓ 2k ×\times 16 44.8 70.2 25.9 34.1
NTK-aware ✓ 2k ×\times 16 47.4 73.9 27.7 32.6
NTK-by-parts ✓ 2k ×\times 16 48.5 76.6 32.7 33.4
YaRN ✓ 2k ×\times 16 48.1 77.2 30.0 35.1
Table 2: Performance of context window extensions methods, fine-tuned for 400 steps, on the Hugging Face Open LLM benchmark suite compared with original LLaMA 7B baselines.
Model Extension Fine- Extension ARC-c Hellaswag MMLU TruthfulQA
Size Method tuned Scale ss
7B None ✗ - 53.1 77.8 43.8 39.0
7B YaRN ✓ 4k ×\times 16 52.3 78.8 42.5 38.2
7B YaRN ✓ 4k ×\times 32 52.1 78.4 41.7 37.3
13B None ✗ - 59.4 82.1 55.8 37.4
13B YaRN ✓ 4k ×\times 16 58.1 82.3 52.8 37.8
13B YaRN ✓ 4k ×\times 32 58.0 82.2 51.9 37.3
Table 3: Performance of YaRN on the Hugging Face Open LLM benchmark suite compared with original Llama 2 baselines.

4.5 Computational Efficiency

For a fixed scale, precomputing modified rotary embeddings adds negligible per-forward work relative to the other interpolation schemes; longer full attention still costs substantially more. Table 4 reports A100-hours, but its models, target lengths and training data differ. Those heterogeneous totals do not isolate the positional method’s efficiency. Appendix B.2 provides the more direct 400-versus-1,000-step comparison.

Model Model Extension Extension Effective Training Time in
Size Name Method Scale ss Context GPU-Hours (A100)
7B LLaMA YaRN YaRN 2k ×\times 16 32k 128
7B Llama 2 YaRN YaRN 4k ×\times 16 64k 256
7B Llama 2 YaRN YaRN 4k ×\times 32 128k 256 + 128
7B (Chen et al., 2023) PI 2k ×\times 8 16k 640
7B (Together.ai, 2023) PI 4k ×\times 8 32k ?
7B (Xiong et al., 2023) NTK-aware 4k ×\times 44.2 ≈\approx 50k 64000
7B (Rozière et al., 2023) NTK-aware 4k ×\times 88.6 ≈\approx 100k 6400
Table 4: Comparison of training time in A100-hours for different open and closed models using different extension methods.

5 Conclusion

Frequency-band interpolation plus attention rescaling offers a practical way to adapt existing rotary models to longer context. The demonstrated benefits are empirical and checkpoint-dependent: short-task scores can decline, retrieval and perplexity measure different abilities, and fixed-target models can fail beyond their chosen scale. Dynamic scaling reduces that boundary in limited inference-only tests.

6 Reproducibility

The authors release implementation resources and describe training on PG-19. The retained appendices specify the scaling rules and additional evaluations. This edition reviews the reported experiments; it does not claim to reproduce their training runs.

Appendix A Additional details on interpolation methods

The appendix expands the exact position/frequency mappings and the temperature sweep used to motivate YaRN’s scaling rule.

A.1 Position Interpolation

Equation 16 states PI as g(m)=mL/L′ and h(θ)=θ: positions are compressed by the extension ratio. This agrees with the method’s verbal description and exposes the reversed scale in Equation 9.

Equation 16
f𝑾′​(𝒙m,m,𝜽)=f𝑾​(𝒙m,m​LL′,𝜽),\displaystyle f^{\prime}_{\bm{W}}\left({\bm{x}}_{m},m,\bm{\theta}\right)=f_{\bm{W}}\left({\bm{x}}_{m},\dfrac{mL}{L^{\prime}},\bm{\theta}\right), (16)

A.2 Details of "NTK-aware" interpolation

Definition 3 changes the frequency base to b′=b·s^(D/(D−2)). The derivation matches the lowest frequency to linear interpolation while leaving the highest unchanged, subject to the source’s coordinate-pair indexing convention. Intermediate frequencies receive different scaling. The nominal factor still requires empirical calibration; the paper reports poor fine-tuned NTK-aware performance at the 32k endpoint even though some shorter-window scores are better.

Definition 3

The "NTK-aware" interpolation is a modification of RoPE using Eq. 7 with the following functions, given ss as the scale factor.

Equations 17–18
g⁡(m)\displaystyle g(m) =m\displaystyle=m (17)
h⁡(θd)\displaystyle h(\theta_{d}) =b′−2d/|D|,\displaystyle={b^{\prime}}^{-2d/|D|}, (18)

where

Equations 19–20
b′\displaystyle{b^{\prime}} =b⋅s|D||D|−2\displaystyle=b\cdot s^{\frac{|D|}{|D|-2}} (19)
s\displaystyle{s} =L′L.\displaystyle=\frac{L^{\prime}}{L}. (20)
Equations 21–22
b′|D|−2|D|=s⋅b|D|−2|D|.{b^{\prime}}^{\frac{|D|-2}{|D|}}=s\cdot b^{\frac{|D|-2}{|D|}}. (21)
b′=b⋅s|D||D|−2.{b^{\prime}}=b\cdot s^{\frac{|D|}{|D|-2}}. (22)

A.3 The impact of pre-softmax scaling of YaRN on perplexity

Sweep the pre-softmax multiplier on 896 RedPajama documents of 16k tokens at s=8; the recommended multiplier is approximately 1.208. Figures 4–6 show mean perplexity with a standard-deviation band, relative changes in eight 2,048-token segments, and counts of each segment’s best multiplier. The plot title inside Figure 4 says 56 documents while the caption and prose say 896; both source statements remain. These sweeps motivate an empirical rule, not a proof that the same temperature is optimal for every dataset or position.

Equation 23
1t=0.1​ln⁡(s)+1≈1.208.\sqrt{\frac{1}{t}}=0.1\ln({s})+1\approx 1.208. (23)
Refer to caption
Figure 4: Fix s=8s=8, compare the LLaMA 7b perplexity on 896896 1616k-token documents over different scaling 1/t1/\sqrt{t}. The shaded area represents 11 standard deviation (68%68\%).
Equation 24
ppl​(t)−ppl​(t=1)ppl​(t=1)\dfrac{\text{ppl}(t)-\text{ppl}(t=1)}{\text{ppl}(t=1)} (24)
Refer to caption
Figure 5: Fix s=8s=8, compare the mean of perplexity change percentages ppl​(t)−ppl​(t=1)ppl​(t=1)\dfrac{\text{ppl}(t)-\text{ppl}(t=1)}{\text{ppl}(t=1)} at different segments of token positions on 896896 1616k-token documents over different scaling 1/t1/\sqrt{t}.
Refer to caption
Figure 6: The sample counts (out of the 896896 samples) with minimal perplexity at a given 1/t1/\sqrt{t} for a given segment of token positions over the 1616k-token range.

Appendix B Additional tables and charts

The full tables retain all scales, checkpoints, evaluation lengths and comparison methods, including failures outside the target window.

B.1 Ablation Study

Table 5 compares fixed scales 2/4/8/16, dynamic scaling, and 400-step fine-tuning. At the fine-tuned 32k endpoint, perplexity is 2.77 for YaRN, 2.81 for NTK-by-parts, 3.57 for PI and 8.49 for NTK-aware. YaRN is not best in every shorter-window cell: NTK-by-parts is slightly lower at 2k and 4k. Comparing different evaluation lengths can also confound claims of improvement over the original model.

Extension Fine- Training Extension Evaluation Context Window Size
Method tuned Steps Scale ss 2048 4096 8192 16384 32768
None ✗ - - 4.05 - - - -
PI ✗ - 2k ×\times 2 4.36 3.90 - - -
NTK-aware ✗ - 2k ×\times 2 4.08 5.97 - - -
NTK-by-parts ✗ - 2k ×\times 2 4.12 3.71 - - -
YaRN ✗ - 2k ×\times 2 4.07 3.67 - - -
PI ✗ - 2k ×\times 4 7.09 6.39 6.18 - -
NTK-aware ✗ - 2k ×\times 4 4.27 3.84 >101>10^{1} - -
NTK-by-parts ✗ - 2k ×\times 4 4.39 4.03 4.11 - -
YaRN ✗ - 2k ×\times 4 4.19 3.77 3.65 - -
PI ✗ - 2k ×\times 8 >101>10^{1} >101>10^{1} >101>10^{1} >101>10^{1} -
NTK-aware ✗ - 2k ×\times 8 4.64 4.27 4.24 >101>10^{1} -
NTK-by-parts ✗ - 2k ×\times 8 4.98 4.91 5.33 5.79 -
YaRN ✗ - 2k ×\times 8 4.37 3.95 3.81 3.33 -
PI ✗ - 2k ×\times 16 >102>10^{2} >102>10^{2} >102>10^{2} >102>10^{2} >102>10^{2}
NTK-aware ✗ - 2k ×\times 16 5.23 5.02 5.22 6.85 >101>10^{1}
NTK-by-parts ✗ - 2k ×\times 16 6.04 7.54 >101>10^{1} >101>10^{1} >101>10^{1}
YaRN ✗ - 2k ×\times 16 4.61 4.24 4.18 3.66 3.45
PI ✗ - Dynamic 4.05 3.90 6.18 >101>10^{1} >102>10^{2}
NTK-aware ✗ - Dynamic 4.05 5.97 >101>10^{1} >101>10^{1} >101>10^{1}
NTK-by-parts ✗ - Dynamic 4.05 3.71 4.11 5.79 >101>10^{1}
YaRN ✗ - Dynamic 4.05 3.67 3.65 3.33 3.45
PI ✓ 400 2k ×\times 16 5.70 4.95 4.64 3.97 3.57
NTK-aware ✓ 400 2k ×\times 16 4.39 3.92 3.73 3.21 8.49
NTK-by-parts ✓ 400 2k ×\times 16 4.14 3.75 3.62 3.12 2.81
YaRN ✓ 400 2k ×\times 16 4.19 3.77 3.30 3.09 2.77
Table 5: Sliding window perplexity (S=256S=256) of ten 128k Proof-pile documents over the LLaMA 7B model extended via different methods.

B.2 Training Efficiency of YaRN

For Llama 2 7B extended from 4k to 8k, YaRN’s 400-step run reaches 8k perplexity 3.35, close to PI’s 3.34 after 1,000 steps. This is the basis of the 2.5× fewer-steps comparison. The PI checkpoint is LLongMA-2, whose footnote specifies RedPajama training, whereas the main YaRN protocol uses PG-19; the comparison does not isolate only the interpolation rule.

Model Extension Fine- Training Extension Evaluation Context Window Size
Size Method tuned Steps Scale ss 2048 4096 6144 8192
7B None ✗ - - 4.00 3.58 - -
7B PI ✗ - 4k ×\times 2 4.30 3.84 3.83 3.65
7B YaRN ✗ - 4k ×\times 2 4.03 3.61 3.60 3.49
7B PI ✓ 1000 4k ×\times 2 3.92 3.51 3.51 3.34
7B YaRN ✓ 400 4k ×\times 2 3.91 3.50 3.51 3.35
Table 6: Sliding window perplexity (S=256S=256) of ten 128k Proof-pile documents over the Llama-2 7b model extended via PI and YaRN with different training steps. YaRN obtains comparable results to PI using much less training steps.

B.3 Comparing the perplexity of various methods over a sliding window

Compare YaRN with Together’s 32k PI model and Code Llama’s longer-window NTK model. At 128k, YaRN’s 7B/13B perplexities are 2.37/2.24 versus Code Llama’s 2.71/2.54. These checkpoints differ in training data and specialization. Table 7 specifies ten 128k Proof-pile documents; Figure 7’s “1.28M-token … documents” caption is retained, without interpreting it as evidence for a single 1.28M-token context.

Model Model Context Extension Evaluation Context Window Size
Size Name Window Method 8192 32768 65536 98304 131072
7B Together 32k PI 3.50 2.64 >102>10^{2} >103>10^{3} >104>10^{4}
7B Code Llama 100k NTK 3.71 2.74 2.55 2.54 2.71
7B YaRN (s=16s=16) 64k YaRN 3.51 2.65 2.42 >101>10^{1} >101>10^{1}
7B YaRN (s=32s=32) 128k YaRN 3.56 2.70 2.45 2.36 2.37
13B Code Llama 100k NTK 3.54 2.63 2.41 2.37 2.54
13B YaRN (s=16s=16) 64k YaRN 3.25 2.50 2.29 >101>10^{1} >101>10^{1}
13B YaRN (s=32s=32) 128k YaRN 3.29 2.53 2.31 2.23 2.24
Table 7: Sliding window perplexity (S=256S=256) of ten 128k Proof-pile documents truncated to evaluation context window size
Complete original Figure 7: sliding-window perplexity curves and legend
Figure 7: Sliding window perplexity (S=256S=256) of a 1.28M-token Proof-pile documents truncated to the context window size of the fine-tuned model

B.4 GovReport evaluations

Use fifty untruncated GovReport test documents of at least 16k tokens, a fixed maximum context of 32k and stride 256, without dynamic scaling. YaRN-64k scores 3.59/3.35 for 7B/13B, slightly lower than YaRN-128k’s 3.64/3.39. These are language-model perplexities on reports, not summarization-quality scores.

Model Model Context Extension Perplexity
Size Name Window Method
7B Together 32k PI 3.67
7B Code Llama 100k NTK 4.44
7B YaRN (s=16s=16) 64k YaRN 3.59
7B YaRN (s=32s=32) 128k YaRN 3.64
13B Code Llama 100k NTK 4.22
13B YaRN (s=16s=16) 64k YaRN 3.35
13B YaRN (s=32s=32) 128k YaRN 3.39
Table 8: Sliding window perplexity (S=256S=256) of 50 long GovReport documents with a fixed context window size of 32k

B.5 Passkey Retrieval

For 64k/128k checkpoints, run ten random-location passkey trials at each tested length from 8k to 128k. “Passkey context” is the largest tested length reaching at least 80% accuracy; the reported accuracy averages all tested lengths up to that point. The 128k YaRN models score 99.4% by this aggregate definition, not a guarantee for every position. Retrieval can remain strong where perplexity worsens; the extra 200 training steps also confound comparisons between the two YaRN scales.

Model Model Scaling Context Training Extension Passkey Passkey
Size Name Factor (s)(s) Window Data Context Method Context Accuracy
7B Together 4 32k 32k PI 32k 100%
7B Code Llama 88.6 100k 16k NTK 112k 94.3%
7B YaRN 16 64k 64k YaRN 64k 96.3%
7B YaRN 32 128k 64k YaRN 128k 99.4%
13B Code Llama 88.6 100k 16k NTK 128k 99.4%
13B YaRN 16 64k 64k YaRN 64k 97.5%
13B YaRN 32 128k 64k YaRN 128k 99.4%
Table 9: Passkey retrieval performance of various models. The passkey context denotes the maximum tested context window size where the accuracy of passkey retrieval was >=80%>=80\%, and the passkey accuracy is the average accuracy of passkey retrieval on all context sizes tested that were smaller or equal than the passkey context size.

B.6 Standardized Benchmarks

Table 10 repeats short-task scores alongside Together and Code Llama. YaRN generally preserves more broad benchmark performance than these particular long-context baselines, but its MMLU and some other scores still decline relative to original Llama 2. Code Llama’s code specialization makes this an imperfectly controlled comparison.

Model Model Context Extension ARC-c Hellaswag MMLU TruthfulQA
Size Name Window Method
7B Llama 2 4k None 53.1 77.8 43.8 39.0
7B Together 32k PI 47.6 76.1 43.3 39.2
7B Code Llama 100k NTK-a 39.9 60.8 31.1 37.8
7B YaRN (s=16s=16) 64k YaRN 52.3 78.8 42.5 38.2
7B YaRN (s=32s=32) 128k YaRN 52.1 78.4 41.7 37.3
13B Llama 2 4k None 59.4 82.1 55.8 37.4
13B Code Llama 100k NTK-a 40.9 63.4 32.8 43.8
13B YaRN (s=16s=16) 64k YaRN 58.1 82.3 52.8 37.8
13B YaRN (s=32s=32) 128k YaRN 58.0 82.2 51.9 37.3
Table 10: Performance of context window extensions methods on the Hugging Face Open LLM benchmark suite compared with original Llama 2 baselines

B.7 Dynamic scaling on models without any fine-tuning

Figure 8 applies dynamic PI and dynamic YaRN to an unfine-tuned Llama 2 model on one long GovReport sample, using stride 256. Both avoid the original RoPE curve’s blow-up beyond approximately 4k; dynamic YaRN has lower perplexity in the displayed extended interval. This single-sample demonstration is narrower than a general no-fine-tuning guarantee.

Refer to caption
Figure 8: The comparison between RoPE, Dynamic-PI and Dynamic-YaRN using Llama 2 on a long GovReport sample. This model has not been finetuned for long context.
Concept demonstration

Open image ↗