Overview
This edition follows the complete twenty-page arXiv v3, revised 6 February 2026, including both appendices. The work appeared at ICLR 2024. The authors provide code and model resources. All original figures, tables, formal definitions and displayed formulas remain.
At a glance
Intuition. RoPE contains several position “clocks”: fast clocks distinguish nearby tokens, while slow clocks describe longer spans. Uniformly slowing every clock can blur local distinctions. YaRN treats frequency bands differently, rather like adjusting an equalizer’s bands separately.
Reading the math. The extension factor s compares the target window with the original window. High-frequency rotations are retained, low-frequency rotations are stretched, and intermediate frequencies blend those treatments. A second adjustment rescales attention scores, controlling how concentrated their softmax weights become. This temperature change modifies the attention distribution, not the text length. The choices are supported by the reported training and evaluations, rather than a universal long-context guarantee.
YaRN combines frequency-dependent RoPE interpolation with attention-temperature scaling to extend pretrained language models’ context. Experiments adapt Llama 2 to 128k-token evaluation using at most 64k-token training examples. Dynamic scaling also offers a narrower, inference-only extension. The headline token and training-step savings depend on the comparison protocol; the results do not establish universal preservation of short-context quality.
1 Introduction
Longer inputs expose rotary models to relative phases absent during pretraining. YaRN builds on Position Interpolation, NTK-aware scaling, NTK-by-parts and dynamic scaling, documenting the relationship to previously unpublished community work. Figure 1 preserves that lineage. The central contribution combines different treatment of frequency bands with a simple attention-logit adjustment.
2 Background and Related Work
The notation separates a change to token positions from a change to the frequencies used to rotate query/key pairs.
2.1 Rotary Position Embeddings
For an even-dimensional attention head, pair real coordinates into complex coordinates. RoPE multiplies each pair by a position-dependent phase; query/key inner products then depend on relative displacement. Equations 1–7 retain the real/complex correspondence and the general form using position map g and frequency map h. The original frequency base is 10,000.
Equations 1–7
| (1) |
| (2) |
| (3) | |||
| (4) |
| (5) |
| (6) |
| (7) |
2.2 Additional notations
Let s=L′/L be the target-to-pretrained context ratio. A frequency’s wavelength is 2π/θ, and L divided by that wavelength counts its rotations across the trained window. This count guides which bands should be interpolated.
Equation 8
| (8) |
2.3 Related work
PI compresses positions into the trained interval. Equation 9 prints g(m)=s·m, inconsistent with that description and with the correct inverse mapping mL/L′ in Appendix A.1; both original formulas are preserved. The paper excludes ReRoPE and LM-Infinite from its direct comparison because they modify attention and did not fit its FlashAttention-2 implementation assumptions.
Equation 9
| (9) |
3 Methodology
Uniform interpolation crowds together high-frequency phases that distinguish nearby positions. YaRN aims to preserve this local information while reducing extrapolation in lower-frequency bands. The NTK terminology motivates the design; these transformations are not proved to preserve a transformer’s full neural tangent kernel.
3.1 Loss of High Frequency information - "NTK-aware" interpolation
NTK-aware interpolation changes the RoPE base so that lower frequencies are stretched more than higher frequencies. It can extend an unfine-tuned model more effectively than uniform PI in the reported tests, but its nominal scale need not equal its usable context length. Some coordinates still extrapolate, and selecting a useful base remains empirical.
3.2 Loss of Relative Local Distances - "NTK-by-parts" interpolation
High-frequency coordinates complete many rotations during pretraining and can encode local relative distance; very low frequencies see less than one rotation and may act more like absolute-position features. NTK-by-parts leaves sufficiently high frequencies unchanged, fully interpolates low frequencies, and linearly blends the two treatments in between. Definition 1 retains the precise ramp and frequency map. For the tested LLaMA models, the chosen rotation-count thresholds are α=1 and β=32; these are empirical settings.
Equations 10–11
| (10) |
| (11) |
Definition 1
The "NTK-by-parts" interpolation is a modification of RoPE using Eq. 7 with the following functions‡‡ ‡ The interpolation by linear ramp on may have alternatives, such as a harmonic mean over and converted from a linear interpolation on wavelengths. The choice of here was for the simplicity of implementation, but both would work..
Equations 12–13
| (12) | ||||
| (13) |
3.3 YaRN
Divide attention logits by temperature t, equivalently scaling both queries and keys by √(1/t). Combining this with NTK-by-parts gives YaRN (Definition 2). The recommended multiplier is 1+.1 ln s, fitted on unfine-tuned LLaMA 7B/13B/33B/65B and then used for Llama 2. Scaling cached rotary embeddings makes the fixed-scale implementation inexpensive, but the empirical formula is not a universal optimal-temperature law.
Equation 14
| (14) |
Definition 2
Equation 15
| (15) |
3.4 Dynamic Scaling - "Dynamic NTK" interpolation
Dynamic scaling uses s=max(1,current sequence length/L) instead of a fixed target ratio, increasing the extension gradually as generation continues. This avoids the abrupt out-of-range behavior of a fixed scale in the displayed tests. Because changing frequencies changes every key’s rotary phase, a correct implementation must retain pre-RoPE keys or otherwise recompute their rotations; blindly reusing already-rotated cached keys is inconsistent.
4 Experiments
The experiments separate fixed-scale fine-tuning, ablations on the original LLaMA model, and inference-only dynamic scaling.
4.1 Training
Llama 2 7B/13B start with 4k context. Train the s=16 models for 400 steps on contiguous 64k-token PG-19 chunks, then initialize s=32 models from those checkpoints and train 200 more steps, still on 64k chunks. Global batch is 64. AdamW uses learning rate 2×10⁻⁵, no weight decay, betas (.9,.95), and twenty warmup steps. FSDP and FlashAttention-2 support training. The ablation instead extends original 2k-context LLaMA 7B to 32k for 400 steps.
4.2 Long Sequence Language Modeling
Evaluate ten randomly selected Proof-pile test documents with at least 128k tokens, using truncation lengths from 2k to 128k and a sliding-window stride of 256. The 128k YaRN checkpoints reach perplexity 2.37 (7B) and 2.24 (13B) at the full window. The 64k-target checkpoints exceed 10 at 128k. Thus the successful 64k-training-to-128k-evaluation result includes a changed target scale and 200 additional steps; it is not unchanged extrapolation by the 64k model. Figure 3 and Appendix B preserve the ablations and longer-window curves.
| Model | Extension | Fine- | Training | Extension | Evaluation Context Window Size | ||||
|---|---|---|---|---|---|---|---|---|---|
| Size | Method | tuned | Steps | Scale | 8192 | 16384 | 32768 | 65536 | 131072 |
| 7B | YaRN | ✓ | 400 | 4k 16 | 3.51 | 2.99 | 2.65 | 2.42 | |
| 7B | YaRN | ✓ | 400+200 | 4k 32 | 3.56 | 3.04 | 2.70 | 2.45 | 2.37 |
| 13B | YaRN | ✓ | 400 | 4k 16 | 3.25 | 2.79 | 2.50 | 2.29 | |
| 13B | YaRN | ✓ | 400+200 | 4k 32 | 3.29 | 2.83 | 2.53 | 2.31 | 2.24 |
4.3 Passkey Retrieval
Hide a five-digit passkey at a uniformly random location in distracting text, following Landmark Attention. The 32k ablation uses fifty trials per prompt length from 2k to 32k. YaRN retains strong retrieval across the plotted interval. This synthetic retrieval test does not establish comparable performance on long-document reasoning.
4.4 Standardized Benchmarks
Evaluate ARC-Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot) and TruthfulQA (zero-shot). Results are mixed: in the LLaMA 7B ablation, NTK-by-parts beats YaRN on ARC and MMLU, while YaRN wins HellaSwag and TruthfulQA. Against the original Llama 2 13B, the 128k checkpoint’s MMLU score falls from 55.8 to 51.9. The source’s “minimal degradation” should therefore be read task by task, rather than as lossless preservation.
| Extension | Fine- | Extension | ARC-c | Hellaswag | MMLU | TruthfulQA |
|---|---|---|---|---|---|---|
| Method | tuned | Scale | ||||
| None | ✗ | - | 51.0 | 77.8 | 35.7 | 34.3 |
| PI | ✓ | 2k 16 | 44.8 | 70.2 | 25.9 | 34.1 |
| NTK-aware | ✓ | 2k 16 | 47.4 | 73.9 | 27.7 | 32.6 |
| NTK-by-parts | ✓ | 2k 16 | 48.5 | 76.6 | 32.7 | 33.4 |
| YaRN | ✓ | 2k 16 | 48.1 | 77.2 | 30.0 | 35.1 |
| Model | Extension | Fine- | Extension | ARC-c | Hellaswag | MMLU | TruthfulQA |
|---|---|---|---|---|---|---|---|
| Size | Method | tuned | Scale | ||||
| 7B | None | ✗ | - | 53.1 | 77.8 | 43.8 | 39.0 |
| 7B | YaRN | ✓ | 4k 16 | 52.3 | 78.8 | 42.5 | 38.2 |
| 7B | YaRN | ✓ | 4k 32 | 52.1 | 78.4 | 41.7 | 37.3 |
| 13B | None | ✗ | - | 59.4 | 82.1 | 55.8 | 37.4 |
| 13B | YaRN | ✓ | 4k 16 | 58.1 | 82.3 | 52.8 | 37.8 |
| 13B | YaRN | ✓ | 4k 32 | 58.0 | 82.2 | 51.9 | 37.3 |
4.5 Computational Efficiency
For a fixed scale, precomputing modified rotary embeddings adds negligible per-forward work relative to the other interpolation schemes; longer full attention still costs substantially more. Table 4 reports A100-hours, but its models, target lengths and training data differ. Those heterogeneous totals do not isolate the positional method’s efficiency. Appendix B.2 provides the more direct 400-versus-1,000-step comparison.
| Model | Model | Extension | Extension | Effective | Training Time in |
| Size | Name | Method | Scale | Context | GPU-Hours (A100) |
| 7B | LLaMA YaRN | YaRN | 2k 16 | 32k | 128 |
| 7B | Llama 2 YaRN | YaRN | 4k 16 | 64k | 256 |
| 7B | Llama 2 YaRN | YaRN | 4k 32 | 128k | 256 + 128 |
| 7B | (Chen et al., 2023) | PI | 2k 8 | 16k | 640 |
| 7B | (Together.ai, 2023) | PI | 4k 8 | 32k | ? |
| 7B | (Xiong et al., 2023) | NTK-aware | 4k 44.2 | 50k | 64000 |
| 7B | (Rozière et al., 2023) | NTK-aware | 4k 88.6 | 100k | 6400 |
5 Conclusion
Frequency-band interpolation plus attention rescaling offers a practical way to adapt existing rotary models to longer context. The demonstrated benefits are empirical and checkpoint-dependent: short-task scores can decline, retrieval and perplexity measure different abilities, and fixed-target models can fail beyond their chosen scale. Dynamic scaling reduces that boundary in limited inference-only tests.
6 Reproducibility
The authors release implementation resources and describe training on PG-19. The retained appendices specify the scaling rules and additional evaluations. This edition reviews the reported experiments; it does not claim to reproduce their training runs.
Appendix A Additional details on interpolation methods
The appendix expands the exact position/frequency mappings and the temperature sweep used to motivate YaRN’s scaling rule.
A.1 Position Interpolation
Equation 16 states PI as g(m)=mL/L′ and h(θ)=θ: positions are compressed by the extension ratio. This agrees with the method’s verbal description and exposes the reversed scale in Equation 9.
Equation 16
| (16) |
A.2 Details of "NTK-aware" interpolation
Definition 3 changes the frequency base to b′=b·s^(D/(D−2)). The derivation matches the lowest frequency to linear interpolation while leaving the highest unchanged, subject to the source’s coordinate-pair indexing convention. Intermediate frequencies receive different scaling. The nominal factor still requires empirical calibration; the paper reports poor fine-tuned NTK-aware performance at the 32k endpoint even though some shorter-window scores are better.
Definition 3
Equations 21–22
| (21) |
| (22) |
A.3 The impact of pre-softmax scaling of YaRN on perplexity
Sweep the pre-softmax multiplier on 896 RedPajama documents of 16k tokens at s=8; the recommended multiplier is approximately 1.208. Figures 4–6 show mean perplexity with a standard-deviation band, relative changes in eight 2,048-token segments, and counts of each segment’s best multiplier. The plot title inside Figure 4 says 56 documents while the caption and prose say 896; both source statements remain. These sweeps motivate an empirical rule, not a proof that the same temperature is optimal for every dataset or position.
Equation 23
| (23) |
Equation 24
| (24) |
Appendix B Additional tables and charts
The full tables retain all scales, checkpoints, evaluation lengths and comparison methods, including failures outside the target window.
B.1 Ablation Study
Table 5 compares fixed scales 2/4/8/16, dynamic scaling, and 400-step fine-tuning. At the fine-tuned 32k endpoint, perplexity is 2.77 for YaRN, 2.81 for NTK-by-parts, 3.57 for PI and 8.49 for NTK-aware. YaRN is not best in every shorter-window cell: NTK-by-parts is slightly lower at 2k and 4k. Comparing different evaluation lengths can also confound claims of improvement over the original model.
| Extension | Fine- | Training | Extension | Evaluation Context Window Size | ||||
|---|---|---|---|---|---|---|---|---|
| Method | tuned | Steps | Scale | 2048 | 4096 | 8192 | 16384 | 32768 |
| None | ✗ | - | - | 4.05 | - | - | - | - |
| PI | ✗ | - | 2k 2 | 4.36 | 3.90 | - | - | - |
| NTK-aware | ✗ | - | 2k 2 | 4.08 | 5.97 | - | - | - |
| NTK-by-parts | ✗ | - | 2k 2 | 4.12 | 3.71 | - | - | - |
| YaRN | ✗ | - | 2k 2 | 4.07 | 3.67 | - | - | - |
| PI | ✗ | - | 2k 4 | 7.09 | 6.39 | 6.18 | - | - |
| NTK-aware | ✗ | - | 2k 4 | 4.27 | 3.84 | - | - | |
| NTK-by-parts | ✗ | - | 2k 4 | 4.39 | 4.03 | 4.11 | - | - |
| YaRN | ✗ | - | 2k 4 | 4.19 | 3.77 | 3.65 | - | - |
| PI | ✗ | - | 2k 8 | - | ||||
| NTK-aware | ✗ | - | 2k 8 | 4.64 | 4.27 | 4.24 | - | |
| NTK-by-parts | ✗ | - | 2k 8 | 4.98 | 4.91 | 5.33 | 5.79 | - |
| YaRN | ✗ | - | 2k 8 | 4.37 | 3.95 | 3.81 | 3.33 | - |
| PI | ✗ | - | 2k 16 | |||||
| NTK-aware | ✗ | - | 2k 16 | 5.23 | 5.02 | 5.22 | 6.85 | |
| NTK-by-parts | ✗ | - | 2k 16 | 6.04 | 7.54 | |||
| YaRN | ✗ | - | 2k 16 | 4.61 | 4.24 | 4.18 | 3.66 | 3.45 |
| PI | ✗ | - | Dynamic | 4.05 | 3.90 | 6.18 | ||
| NTK-aware | ✗ | - | Dynamic | 4.05 | 5.97 | |||
| NTK-by-parts | ✗ | - | Dynamic | 4.05 | 3.71 | 4.11 | 5.79 | |
| YaRN | ✗ | - | Dynamic | 4.05 | 3.67 | 3.65 | 3.33 | 3.45 |
| PI | ✓ | 400 | 2k 16 | 5.70 | 4.95 | 4.64 | 3.97 | 3.57 |
| NTK-aware | ✓ | 400 | 2k 16 | 4.39 | 3.92 | 3.73 | 3.21 | 8.49 |
| NTK-by-parts | ✓ | 400 | 2k 16 | 4.14 | 3.75 | 3.62 | 3.12 | 2.81 |
| YaRN | ✓ | 400 | 2k 16 | 4.19 | 3.77 | 3.30 | 3.09 | 2.77 |
B.2 Training Efficiency of YaRN
For Llama 2 7B extended from 4k to 8k, YaRN’s 400-step run reaches 8k perplexity 3.35, close to PI’s 3.34 after 1,000 steps. This is the basis of the 2.5× fewer-steps comparison. The PI checkpoint is LLongMA-2, whose footnote specifies RedPajama training, whereas the main YaRN protocol uses PG-19; the comparison does not isolate only the interpolation rule.
| Model | Extension | Fine- | Training | Extension | Evaluation Context Window Size | |||
| Size | Method | tuned | Steps | Scale | 2048 | 4096 | 6144 | 8192 |
| 7B | None | ✗ | - | - | 4.00 | 3.58 | - | - |
| 7B | PI | ✗ | - | 4k 2 | 4.30 | 3.84 | 3.83 | 3.65 |
| 7B | YaRN | ✗ | - | 4k 2 | 4.03 | 3.61 | 3.60 | 3.49 |
| 7B | PI | ✓ | 1000 | 4k 2 | 3.92 | 3.51 | 3.51 | 3.34 |
| 7B | YaRN | ✓ | 400 | 4k 2 | 3.91 | 3.50 | 3.51 | 3.35 |
B.3 Comparing the perplexity of various methods over a sliding window
Compare YaRN with Together’s 32k PI model and Code Llama’s longer-window NTK model. At 128k, YaRN’s 7B/13B perplexities are 2.37/2.24 versus Code Llama’s 2.71/2.54. These checkpoints differ in training data and specialization. Table 7 specifies ten 128k Proof-pile documents; Figure 7’s “1.28M-token … documents” caption is retained, without interpreting it as evidence for a single 1.28M-token context.
| Model | Model | Context | Extension | Evaluation Context Window Size | ||||
| Size | Name | Window | Method | 8192 | 32768 | 65536 | 98304 | 131072 |
| 7B | Together | 32k | PI | 3.50 | 2.64 | |||
| 7B | Code Llama | 100k | NTK | 3.71 | 2.74 | 2.55 | 2.54 | 2.71 |
| 7B | YaRN () | 64k | YaRN | 3.51 | 2.65 | 2.42 | ||
| 7B | YaRN () | 128k | YaRN | 3.56 | 2.70 | 2.45 | 2.36 | 2.37 |
| 13B | Code Llama | 100k | NTK | 3.54 | 2.63 | 2.41 | 2.37 | 2.54 |
| 13B | YaRN () | 64k | YaRN | 3.25 | 2.50 | 2.29 | ||
| 13B | YaRN () | 128k | YaRN | 3.29 | 2.53 | 2.31 | 2.23 | 2.24 |
B.4 GovReport evaluations
Use fifty untruncated GovReport test documents of at least 16k tokens, a fixed maximum context of 32k and stride 256, without dynamic scaling. YaRN-64k scores 3.59/3.35 for 7B/13B, slightly lower than YaRN-128k’s 3.64/3.39. These are language-model perplexities on reports, not summarization-quality scores.
| Model | Model | Context | Extension | Perplexity |
| Size | Name | Window | Method | |
| 7B | Together | 32k | PI | 3.67 |
| 7B | Code Llama | 100k | NTK | 4.44 |
| 7B | YaRN () | 64k | YaRN | 3.59 |
| 7B | YaRN () | 128k | YaRN | 3.64 |
| 13B | Code Llama | 100k | NTK | 4.22 |
| 13B | YaRN () | 64k | YaRN | 3.35 |
| 13B | YaRN () | 128k | YaRN | 3.39 |
B.5 Passkey Retrieval
For 64k/128k checkpoints, run ten random-location passkey trials at each tested length from 8k to 128k. “Passkey context” is the largest tested length reaching at least 80% accuracy; the reported accuracy averages all tested lengths up to that point. The 128k YaRN models score 99.4% by this aggregate definition, not a guarantee for every position. Retrieval can remain strong where perplexity worsens; the extra 200 training steps also confound comparisons between the two YaRN scales.
| Model | Model | Scaling | Context | Training | Extension | Passkey | Passkey |
| Size | Name | Factor | Window | Data Context | Method | Context | Accuracy |
| 7B | Together | 4 | 32k | 32k | PI | 32k | 100% |
| 7B | Code Llama | 88.6 | 100k | 16k | NTK | 112k | 94.3% |
| 7B | YaRN | 16 | 64k | 64k | YaRN | 64k | 96.3% |
| 7B | YaRN | 32 | 128k | 64k | YaRN | 128k | 99.4% |
| 13B | Code Llama | 88.6 | 100k | 16k | NTK | 128k | 99.4% |
| 13B | YaRN | 16 | 64k | 64k | YaRN | 64k | 97.5% |
| 13B | YaRN | 32 | 128k | 64k | YaRN | 128k | 99.4% |
B.6 Standardized Benchmarks
Table 10 repeats short-task scores alongside Together and Code Llama. YaRN generally preserves more broad benchmark performance than these particular long-context baselines, but its MMLU and some other scores still decline relative to original Llama 2. Code Llama’s code specialization makes this an imperfectly controlled comparison.
| Model | Model | Context | Extension | ARC-c | Hellaswag | MMLU | TruthfulQA |
|---|---|---|---|---|---|---|---|
| Size | Name | Window | Method | ||||
| 7B | Llama 2 | 4k | None | 53.1 | 77.8 | 43.8 | 39.0 |
| 7B | Together | 32k | PI | 47.6 | 76.1 | 43.3 | 39.2 |
| 7B | Code Llama | 100k | NTK-a | 39.9 | 60.8 | 31.1 | 37.8 |
| 7B | YaRN () | 64k | YaRN | 52.3 | 78.8 | 42.5 | 38.2 |
| 7B | YaRN () | 128k | YaRN | 52.1 | 78.4 | 41.7 | 37.3 |
| 13B | Llama 2 | 4k | None | 59.4 | 82.1 | 55.8 | 37.4 |
| 13B | Code Llama | 100k | NTK-a | 40.9 | 63.4 | 32.8 | 43.8 |
| 13B | YaRN () | 64k | YaRN | 58.1 | 82.3 | 52.8 | 37.8 |
| 13B | YaRN () | 128k | YaRN | 58.0 | 82.2 | 51.9 | 37.3 |
B.7 Dynamic scaling on models without any fine-tuning
Figure 8 applies dynamic PI and dynamic YaRN to an unfine-tuned Llama 2 model on one long GovReport sample, using stride 256. Both avoid the original RoPE curve’s blow-up beyond approximately 4k; dynamic YaRN has lower perplexity in the displayed extended interval. This single-sample demonstration is narrower than a general no-fine-tuning guarantee.