Literature
Display
← All papers

Extending Context Window of Large Language Models via Position Interpolation

Shouyuan Chen · Sherman Wong · Liangjian Chen · Yuandong Tian

Preprint first posted · arXiv ↗

Language models7 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete arXiv v2, including the interpolation proof, bound visualization and both plotting-code appendices. Its title follows the PDF/HTML heading, “Position Interpolation”; the arXiv landing page uses “Positional Interpolation”.

At a glance

Intuition. A pretrained model has learned to interpret positions within a particular range. Position interpolation fits a longer sequence onto that familiar ruler. When doubling the window, position 6,000 is supplied to the rotary calculation as 3,000; the token itself is still present.

Reading the math. If the new length is s times the old length, position m becomes m/s. Neighboring tokens are consequently separated by 1/s of an old position unit, changing positional detail. Fine-tuning helps the model adapt to that denser ruler. This does not shorten the actual sequence or remove attention’s memory cost. Treat the reported experiments separately from the qualified theoretical bound.

Position Interpolation (PI) extends a pretrained rotary-position language model by shrinking its position indices into the original range, then fine-tuning on longer sequences. LLaMA experiments reach contexts up to 32,768 tokens with generally 1,000 fine-tuning steps. Long-text perplexity and passkey retrieval improve over the tested direct-extension baseline, with some degradation on short-context tasks. The paper’s approximately 600-fold theoretical bound comparison requires qualification because Appendix A contains an incorrect inequality.

1 Introduction

LLaMA was trained with 2,048-token context. Directly fine-tuning at greater lengths adapts slowly in the reported retrieval test. PI instead maps the new interval back into the trained positional interval, retaining the architecture and introducing no weights. The study tests long-text modeling, synthetic retrieval, short-context benchmarks and separately fine-tuned summarization. It acknowledges concurrent SuperHOT work; the experiments here use full fine-tuning rather than establishing a general low-rank-adaptation result.

Complete original Figure 1: An illustration of our Position Interpolation method. Consider a Llama model pre-trained with a 2048 context window length. Upper left illustrates the normal usage of an LLM model: input position indices (blue dots) are within the pre-trained range. Upper right illustrates length extrapolation where models are required to operate unseen positions (red dots) up to 4096. Lower left illustrates Position Interpolation where we downscale the position indices (blue and green dots) themselves from [0, 4096] to [0, 2048] to force them to reside in the pretrained range.
Figure 1: An illustration of our Position Interpolation method. Consider a Llama model pre-trained with a 2048 context window length. Upper left illustrates the normal usage of an LLM model: input position indices (blue dots) are within the pre-trained range. Upper right illustrates length extrapolation where models are required to operate unseen positions (red dots) up to 4096. Lower left illustrates Position Interpolation where we downscale the position indices (blue and green dots) themselves from [0, 4096] to [0, 2048] to force them to reside in the pretrained range.

2 Method

The method distinguishes increasing the number of tokens from extrapolating their numerical RoPE coordinates.

2.1 Background: Rotary Position Embedding (RoPE)

RoPE rotates query/key coordinate pairs at geometrically spaced frequencies. Their dot product is a sum of sinusoidal functions of relative displacement, weighted by content-dependent query/key products. Equations 1–2 retain the exact complex and real formulations.

Equations 1–2
𝐟⁡(𝐱,m)=[(x0+i​x1)​ei​m​θ0,(x2+i​x3)​ei​m​θ1,…,(xd−2+i​xd−1)​ei​m​θd/2−1]⊤\mathbf{f}(\mathbf{x},m)=[(x_{0}+\mathrm{i}x_{1})e^{\mathrm{i}m\theta_{0}},(x_{2}+\mathrm{i}x_{3})e^{\mathrm{i}m\theta_{1}},\ldots,(x_{d-2}+\mathrm{i}x_{d-1})e^{\mathrm{i}m\theta_{d/2-1}}]^{\top} (1)
a⁡(m,n)\displaystyle a(m,n) =\displaystyle= Re​⟨𝐟⁡(𝐪,m),𝐟⁡(𝐤,n)⟩\displaystyle\mathrm{Re}\langle\mathbf{f}(\mathbf{q},m),\mathbf{f}(\mathbf{k},n)\rangle (2)
=\displaystyle= Re⁡[∑j=0d/2−1(q2​j+i​q2​j+1)​(k2​j−i​k2​j+1)​ei⁡(m−n)​θj]\displaystyle\mathrm{Re}\left[\sum_{j=0}^{d/2-1}(q_{2j}+\mathrm{i}q_{2j+1})(k_{2j}-\mathrm{i}k_{2j+1})e^{\mathrm{i}(m-n)\theta_{j}}\right]
=\displaystyle= ∑j=0d/2−1(q2​j​k2​j+q2​j+1​k2​j+1)​cos⁡((m−n)​θj)+(q2​j​k2​j+1−q2​j+1​k2​j)​sin⁡((m−n)​θj)\displaystyle\sum_{j=0}^{d/2-1}(q_{2j}k_{2j}+q_{2j+1}k_{2j+1})\cos((m-n)\theta_{j})+(q_{2j}k_{2j+1}-q_{2j+1}k_{2j})\sin((m-n)\theta_{j})
=:\displaystyle=: a⁡(m−n)\displaystyle a(m-n)

2.2 Direct Extrapolation

A trigonometric score that remains moderate on the trained interval can become large outside it. Figure 2 illustrates this with least-squares fits to random targets; it is a constructed function example, not a measurement of every trained attention head. An earlier distance-decay bound can still be large because its coefficient term depends on query/key magnitudes. Directly extending the tested original model produces perplexity above 1,000.

Complete original Figure 2: Extrapolation versus interpolation. Left: a fitted attention score function (in red) in the form of Eqn. 3 with d = d model / n head = 4096 / 32 = 128 d=d_{\mathrm{model}}/n_{\mathrm{head}}=4096/32=128 (setting of LLaMA 7B). Dots are random input points to be fitted and red curve is the fitted score function via least square, which is approximately within [ − 1 , 1 ] [-1,1] . Middle: While the fitted function seems to be well bounded in [ 0 , L ] [0,L] , where L = 2048 L=2048 , out of this region it may goes beyond 8000 8000 , causing catastrophic issues in attention computation. Note that here we do not cherry pick at all: almost every learned curve from a set of randomly generated input points within [ 0 , L ] [0,L] has the extrapolation issue. Right: On the other hand, interpolation is much more stable. Curves in between vertical dotted lines (i.e., integer positional difference) are smooth and well-behaved. Please check Appendix C.1 for the source code used to generate the figure.
Figure 2: Extrapolation versus interpolation. Left: a fitted attention score function (in red) in the form of Eqn. 3 with d=dmodel/nhead=4096/32=128d=d_{\mathrm{model}}/n_{\mathrm{head}}=4096/32=128 (setting of LLaMA 7B). Dots are random input points to be fitted and red curve is the fitted score function via least square, which is approximately within [−1,1][-1,1]. Middle: While the fitted function seems to be well bounded in [0,L][0,L], where L=2048L=2048, out of this region it may goes beyond 80008000, causing catastrophic issues in attention computation. Note that here we do not cherry pick at all: almost every learned curve from a set of randomly generated input points within [0,L][0,L] has the extrapolation issue. Right: On the other hand, interpolation is much more stable. Curves in between vertical dotted lines (i.e., integer positional difference) are smooth and well-behaved. Please check Appendix C.1 for the source code used to generate the figure.
Equation 3
a⁡(s)=Re⁡[∑j=0d/2−1hj​ei​s​θj]a(s)=\mathrm{Re}\left[\sum_{j=0}^{d/2-1}h_{j}e^{\mathrm{i}s\theta_{j}}\right] (3)

2.3 Proposed approach: Position Interpolation (PI)

For original length L and target length L′, replace position m by mL/L′ before applying RoPE. All relative coordinates remain within the original numerical range, although token density changes. Fine-tune with next-token prediction on longer inputs. The method changes neither weights’ shapes nor attention itself, so existing infrastructure remains usable; full attention still has its original long-sequence cost.

The source bounds deviation from linear interpolation between neighboring grid points and compares it with an extrapolation envelope. Its stated approximately 600× advantage is not a verified universal guarantee: Appendix A uses a reversed exponential inequality, and the extrapolation term is assessed numerically for selected settings. The original theorem and formulas remain below. Regularizing query/key product magnitudes is proposed as future work, not tested as an LLM-training method here.

Equation 4
𝐟​’​(𝐱,m)=𝐟⁡(𝐱,m​LL′).\mathbf{f}’(\mathbf{x},m)=\mathbf{f}\left(\mathbf{x},\frac{mL}{L^{\prime}}\right). (4)
Theorem 2.1 (Interpolation bound).

For attention score a⁡(s)=Re⁡[∑j=0d/2−1hj​ei​s​θj]a(s)=\mathrm{Re}\left[\sum_{j=0}^{d/2-1}h_{j}e^{\mathrm{i}s\theta_{j}}\right], where θj=c−2j/d\theta_{j}=c^{-2j/d}, its interpolation value a⁡(s)a(s) for s∈[s1,s2]s\in[s_{1},s_{2}] is bounded as follows:

Equation 5
|a⁡(s)−alinear​(s)|≤d⁡(maxj⁡|hj|)​(s−s1)​(s2−s)8​ln⁡c|a(s)-a_{\mathrm{linear}}(s)|\leq d\left(\max_{j}|h_{j}|\right)\frac{(s-s_{1})(s_{2}-s)}{8\ln c} (5)

where alinear​(s)a_{\mathrm{linear}}(s) is the linear interpolation of two grid point a⁡(s1)a(s_{1}) and a⁡(s2)a(s_{2}) that are known to behave well, enforced by LLM pre-training:

Equation 6
alinear​(s):=(1−λ⁡(s))​a​(s1)+λ⁡(s)​a​(s2),λ⁡(s):=s−s1s2−s1a_{\mathrm{linear}}(s):=(1-\lambda(s))a(s_{1})+\lambda(s)a(s_{2}),\quad\quad\lambda(s):=\frac{s-s_{1}}{s_{2}-s_{1}} (6)
Equations 7–8
|a⁡(s)−alinear​(s)|≤d32​ln⁡c​maxj​|hj|≈d​maxj​|hj|294.73|a(s)-a_{\mathrm{linear}}(s)|\leq\frac{d}{32\ln c}\max_{j}|h_{j}|\approx\frac{d\max_{j}|h_{j}|}{294.73} (7)
|a⁡(s)|≤(maxj⁡|hj−hj+1|)​∑k=0d/2−1|Ak+1​(s)|≤2​(maxj⁡|hj|)​∑k=0d/2−1|Ak+1​(s)|,|a(s)|\leq\left(\max_{j}|h_{j}-h_{j+1}|\right)\sum_{k=0}^{d/2-1}|A_{k+1}(s)|\leq 2\left(\max_{j}|h_{j}|\right)\sum_{k=0}^{d/2-1}|A_{k+1}(s)|, (8)

3 Experiments

Reported configurations extend 2,048-token LLaMA to at most 32,768 tokens, a 16× extension. The source also says “32 times”, which is inconsistent with its displayed model settings. Model sizes do not all receive every target length.

3.1 Setup

Use LLaMA 7B, 13B, 33B and 65B with unchanged architecture. PI generally receives 1,000 next-token fine-tuning steps on the Pile; direct fine-tuning receives 10,000. AdamW uses betas (.9,.95), zero weight decay and twenty warmup steps from one-tenth the peak rate. Peak rate is 2×10⁻⁵ for 7B/13B and 10⁻⁵ for 33B/65B. The 7B–33B 8,192-token runs use 32 A100 GPUs and global batch 64; other configurations use 128 GPUs and batch 128, primarily for memory capacity. PyTorch FSDP and FlashAttention support training.

3.2 Long Sequence Language Modeling

Evaluate all 100 PG-19 test documents and 128 sampled Proof-pile documents of at least 32,768 SentencePiece tokens, truncating the latter to that length. Perplexity uses sliding windows with stride 256. PI benefits from longer context more than the direct fine-tuning baseline. For 7B at target length 16,384, full-window PG-19 perplexity falls from the original model’s 7.20 at 2,048 to 6.83; Proof-pile falls from 2.77 to 2.25.

The trend is not strictly monotone: for the 32,768-target 7B model, Proof-pile rises from 2.24 at evaluation length 16,384 to 2.48 at 32,768. Some short-window perplexities also regress. Without fine-tuning, PI’s 7B PG-19 perplexity is 16.10 at 8,192 and 112.13 at 16,384; after 200 steps these become 7.12 and 7.05, then 6.95 and 6.83 after 1,000 steps. Tables 1–3 preserve all configurations and intermediate results.

Model Evaluation Context Window Size
Size Context Window Method 2048 4096 8192 16384 32768
7B 2048 None 7.20 >103>10^{3} >103>10^{3} >103>10^{3} >103>10^{3}
7B 8192 FT 7.21 7.34 7.69 - -
7B 8192 PI 7.13 6.96 6.95 - -
7B 16384 PI 7.11 6.93 6.82 6.83 -
7B 32768 PI 7.23 7.04 6.91 6.80 6.77
13B 2048 None 6.59 - - - -
13B 8192 FT 6.56 6.57 6.69 - -
13B 8192 PI 6.55 6.42 6.42 - -
13B 16384 PI 6.56 6.42 6.31 6.32 -
13B 32768 PI 6.54 6.40 6.28 6.18 6.09
33B 2048 None 5.82 - - - -
33B 8192 FT 5.88 5.99 6.21 - -
33B 8192 PI 5.82 5.69 5.71 - -
33B 16384 PI 5.87 5.74 5.67 5.68 -
65B 2048 None 5.49 - - - -
65B 8192 PI 5.42 5.32 5.37 - -
Table 1: Evaluation perplexity on PG19 dataset (Rae et al., 2020). FT: Direct Fine-tuning. PI: Position Interpolation. Model fine-tuned with PI shows progressively lower perplexity with longer context window, showing that PI can leverage long context well, while the perplexity of FT increases over longer window. Note that overall the perplexity is higher compared to Table 2 since PG19 has very different writing styles.
Model Evaluation Context Window Size
Size Context Window Method 2048 4096 8192 16384 32768
7B 2048 None 2.77 - - - -
7B 8192 FT 2.85 2.74 2.73 - -
7B 8192 PI 2.79 2.57 2.39 - -
7B 16384 PI 2.79 2.57 2.37 2.25 -
7B 32768 PI 2.82 2.59 2.39 2.24 2.48
13B 2048 None 2.66 - - - -
13B 8192 FT 2.71 2.56 2.50 - -
13B 8192 PI 2.67 2.47 2.30 - -
13B 16384 PI 2.68 2.47 2.29 2.18 -
13B 32768 PI 2.68 2.46 2.28 2.15 2.35
33B 2048 None 2.49 - - - -
33B 8192 FT 2.56 2.48 2.47 - -
33B 8192 PI 2.50 2.32 2.18 - -
33B 16384 PI 2.53 2.34 2.18 2.07 -
65B 2048 None 2.42 - - - -
65B 8192 PI 2.43 2.26 2.12 - -
Table 2: Evaluation perplexity on Arxiv Math Proof-pile dataset (Azerbayev et al., 2022). FT: Direct Fine-tuning. PI: Position Interpolation.
Model Number of fine-tuning steps
Size Context Window 0 200 400 600 800 1000
7B 8192 16.10 7.12 7.10 7.02 6.99 6.95
7B 16384 112.13 7.05 6.93 6.88 6.84 6.83
Table 3: Evaluation perplexity on PG19 dataset (Rae et al., 2020) with respect to the number of fine-tuning steps using Position Interpolation.

3.3 Measuring Effective Context Window Size through Passkey Retrieval

Hide a random five-digit passkey in repetitive irrelevant text, following Landmark Attention. Test ten trials at each of 32 uniformly spaced distances. The reported effective length is the greatest tested distance such that every shorter tested distance reaches at least 20% success—not perfect retrieval. PI reaches its target at 200 steps across the displayed configurations. Results are not uniformly stable: the 7B/32,768 run falls to 18,432 at 600 steps before recovering. Direct 7B fine-tuning reaches only 2,560 after 10,000 steps. Figure 3 retains the full prompt.

Model Fine-tuning steps
Size Context Window Method 200 400 600 800 1000 10000
7B 8192 FT 1792 2048 2048 2048 2304 2560
33B 8192 FT 1792 2048 1792 2048 2304 -
7B 8192 PI 8192 8192 8192 8192 8192 -
7B 16384 PI 16384 16384 16384 16384 16384 -
7B 32768 PI 32768 32768 18432 32768 32768 -
33B 8192 PI 8192 8192 8192 8192 8192 -
33B 16384 PI 16384 16384 16384 16384 16384 -
Table 4: Effective context window sizes after fine-tuning. FT: Direct fine-tuning. PI: Position Interpolation.
Complete original Figure 3: Prompt format for passkey retrieval. We use the exact same prompt as proposed by Mohtashami & Jaggi (2023) . Here the passkey 12345 is replaced with a random 5-digit numbers during test.
Figure 3: Prompt format for passkey retrieval. We use the exact same prompt as proposed by Mohtashami & Jaggi (2023). Here the passkey 12345 is replaced with a random 5-digit numbers during test.
Selectable original prompt

There is an important info hidden inside a lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. (repeat X times)
The pass key is 12345. Remember it. 12345 is the pass key.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. (repeat Y times)
What is the pass key? The pass key is

3.4 Benchmarks on Original Context Window Size

Table 5 evaluates original-window zero-shot tasks. The 7B 8,192-target Pile model drops BoolQ from 76.1 to 73.2, a 2.9-point loss, despite the source’s “up to 2%” summary. At 32,768, BoolQ falls to 64.7 and other scores also decline. The 33B/8,192 results are closer to baseline. A RedPajama comparison gives a different task-wise pattern; this small comparison does not establish that adaptation data never matters.

Model Size Context Window Fine-tune on BoolQ PIQA Race-M Race-H WinoGrande
7B 2048 None 76.1 78.9 55.7 42.2 69.6
7B 8192 Pile 73.2 78.2 53.8 41.7 69.0
7B 16384 Pile 69.8 77.6 53.3 40.9 67.8
7B 32768 Pile 64.7 77.2 50.1 39.6 66.9
7B 8192 RedPajama 75.5 77.4 54.5 41.5 68.1
33B 2048 None 81.6 80.2 61.1 45.9 76.2
33B 8192 Pile 80.2 80.7 60.2 45.7 75.9
Table 5: Zero-shot performance on a subset of LLaMA Benchmarks. Models extended by Position Interpolation comparable performance as the original models, except for BoolQ dataset that may require models to pay close attention to word ordering in a short reference paragraph.

3.5 Long Document Summarization

GovReport supplies 17,457 training and 972 evaluation documents. Starting from a PI-extended 16,384-context model, truncate documents to 15,000 tokens and summaries to 1,000, then fine-tune for ten additional epochs with loss only on summary tokens. Generate at temperature .5 and top-p .95, capped at 1,000 tokens. LLaMA-7B scores ROUGE-1/2/L 60.0/28.0/29.5: ROUGE-1 lies between CoLT5 Base and XL, while ROUGE-2/L are below both. This is a separately supervised summarization result, not zero-shot performance obtained from the 1,000-step context adaptation alone.

Complete original Figure 4: Input format for long doc summarization.
Figure 4: Input format for long doc summarization.
Selectable original prompt

Read the following article and then summarize it.
# .... Document goes here
Now summarize the above article.
Summary:

Model Evaluation Score
Model Context Window ROUGE-1 ROUGE-2 ROUGE-L
CoLT5 Base (Ainslie et al., 2023) 16K 58.7 29.6 31.4
CoLT5 XL (Ainslie et al., 2023) 16K 61.3 32.2 33.8
LLaMA-7B Extended 16K 60.0 28.0 29.5
Table 6: ROUGE Score on GovReport Dataset.

4 Related Work

PI can complement retrieval by fitting more retrieved text in context. Recurrent/compressed-memory approaches trade access to past detail for lower cost; PI retains full attention over the longer input. Sparse or approximate attention addresses a different computational issue. Unlike methods requiring a new positional scheme during pretraining, PI targets existing rotary models. Vision Transformer interpolation resizes learned position embeddings; PI rescales continuous position indices without adding learned entries. Application to other position schemes is proposed future work.

5 Conclusions

Rescaling rotary coordinates plus brief fine-tuning makes substantially longer context usable in the tested LLaMA models. The evidence supports an economical adaptation of existing checkpoints, with explicit limits: full-attention cost remains, model/length coverage is uneven, short-context quality can decline and synthetic retrieval uses a low success threshold. Broader positional schemes and the source’s exact theoretical constant remain unresolved here.

Acknowledgements

The authors thank Mike Lewis for input on evaluation.

Appendix A Proof

The proof attempts to bound interpolation error using Taylor remainders and a bound on the second derivative of the trigonometric score. Equations 9–17 remain exactly as printed. The finite geometric series in Equation 13 is written without its numerator, and the next step uses cˣ≤1+x ln c, whereas convexity gives cˣ≥1+x ln c. Thus the stated d/(4 ln c) derivative bound and resulting numerical constant do not follow from this proof as written. This does not negate the experimental comparisons or the general motivation for keeping positions within the trained range.

Equations 9–17
a⁡(s1)\displaystyle a(s_{1}) =\displaystyle= a⁡(s)+a′​(s)​(s−s1)+12​a′′​(ξ1)​(s−s1)2\displaystyle a(s)+a^{\prime}(s)(s-s_{1})+\frac{1}{2}a^{\prime\prime}(\xi_{1})(s-s_{1})^{2} (9)
a⁡(s2)\displaystyle a(s_{2}) =\displaystyle= a⁡(s)+a′​(s)​(s−s2)+12​a′′​(ξ2)​(s−s2)2\displaystyle a(s)+a^{\prime}(s)(s-s_{2})+\frac{1}{2}a^{\prime\prime}(\xi_{2})(s-s_{2})^{2} (10)
a⁡(s)−alinear​(s)=R⁡(s):=−(s−s1)​(s−s2)2​(s1−s2)​[a′′​(ξ1)​(s−s1)−a′′​(ξ2)​(s−s2)]a(s)-a_{\mathrm{linear}}(s)=R(s):=-\frac{(s-s_{1})(s-s_{2})}{2(s_{1}-s_{2})}\left[a^{\prime\prime}(\xi_{1})(s-s_{1})-a^{\prime\prime}(\xi_{2})(s-s_{2})\right] (11)
|a′′​(s)|\displaystyle|a^{\prime\prime}(s)| ≤\displaystyle\leq ∑j=0d/2−1|hj|​|ϕj′′​(s)|≤∑j=0d/2−1|hj|​θj2\displaystyle\sum_{j=0}^{d/2-1}|h_{j}||\phi^{\prime\prime}_{j}(s)|\leq\sum_{j=0}^{d/2-1}|h_{j}|\theta^{2}_{j} (12)
≤\displaystyle\leq (maxj|hj|)∑j=0d/2−1c−4j/d=(maxj|hj|)11−c−4/d\displaystyle\left(\max_{j}|h_{j}|\right)\sum_{j=0}^{d/2-1}c^{-4j/d}=\left(\max_{j}|h_{j}|\right)\frac{1}{1-c^{-4/d}} (13)
11−c−4/d≤14/d​ln⁡c=d4​ln⁡c\frac{1}{1-c^{-4/d}}\leq\frac{1}{4/d\ln c}=\frac{d}{4\ln c} (14)
|a′′​(s)|≤(maxj⁡|hj|)​d4​ln⁡c=:M|a^{\prime\prime}(s)|\leq\left(\max_{j}|h_{j}|\right)\frac{d}{4\ln c}=:M (15)
|R⁡(s)|≤(s−s1)​(s2−s)2​(s2−s1)​[M⁡(s−s1)+M⁡(s2−s)]=M2​(s−s1)​(s2−s)|R(s)|\leq\frac{(s-s_{1})(s_{2}-s)}{2(s_{2}-s_{1})}\left[M(s-s_{1})+M(s_{2}-s)\right]=\frac{M}{2}(s-s_{1})(s_{2}-s) (16)
|a⁡(s)−alinear​(s)|=|R⁡(s)|≤d⁡(maxj⁡|hj|)​(s−s1)​(s2−s)8​ln⁡c|a(s)-a_{\mathrm{linear}}(s)|=|R(s)|\leq d\left(\max_{j}|h_{j}|\right)\frac{(s-s_{1})(s_{2}-s)}{8\ln c} (17)

Appendix B Visualization of quantities in extrapolation bound

Figure 5 evaluates the partial-phase-sum term B(s)/d with c=10,000 and head dimension 128, the LLaMA-7B setting. Its oscillatory envelope is typically above one over the plotted range. This numerical illustration explains why the earlier extrapolation bound can be loose; it is not a proof for every displacement or configuration.

Complete original Figure 5: The bound B ⁡ ( s ) / d B(s)/d decays with s s . While the bounds goes down with large positional difference s s , numerically B ⁡ ( s ) / d ≥ 1 B(s)/d\geq 1 and at many s s much larger than 1 1 (the dotted horizontal line). Please check Appendix C.2 for the source code used to draw the figure.
Figure 5: The bound B⁡(s)/dB(s)/d decays with ss. While the bounds goes down with large positional difference ss, numerically B⁡(s)/d≥1B(s)/d\geq 1 and at many ss much larger than 11 (the dotted horizontal line). Please check Appendix C.2 for the source code used to draw the figure.

Appendix C Code

The original two code listings are retained with line breaks and indentation. They generate the illustrative fit and bound plots, not model-training experiments.

C.1 Code for Fig. 2

Construct sine/cosine features for 2,048 positions with dimension 128, fit random targets by solving the unregularized normal equations, then plot extrapolation to 4,096 and interpolation on a finer grid. The example is sensitive to numerical conditioning and the random sample. The source listing remains unchanged, including the loop that draws the interpolation grid.

# build basis function
d = 4096 // 32
theta = 10000
# Frequency computation,
freqs = 1.0 / (theta ** (torch.arange(0, d, 2)[: (d // 2)].float() / d))

# construct basis function
L = 2048

x = torch.zeros(L)
x[:L] = torch.arange(0, L)

# basis functions
xfreq = torch.outer(x, freqs)

y = torch.randn(x.shape[0])

# do linear regression
X = torch.cat([xfreq.sin(), xfreq.cos()], dim=1)

eps = 0.000
coeffs = torch.linalg.solve(X.t() @ X + torch.eye(X.shape[1]) * eps, X.t() @ y)

x2 = torch.arange(0, 2*L)
xfreq2 = torch.outer(x2, freqs)
X2 = torch.cat([xfreq2.sin(), xfreq2.cos()], dim=1)

y2 = X2 @ coeffs

x3 = torch.arange(25, 75, 0.125)
xfreq3 = torch.outer(x3, freqs)
X3 = torch.cat([xfreq3.sin(), xfreq3.cos()], dim=1)

y3 = X3 @ coeffs

plt.figure(figsize=(16,5))

plt.subplot(1, 3, 1)
plt.plot(x2[:L], y2[:L], "r")
plt.scatter(x, y)
plt.ylabel("attention score $a(s)$")
plt.xlabel("Positional difference $s$")

plt.subplot(1, 3, 2)

plt.plot(x2, y2, "r")
plt.scatter(x, y)
plt.axvline(L, color="k", linestyle="--", linewidth=0.5)

plt.title("Effect of Extrapolation")
plt.xlabel("Positional difference $s$")


plt.subplot(1, 3, 3)
plt.plot(x3, y3, "r")
for i in range(25,75):
    plt.axvline(i, color="k", linestyle="--", linewidth=0.5)
plt.title("Effect of Interpolation")
plt.xlabel("Positional difference $s$")
plt.show()

C.2 Code for Fig. 5

Compute cumulative complex-phase magnitudes and plot their sum divided by d over 4,096 displacements. The printed listing contains typographic quotation marks around k in the color argument; these source characters are preserved rather than silently converted into executable code.

L = 2048
x = torch.arange(0, 2*L)
d = 4096 // 32
theta = 10000
freqs = 1.0 / (theta ** (torch.arange(0, d, 2)[: (d // 2)].float() / d))

xfreq = torch.outer(x, freqs)

mags = (xfreq.sin().cumsum(dim=1).pow(2) + xfreq.cos().cumsum(dim=1).pow(2)).sqrt()

plt.plot(mags.sum(dim=1)/d)
plt.axhline(1.0, color=’k’, linestyle="--")
plt.xlabel("Positional difference $s$")
plt.ylabel("$B(s)/d$")
plt.show()
Concept demonstration

Open image ↗