Overview
This edition follows arXiv v5, including its theoretical derivations, English and Chinese experiments, and explicit limitations. Source resources: author implementation and Transformers integration.
At a glance
Intuition. Imagine each pair of vector coordinates as a clock hand. Moving a token forward rotates the hand. Rotating the query and key by their respective positions makes their comparison depend on the difference between the rotations, so shifting both tokens together leaves that positional relationship unchanged.
Reading the math. q and k hold token content; m and n are positions. Coordinate pair j rotates at rate θⱼ, giving relative phase (m − n)θⱼ. Different pairs use different rates, providing several position scales. Rotation preserves vector length. This algebraic property does not guarantee that attention always decreases with distance or that an unmodified model handles arbitrarily long text.
Rotary Position Embedding (RoPE) multiplies projected queries and keys by position-dependent rotations. Their inner product then encodes relative displacement while each vector carries an absolute-position rotation. The construction preserves vector norms and can be combined with linear attention. Selected language-modeling and long-document experiments favor RoFormer, but the GLUE results are mixed. The theoretical distance envelope does not establish uniformly decreasing attention or reliable extrapolation to arbitrary context lengths.
1 Introduction
Self-attention needs a mechanism for token order. Transformers commonly add position vectors, while relative-position methods modify pairwise scores. RoPE instead rotates query/key features, producing a relative-position inner product without an added pairwise bias. The paper develops this geometry, studies an oscillatory distance-decay bound, adapts it to linear attention and tests translation, pretraining and downstream language tasks.
2 Background and Related Work
The background separates token/position-dependent query, key and value maps from the attention operation, then compares additive absolute and relative encodings.
2.1 Preliminary
Each token embedding and its position are transformed into a query, key and value. Scaled query–key inner products become softmax weights, and their weighted value sum becomes the output. Equations 1–2 establish the notation that the positional methods modify.
Equations 1–2
| (1) | ||||
| (2) | ||||
2.2 Absolute position embedding
Absolute encodings add a position vector before the query/key/value projection. Learned vectors typically assume a configured position range; sinusoidal vectors provide a deterministic frequency basis. RoPE uses related trigonometric structure through multiplication. Equation 4 is reproduced as printed, including its inconsistent position-index notation.
Equations 3–4
| (3) |
| (4) |
2.3 Relative position embedding
Shaw et al. add clipped relative embeddings to keys and values. Transformer-XL decomposes content/position interactions, while T5 uses a relative score bias. Other variants retain different cross terms. These additive constructions motivate seeking a query/key transformation whose inner product itself has the required relative dependence.
Equations 5–10
| (5) | |||
| (6) |
| (7) |
| (8) |
| (9) |
| (10) |
3 Proposed approach
The method specifies a relative-position condition, constructs rotations satisfying it, and derives a sparse implementation and a compatible linear-attention form.
3.1 Formulation
Require the query–key inner product to depend on the two token contents and their relative displacement, rather than on their absolute positions separately. Equation 11 states this target. It is a design condition, not a claim that rotations are the only possible positional representation.
Equation 11
| (11) |
3.2 Rotary position embedding
The two-dimensional construction extends to even-dimensional vectors by independently rotating coordinate pairs.
3.2.1 A 2D case
Identify a pair of coordinates with a complex number. Rotate a projected query by mθ and a projected key by nθ. The conjugated complex product cancels the common absolute phase, leaving the relative phase (m−n)θ. Equations 12–13 express the same operation in complex and real-matrix notation.
Equations 12–13
| (12) | ||||
| (13) |
3.2.2 General form
Split an even-dimensional vector into coordinate pairs and use a block-diagonal matrix of two-dimensional rotations, with geometrically spaced frequencies. Orthogonality preserves norms and gives RₘᵀRₙ=Rₙ₋ₘ, making the attention score relative. Figure 1 shows the operation. Equation 16’s final expanded expression has inconsistent token/projection-transpose notation; its original formula remains unchanged, while the preceding rotated-inner-product equality supplies the intended construction.
Equations 14–16
| (14) |
| (15) |
| (16) |
3.3 Properties of RoPE
Two properties are examined: a distance-dependent bound induced by the frequency schedule, and a rotation-based extension of linear attention.
Long-term decay:
The chosen geometric frequencies produce the oscillatory upper-envelope behavior analyzed in §3.4.3. This supports a form of long-distance attenuation; it does not mean that every content-dependent attention score decreases monotonically as tokens move apart.
RoPE with linear attention:
Linear attention reassociates sums over feature-mapped keys and values to avoid a full pairwise attention matrix. RoPE rotates these query/key features in the numerator while leaving the normalization denominator unrotated. The paper does this to avoid introducing zero denominators through rotation. Numerator contributions can become negative, so this variant is not a probability distribution over values; it retains the associative computational structure.
Equations 17–19
| (17) |
| (18) |
| (19) |
3.4 Theoretical Explanation
The theory derives one family of valid two-dimensional maps, explains an elementwise implementation, and bounds the relative-position inner product using sums of complex phases.
3.4.1 Derivation of RoPE under 2D
Write query/key maps in polar form and impose their zero-position values and relative inner-product condition. A position-independent radius and content-independent relative phase lead to constant phase increments, hence a phase linear in position. Choosing linear token projections and zero common phase yields RoPE. This derives a particular solution rather than uniqueness among all encodings. Equations 20–33 are retained verbatim; the source’s exponential notation in Equation 31 and query-token index in Equation 32 contain apparent typesetting inconsistencies.
Equations 20–33
| (20) | ||||
| (21) |
| (22) | ||||
| (23) | ||||
| (24) | ||||
| (25) | ||||
| (26a) | ||||
| (26b) | ||||
| (27) | ||||
| (28) |
| (29) |
| (30) |
| (31) | ||||
| (32) | ||||
| (33) | ||||
3.4.2 Computational efficient realization of rotary matrix multiplication
A dense rotation matrix is unnecessary. Multiply the original coordinates by cosines and a signed, pair-swapped vector by sines, then add them. Equation 34 exposes this elementwise implementation and preserves the pairwise rotation exactly.
Equation 34
| (34) |
3.4.3 Long-term decay of RoPE
Express the score as a real part of a sum of complex coordinate-pair products. Abel transformation bounds it by a content-dependent adjacent-product variation term times a sum of partial phase sums. Figure 2 plots the oscillatory relative-distance bound for the selected frequency schedule. The argument concerns this bound, not a general guarantee of context-length generalization or strictly monotone attention weights.
Equations 35–37
| (35) |
| (36) |
| (37) | ||||
4 Experiments and Evaluation
Experiments cover translation, English masked-language pretraining, GLUE, Performer language modeling and Chinese long-document matching. The tasks test different implementations and protocols; their results should be interpreted separately.
4.1 Machine Translation
Translation compares a Transformer baseline with the corresponding rotary-position variant on WMT 2014 English–German.
4.1.1 Experimental Settings
The dataset contains approximately 4.5 million parallel sentence pairs. The baseline follows the Transformer setup.
4.1.2 Implementation details
Use a joint 37,000-token byte-pair vocabulary, average the final five checkpoints, and decode with beam size four and length penalty .6. Fairseq training uses Adam with betas (.9,.98), learning-rate warmup from 10⁻⁷ to 5×10⁻⁴ followed by inverse-square-root decay, and label smoothing .1. Report BLEU.
4.1.3 Results
Table 1 reports BLEU 27.5 for RoFormer versus 27.3 for Transformer-base under the stated common setup: a modest .2-point improvement.
| Model | BLEU |
| Transformer-baseVaswani et al. (2017) | 27.3 |
| RoFormer | 27.5 |
4.2 Pre-training Language Modeling
English pretraining compares a BERT-based model with a rotary variant. The source calls BERT’s replaced position encoding “sinusoidal”; this is the paper’s implementation description, not a property of standard learned-position BERT.
4.2.1 Experimental Settings
BookCorpus and English Wikipedia are split 80:20 for training and validation. Masked-language-modeling loss is the metric, and bert-base-uncased supplies the baseline configuration.
4.2.2 Implementation details
The stated protocol trains for 100,000 steps with batch size 64, maximum length 512 and AdamW learning rate 10⁻⁵. Rotary query/key attention follows Equation 16.
4.2.3 Results
Figure 3’s left panel shows faster training-loss reduction for RoFormer. This plot is evidence about optimization under the reported setup, not by itself a held-out generalization or wall-clock efficiency result.
4.3 Fine-tuning on GLUE tasks
GLUE fine-tuning tests whether pretrained rotary representations transfer across six language-understanding tasks.
4.3.1 Experimental Settings
Report F1 for MRPC and QQP, Spearman correlation for STS-B, and accuracy for SST-2, QNLI and the matched/mismatched MNLI sets. The tasks use different score definitions.
4.3.2 Implementation details
Fine-tune with the Transformers library for three epochs, maximum length 512 and batch size 32. Search learning rates 2, 3, 4 and 5×10⁻⁵, reporting the source’s best-averaged validation results. Table 2 preserves the complete comparison.
4.3.3 Results
RoFormer improves MRPC (89.5 versus 88.9), STS-B (87.0 versus 85.8) and QQP (86.4 versus 71.2), but scores lower on SST-2, QNLI and both MNLI sets. Thus this table supports task-specific gains, not consistent superiority across GLUE.
4.4 Performer with RoPE
Performer provides a linear-attention test bed. Apply the rotary construction from §3.3 while retaining the reassociated attention computation.
4.4.1 Implementation details
Character-level Enwik8 modeling uses twelve layers, hidden width 768, twelve heads, learning rate 10⁻⁴, batch size 128 and fixed sequence length 1,024. The source identifies the Performer implementation.
4.4.2 Results
Figure 3’s right panel shows lower training loss with RoPE at the same training-step budget. It illustrates compatibility and an optimization benefit in this setup; it does not measure a complete runtime comparison.
4.5 Evaluation on Chinese Data
Chinese experiments evaluate document matching with texts commonly longer than 512 characters, using a word-tokenized rotary model and several pretrained baselines.
4.5.1 Implementation
Replace WoBERT’s absolute positions with RoPE. Table 3 distinguishes this word-level tokenizer from the character-level BERT and NEZHA baselines, alongside their position encodings. Tokenization is therefore an additional difference in some comparisons.
| Model | BERTDevlin et al. (2019) | WoBERTSu (2020) | NEZHAWei et al. (2019) | RoFormer |
| Tokenization level | char | word | char | word |
| Position embedding | abs. | abs. | rel. | RoPE |
4.5.2 Pre-training
Pretrain on approximately 34 GB of Chinese Wikipedia, news and forum text in six consecutive stages. Table 4 preserves each length, batch size, step count, loss and accuracy. Lengths range from 128 to 1,536. Because training continues while multiple settings change, differences between stages do not isolate a causal effect of context length.
| Stage | Max seq length | Batch size | Training steps | Loss | Accuracy |
| 1 | 512 | 256 | 200k | 1.73 | 65.0% |
| 2 | 1536 | 256 | 12.5k | 1.61 | 66.8% |
| 3 | 256 | 256 | 120k | 1.75 | 64.6% |
| 4 | 128 | 512 | 80k | 1.83 | 63.4% |
| 5 | 1536 | 256 | 10k | 1.58 | 67.4% |
| 6 | 512 | 512 | 30k | 1.66 | 66.2% |
4.5.3 Downstream Tasks & Dataset
CAIL2019-SCM contains 8,964 triples of legal cases. Given (A,B,C), predict whether A is closer to B or C. Split the data 60:20:20 for training, validation and testing; documents are generally longer than 512 characters.
4.5.4 Results
At length 512, RoFormer’s test accuracy is 68.29%, versus WoBERT’s 68.10% and BERT’s 67.77%. RoFormer at 1,024 reaches 69.79%. The within-RoFormer improvement is 1.50 percentage points; the improvement over WoBERT-512 is 1.69 points. The source prose conflates these comparisons, so Table 5’s exact values are the basis here.
| Model | Validation | Test |
| BERT-512 | 64.13% | 67.77% |
| WoBERT-512 | 64.07% | 68.10% |
| RoFormer-512 | 64.13% | 68.29% |
| RoFormer-1024 | 66.07% | 69.79% |
4.5.5 Limitations of the work
The authors do not fully explain why the rotary model converges faster or performs better on the long-text task. The distance analysis alone does not establish those empirical effects. Pretraining also requires substantial hardware resources.
5 Conclusions
RoPE supplies a norm-preserving positional transform whose query–key product depends on relative displacement, with an efficient coordinatewise implementation and a linear-attention variant. Selected experiments support its usefulness, while mixed downstream results, implementation differences and unresolved explanations limit broader claims.