Token embeddings turn word pieces into vectors. But those vectors alone carry no information about a token's position in the sentence. The sinusoidal positional encoding covered in Part 2.5 solved this with vector addition: it builds a fixed positional vector for each position and adds it to the token embedding before passing it into the transformer block.
RoPE takes a different approach. Instead of adding a positional vector to the token embedding, it rotates the query and key vectors — the ones used to compute attention — according to position. That single sentence is almost all there is to RoPE. Simple as it sounds, this is exactly why RoPE has become the de facto standard in modern decoder-only LLMs like the Llama, Qwen, and Gemma families.
RoPE stands for Rotary Position Embedding, introduced in the RoFormer paper. Despite the word "embedding" in its name, it is better understood from an implementation standpoint as a positional encoding scheme applied inside attention, not as an embedding layer. It plays a completely different role from the embedding table that maps token IDs to vectors.
Figure 1. RoPE does not replace token embeddings; it rotates the query and key inside self-attention.
Adding position vs. rotating position
Sinusoidal positional encoding modifies the input vector itself. Take the sentence "나는 밥을 먹었다": it adds the position-1 vector to the token embedding of 밥을 and the position-2 vector to the token embedding of 먹었다. The transformer then infers both meaning and position from these combined vectors.
RoPE folds position into the moment the attention score is computed. In self-attention, each token's hidden state produces a query, key, and value.
hidden state -> query
hidden state -> key
hidden state -> value
The attention score is the dot product of a query and a key.
score(i, j) = query_i dot key_j
Here i is the position of the token currently being processed and j is the position of the token being attended to. RoPE rotates query_i by angle i and key_j by angle j, then takes the dot product.
score(i, j) = rotate(query_i, i) dot rotate(key_j, j)
This naturally encodes the relationship between positions i and j inside the attention score. More precisely, the dot product retains strong information about the relative distance i - j rather than the absolute values of the two positions. When a language model predicts the next token, what it actually needs most is not "which position in the full sentence is this token?" but rather "how far apart are the token I'm looking at and that other token?"
In short: the addition approach stamps a position label onto the token vector, while RoPE rotates the coordinate frame at attention-computation time.
What it means to rotate a vector
A single 2D vector makes this easy to see. Take a point (x, y). Rotating it around the origin by some angle preserves its length and changes only its direction. The rotation matrix is the same one from high-school math.
[x'] [cos theta -sin theta] [x]
[y'] = [sin theta cos theta] [y]
RoPE does not rotate the entire hidden dimension at once. Instead, it groups the vector's dimensions into pairs and treats each pair as a small 2D plane. For a head dimension of 8, the grouping looks like this:
[x0, x1] [x2, x3] [x4, x5] [x6, x7]
Each pair rotates at a different rate. Lower-indexed dimensions rotate fast; higher-indexed dimensions rotate slowly. This mirrors the reason sinusoidal positional encoding mixes sine and cosine waves at multiple frequencies: some axes need to be sensitive to small positional shifts, while others need to capture long-range variation gradually.
For position pos and dimension pair k, the angle is typically defined as:
theta(pos, k) = pos / base^(2k / head_dim)
Most implementations use base = 10000, though this varies by model. The rotation is applied to the query and key only — not to the value. The value holds the content to be aggregated; the query and key act as address-comparison vectors that determine how much attention each token pays to every other.
Figure 2. RoPE groups the head dimension into pairs and rotates each pair at a different rate according to position.
Why RoPE captures relative position well
RoPE's advantage comes from a property of rotations. Rotate one vector by i and another by j, then take their dot product — the difference between the two rotations is reflected in the result. And the difference in rotation angle maps directly to the difference in position.
rotate(q, i) dot rotate(k, j)
This value has a component that depends on i - j. Even without going deep into the math, the intuition is clear. Rotate two objects by the same angle and their relative orientation is preserved. Rotate one by 10 degrees and the other by 20, and there is a 10-degree difference between them. RoPE encodes that difference in the attention score.
Because of this property, even if a few tokens are prepended to a sentence and all absolute positions shift, the relative distance between tokens stays intact. If 밥을 and 먹었다 were originally at positions 1 and 2 and then get pushed to positions 5 and 6, their distance is still 1. That information matters when the model is capturing syntactic relationships.
That said, RoPE does not solve every positional problem. For tokens that are far apart, rotation angles accumulate significantly, and some dimension pairs complete multiple full revolutions. Rotation is periodic — push far enough and two distinct positions can end up looking like similar rotation states. This is one reason why performance can degrade on long contexts.
RoPE and sequence length extension
RoPE comes up frequently in discussions of long-context extension. When you want to use a model trained on 4k context at 32k, 128k, or beyond, several techniques for adjusting RoPE have emerged.
One important caveat: using RoPE does not give you longer sequence lengths for free. A model is calibrated to work reliably within the positional range it saw during training. Feed a 4k-trained model raw 32k position indices and the rotation angle distribution shifts away from what it learned. The attention score statistics change, and the model is effectively reasoning in an unfamiliar coordinate space.
Long-context extension with RoPE therefore generally involves compressing position indices or adjusting rotation frequencies.
The most intuitive approach is position interpolation: squeeze 32k inputs into the 4k coordinate space. Instead of feeding the raw position pos to RoPE, you scale it down — something like pos / scale. Position Interpolation is the canonical method in this family.
original: pos = 0, 1, 2, ..., 32767
scaled: pos = 0, 0.125, 0.25, ..., 4095.875 # scale = 8인 예시
This keeps the model within the positional range it was trained on. Even with 32k tokens, the rotation angles stay near the 4k range from the model's perspective. The apparent sequence length grows by 8×.
The problem is resolution.
Compressing positions also compresses the spacing between nearby positions. Two tokens that were 1 step apart now appear to be 0.125 steps apart in the RoPE coordinate space. You can feed in longer documents, but the position distinction between nearby tokens becomes coarser. Tasks sensitive to the order of adjacent tokens — short sentences, code, mathematical expressions, tables — can suffer.
This is why real long-context fine-tuning rarely amounts to changing a single scale factor. Corrections are needed so that some dimensions preserve short-range resolution while others stay stable at long range. YaRN combines RoPE scaling with fine-tuning to reduce the cost of long-context adaptation, and LongRoPE refines the position interpolation at a finer granularity to push context lengths even further.
Figure 3. RoPE scaling enables longer contexts but creates a trade-off between positional resolution and inference cost.
"Fits in the context window" ≠ "understands it well"
Saying a model supports a longer context length typically means it can accept a longer input. Whether it uses all the information in that input equally well is a separate question.
Anyone who has used long-context models has probably run into the pattern where information in the middle gets lost. The beginning and end are attended to reasonably well, but fine-grained conditions buried in the middle get missed. This phenomenon — known as "Lost in the Middle" — appears even in models that officially support long contexts. The Lost in the Middle paper documents this carefully.
RoPE scaling extends the range of position representations. Attention computation still scales roughly as the square of sequence length. KV cache grows proportionally with length. A model that ran fine at 8k becomes a very different problem at 128k in terms of memory, latency, and throughput. FlashAttention and similar implementation optimizations help, but the cost of processing every token in a long input does not disappear.
Long context is also not a replacement for RAG. Feeding an entire document is simple and powerful in theory, but if the retrieval quality for selecting relevant documents is low, a longer context window just means more noise. The longer the context, the more important it becomes to decide what not to include. The model processes every token in the input at the same computational cost.
The trade-offs of RoPE scaling
The trade-offs of extending sequence length with RoPE are fairly concrete.
- Longer documents fit, but positional resolution degrades.
- Extrapolating to lengths not seen during training destabilizes the attention score distribution.
- Aggressive scaling can hurt short-context performance.
- Long-context fine-tuning is sensitive to data quality and packing strategy.
- Inference cost increases substantially with context length.
- Strong benchmark scores at long context do not guarantee equivalent quality on real workloads like enterprise document retrieval, code editing, or multi-turn agent memory.
In my view, claiming that RoPE scaling alone solves the long-context problem is an overstatement. RoPE scaling is a technique for extending the positional coordinate space. Long-context quality is jointly determined by training data, attention implementation, cache policy, retrieval, and instruction tuning. When adding long-context adaptation on top of a model that already has solid post-training, you must verify that short-input response quality has not regressed.
What RoPE looks like in code
The code below is a minimal example showing the shapes involved when applying RoPE. Real model implementations are more complex due to caching, partial rotary dimensions, dtype handling, and fused kernels, but the flow is the same.
import torch
def rotate_half(x):
x1 = x[..., ::2]
x2 = x[..., 1::2]
return torch.stack((-x2, x1), dim=-1).flatten(-2)
def apply_rope(x, cos, sin):
# x: [batch, heads, seq_len, head_dim]
# cos: [1, 1, seq_len, head_dim]
# sin: [1, 1, seq_len, head_dim]
return (x * cos) + (rotate_half(x) * sin)
batch = 1
heads = 4
seq_len = 8
head_dim = 16
q = torch.randn(batch, heads, seq_len, head_dim)
k = torch.randn(batch, heads, seq_len, head_dim)
positions = torch.arange(seq_len, dtype=torch.float32)
inv_freq = 1.0 / (10000 ** (torch.arange(0, head_dim, 2).float() / head_dim))
freqs = torch.einsum("i,j->ij", positions, inv_freq)
emb = torch.cat((freqs, freqs), dim=-1)
cos = emb.cos()[None, None, :, :]
sin = emb.sin()[None, None, :, :]
q_rope = apply_rope(q, cos, sin)
k_rope = apply_rope(k, cos, sin)
scores = torch.matmul(q_rope, k_rope.transpose(-1, -2))
print(scores.shape) # [1, 4, 8, 8]
The notable absence here is the value tensor. RoPE is applied to the query and key before the attention score is computed. It does not touch the input embedding table and has nothing to do with the vocabulary.
Common points of confusion when first learning RoPE
RoPE does not replace token embeddings. Token IDs still go through the embedding table to become hidden vectors. RoPE then rotates the query and key inside the attention layer.
RoPE does not automatically guarantee long-context quality. If you take a 4k-trained model and simply change a configuration value to allow 128k input, the input will be accepted but the output quality may collapse. Context window size is an API spec; long-context reasoning is a model capability.
RoPE scaling can produce some benefit without fine-tuning, but stable extension typically requires additional training or an adaptation procedure. Real long-context use cases — document QA, codebase understanding, agent logs — are far messier than simply retrieving a token from a distant position.
The next part moves into self-attention, where it becomes clearer why RoPE belongs on the query and key specifically. Attention computes a relationship score for every pair of tokens, and RoPE inscribes positional distance into those scores.