Transformers don't read a sentence one token at a time in sequence. RNNs process tokens from left to right and naturally capture order as part of their computation, but Transformer self-attention computes relationships among all tokens simultaneously. This makes the architecture highly parallelizable, but without explicitly providing order information, the model has no way to distinguish "the cat ate the fish" from "the fish ate the cat" based on structure alone.
Sinusoidal positional encoding solves this problem by adding position information directly to the Transformer input. It is the baseline positional encoding used in Attention Is All You Need, and it works by converting each token's position into a vector of sine and cosine values at multiple frequencies, then adding that vector to the token embedding.
The key insight I keep coming back to when explaining this mechanism is that it does not stamp a single position number onto each token — it represents position as a combination of signals at different wavelengths. With that framing, the formula becomes much less mysterious.
Per-position patterns of sinusoidal positional encoding. Source: Naoki Shibuya
1. Why Transformers need positional information
A sentence is not a bag of words — it is an ordered structure. The same words in a different order can carry an entirely different meaning.
Consider these two sentences: they share almost the same words but mean opposite things.
- The cat ate the fish.
- The fish ate the cat.
Humans parse meaning by reading both the words and their positions together. A model must do the same: it needs to know not just what each token is, but where in the sequence it appears.
RNN-based models handle this naturally because they process tokens sequentially — position is baked into the computation flow. Transformer attention, by contrast, compares all token pairs at once. That is powerful and fast, but without an explicit mechanism, the notion of "which token comes first" is lost entirely.
Positional information injected at the Transformer input stage. Source: Gowri Shankar
2. The naive approach: just add a position index
The simplest idea is to label each token with its index — 0, 1, 2, 3, and so on. But feeding a raw integer has real problems.
First, a position index is dimensionally incompatible with a token embedding. Embeddings are typically vectors of hundreds of dimensions; a position index is a single integer. You cannot just add them together as-is.
Second, large indices create scale problems. Treating position 10 and position 10,000 consistently under the same scheme is non-trivial.
Third, models need relative position, not just absolute position. The relationship "three tokens back from the current one" matters enormously for language understanding.
So instead of using raw position indices, Transformers map position to a vector of the same dimensionality as the token embedding. Sinusoidal positional encoding performs this mapping using sine and cosine functions.
Token embedding and positional encoding are summed to form the Transformer input. Source: Naoki Shibuya
3. The formula is long, but the idea is simple
The sinusoidal positional encoding from the paper is defined as:
PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i + 1) = cos(pos / 10000^(2i / d_model))
pos is the token's position in the sequence — 0 for the first token, 1 for the second, and so on.
i is the dimension index within the vector. Even-indexed dimensions get sine values; odd-indexed dimensions get cosine values.
d_model is the embedding dimension of the Transformer. If d_model = 512, the positional encoding is also a 512-dimensional vector.
The formula looks dense, but the core idea is one thing: represent position pos as a superposition of waves at different rates. Lower dimensions oscillate quickly; higher dimensions change slowly.
A sine wave converts position into a continuous signal value — the basic building block of this encoding. Source: All About Circuits
4. Sine and cosine turn position into a combination of waves
Sine and cosine are periodic functions, which might prompt the concern: "won't positions repeat and become ambiguous?" The key is that sinusoidal positional encoding never relies on a single sine value.
It uses many sine and cosine waves at different frequencies simultaneously. Some dimensions oscillate with a short period; others vary slowly over a long period. The combination of many wavelengths gives each position a reasonably unique fingerprint.
An analogy: instead of representing a position by a single color, represent it as a mix of many colors in specific ratios. One shade of red looks similar to many others, but specifying the proportions of red, green, blue, brightness, and saturation lets you distinguish far more states.
This structure connects loosely to the idea behind Legendre approximation. Legendre approximation represents a function as a linear combination of orthogonal polynomial basis functions. Sinusoidal positional encoding represents position as a combination of trigonometric components at multiple frequencies. Both decompose a complex object into a sum of basis components — they share the same broad conceptual framework.
Positional encoding heatmap varying across positions and dimensions. Source: QuarkML
5. The positional vector is added to the token embedding
The Transformer input is formed by summing the token embedding and the positional encoding:
input_vector = token_embedding + positional_encoding
The token embedding captures what the token is. The positional encoding captures where it appears. Adding them gives the model both meaning and position in a single vector.
For example, the token "cat" has the same embedding regardless of where it appears in the sentence. But because the positional encoding differs, the final input vector the Transformer sees is different when "cat" is the first token versus the fifth.
This approach is simple but effective. It injects position information into the input without modifying the model architecture at all. Because self-attention operates on these vectors, position information is indirectly reflected in every attention computation.
The positional encoding as a matrix indexed by position and dimension. Source: Next Electronics
6. Why use both sine and cosine?
Sine alone would still produce different values at different positions. The reason to include cosine is that it provides a second signal at the same frequency but 90 degrees out of phase — two quadrature components instead of one.
This pairing is useful for expressing positional offsets. The paper argues that sinusoidal encoding makes it easy for the model to learn to attend by relative position, because PE(pos + k) can be expressed as a linear function of PE(pos) for any fixed offset k.
In plain terms: comparing the positional vectors of two tokens leaves behind information about how far apart they are in a form that is easy for the model to use.
Knowing only absolute position tells you "this token is the 10th." Being able to work with relative position easily lets you express "this token is 3 positions after that one" — a relationship that matters constantly in language.
The perspective needed when thinking about both absolute and relative positions in positional encoding. Source: nomulog
7. How does this differ from learned positional embeddings?
There are two broad approaches to adding position information:
- Compute position vectors from a fixed formula.
- Treat position vectors as learned parameters and train them.
Sinusoidal positional encoding is the first approach. It does not change during training. Given a position index and a dimension index, the formula always produces the same value.
Learned positional embeddings are the second approach. The model adjusts each position's vector directly based on the training data. This can produce position representations that are well-tuned to the data and task, but generalization to sequence lengths longer than those seen during training requires additional consideration.
The advantage of the sinusoidal approach is that it is formula-based: no extra parameters, and in principle you can compute a value for any position, even positions beyond the training length. In practice, how well long-sequence generalization actually works depends on the model architecture, training data, attention implementation, and context length configuration together.
A comparison of different approaches to representing positional information. Source: nomulog
8. Parts of the formula that are easy to miss
First, positional encoding is not an algorithm that sorts or reorders tokens. It is an input representation scheme that attaches a position signal to tokens that are already in a given order.
Second, sine and cosine are not decorative. They are the mechanism for projecting position into a high-dimensional vector by combining waves at multiple frequencies.
Third, positional encoding alone does not make the model understand a sentence. It is one ingredient. The actual semantic relationships emerge from self-attention, feed-forward layers, normalization, and the learned weights working together.
Fourth, saying "Transformers don't know position" means more precisely that the architecture does not process order automatically by construction. Once you add positional information to the input, self-attention operates on vectors that already carry that order.
9. What the approximate inference perspective reveals
Sinusoidal positional encoding converts a discrete value — position — into a combination of continuous function values. Each position becomes not a single integer but a vector of wave components.
This connects to the broader idea of approximation. When a complex object is hard to work with directly, you can map it into a combination of simpler, tractable basis components. Just as Legendre approximation compresses a function into polynomial basis coefficients, sinusoidal positional encoding unfolds position into a pattern of trigonometric basis functions.
The Transformer does not directly interpret these positional patterns. It extracts the relationships it needs through learned attention weights and linear projections. So positional encoding is less "the result of understanding position" and more "the coordinate system that makes position understandable."
10. One-sentence summary
Sinusoidal positional encoding converts each position into a high-dimensional vector of sine and cosine values at multiple frequencies, then adds that vector to the token embedding so the Transformer can distinguish token order.
The structure matters more than the formula. A Transformer has no built-in notion of sequence order — so position must be injected explicitly. Sine and cosine let each position be expressed as a combination of waves at different wavelengths, giving the model input from which it can learn both absolute position and relative distance.