Transformers don't read a sentence token by token in sequence. RNNs process left to right and naturally encode order as part of their computation, but Transformer self-attention considers all tokens at once and computes relationships between every pair simultaneously. This architecture parallelizes well, but without explicitly providing token order, the model has no structural way to distinguish "the cat ate the fish" from "the fish ate the cat."
Sinusoidal positional encoding solves this by adding position information directly to the Transformer input. It is the baseline positional encoding used in Attention Is All You Need, and it works by converting each token's position into a vector of sine and cosine values at multiple frequencies, then adding that vector to the token embedding.
The key insight, when I think about this mechanism, is not that position is stamped in as a single number, but that each position is represented as a combination of signals at different wavelengths. From that angle, the formula feels much less arbitrary.
Per-position patterns in sinusoidal positional encoding. Source: Naoki Shibuya
1. Why Transformers need positional information
A sentence is not a bag of words — it is an ordered structure. The same words in different positions carry different meanings.
Consider these two sentences, which share nearly the same words but mean opposite things:
- The cat ate the fish.
- The fish ate the cat.
Humans use word order (and, in languages like Korean, case markers) to resolve meaning. A model likewise needs to know not only what each token is, but where it appears in the sequence.
RNN-based models process tokens sequentially, so position is implicitly encoded in the computation flow. Transformer attention, by contrast, compares all token pairs simultaneously. That approach is fast and powerful, but without an explicit mechanism, the information "which position is this token at?" is simply absent.
Positional information is injected at the Transformer input stage. Source: Gowri Shankar
2. The naïve idea: just add the position index
The simplest approach is to assign each token an index — 0, 1, 2, 3, … — and add it to the token representation. This runs into several problems.
First, a scalar position index is incommensurate with a token embedding. Embeddings are vectors with hundreds of dimensions; a position index is a single integer. Adding them directly makes no representational sense.
Second, large position indices create scale problems. Treating position 10 and position 10,000 consistently becomes difficult.
Third, the model needs relative position information, not just absolute positions. Relationships like "three tokens earlier than the current one" are critical for language understanding.
Transformers therefore don't use raw position indices. Instead, they map each position to a vector with the same dimensionality as the token embedding. Sinusoidal positional encoding performs that mapping using sine and cosine functions.
Adding positional encoding to a token embedding to form the Transformer input. Source: Naoki Shibuya
3. The formula looks long, but the idea is simple
The sinusoidal positional encoding defined in the paper is:
PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i + 1) = cos(pos / 10000^(2i / d_model))
pos is the token's position in the sequence — 0 for the first token, 1 for the second, and so on.
i is the dimension index within the encoding vector. Even dimensions get sine values; odd dimensions get cosine values.
d_model is the embedding dimensionality used by the Transformer. With d_model = 512, the positional encoding is also a 512-dimensional vector.
The formula looks complicated, but the core idea is straightforward: represent position pos as a superposition of oscillations running at different speeds. Lower dimensions use fast-varying waves; higher dimensions use slow-varying waves.
A sine wave is the basic building block for converting a position into a continuous signal value. Source: All About Circuits
4. Sine and cosine turn a position into a combination of waves
Sine and cosine are periodic functions. At first glance, periodicity seems problematic — won't two different positions map to the same value? The key is that sinusoidal positional encoding doesn't use a single sine wave.
It uses sine and cosine at many different frequencies simultaneously. Some dimensions oscillate rapidly with a short period; others change slowly with a long period. Together, the combination of wavelengths produces a pattern that is distinct for each position.
An analogy: rather than identifying a color by a single hue, you represent it as proportions of multiple color channels. One shade of red looks like many others in isolation, but specifying red, green, blue, brightness, and saturation together distinguishes far more states.
This structure also connects to Legendre approximation. Legendre approximation expresses a function as a combination of orthogonal polynomial basis functions. Sinusoidal positional encoding expresses a position as a combination of trigonometric basis functions at different frequencies. Both approaches handle a complex object by decomposing it into a sum of basis components — the same broad conceptual framework.
Positional encoding heatmap varying by position and dimension. Source: QuarkML
5. The position vector is added to the token embedding
The Transformer input is formed by adding the token embedding and the positional encoding:
input_vector = token_embedding + positional_encoding
The token embedding carries "what is this token?" The positional encoding carries "where is this token?" Together, the model receives both meaning and position in a single vector.
For example, the token "cat" at position 0 and "cat" at position 4 have the same token embedding, but different positional encodings, so the final input vectors the Transformer sees are different.
This is simple but effective. Position information is injected into the input without any architectural changes. Self-attention then operates on these combined vectors, so position is indirectly reflected throughout the attention computation.
Positional encoding as a matrix indexed by position and dimension. Source: Next Electronics
6. Why use both sine and cosine?
Sine alone would give different values at different positions. Adding cosine provides a second signal at the same frequency but 90 degrees out of phase.
This pairing is advantageous for representing positional offsets. The paper notes that using sinusoidal encoding makes it easy for the model to learn to attend to relative positions, because PE(pos + k) can be expressed as a linear function of PE(pos) for any fixed offset k.
For a more intuitive reading: comparing the positional vector of one token to that of another leaves the "distance between them" in a form the model can readily compute. Knowing only absolute positions gives you "this token is at index 10." Being able to handle relative positions easily gives you "this token is 3 steps after that one" — which is often far more useful for language understanding.
Positional encoding viewed through the lens of both absolute and relative position. Source: nomulog
7. How does this differ from learned positional embeddings?
There are two main approaches to injecting positional information:
- Compute position vectors from a fixed formula.
- Treat position vectors as parameters and learn them from data.
Sinusoidal positional encoding is the first approach. It does not change during training. Given a position index and a dimension index, the formula always returns the same value.
Learned positional embeddings take the second approach. The model adjusts each position vector directly during training. This can yield position representations well-suited to the data and task, but generalizing to sequence lengths longer than those seen during training requires additional consideration.
The advantage of the sinusoidal approach is that it is formula-based: it adds no trainable parameters, and in principle it can produce values for positions beyond the training length. In practice, however, long-sequence generalization depends on model architecture, training data, the attention implementation, and the context length configuration together.
Comparison of different approaches to representing positional information. Source: nomulog
8. Parts of the formula beginners often miss
First, positional encoding is not an algorithm that sorts or reorders tokens. It is an input representation scheme that attaches a positional signal to tokens that are already in a given order.
Second, sine and cosine are not decorative choices. They are the mechanism that projects a position onto a high-dimensional vector by combining waves at multiple frequencies.
Third, positional encoding alone does not make the model understand a sentence. It is one ingredient. The actual semantic relationships emerge from self-attention, feed-forward layers, normalization, and learned weights working together.
Fourth, saying "Transformers don't know position" means more precisely that the architecture does not process order automatically. Once positional information is added to the input, self-attention operates on vectors that already carry that order.
9. What the approximate-inference perspective reveals
Sinusoidal positional encoding converts discrete position indices into combinations of continuous function values. Each position becomes not a single integer but a vector of wave components.
This connects to the broader idea of approximation. When a complex object is hard to handle directly, it can be decomposed into a sum of tractable basis components. Just as Legendre approximation compresses a function into coefficients over a polynomial basis, sinusoidal positional encoding unfolds a position into a pattern over a trigonometric basis.
The Transformer does not interpret this positional pattern literally. It extracts the relationships it needs through learned attention weights and linear transformations. In that sense, positional encoding is less "the result of understanding position" and more "the coordinate system that makes position understandable."
10. One-sentence summary
Sinusoidal positional encoding adds position information to Transformer inputs by converting each position into a high-dimensional vector of sine and cosine values at multiple frequencies, then adding that vector to the token embedding.
The structure matters more than the formula. Transformers have no built-in notion of order, so position must be encoded into the input explicitly. Sine and cosine at varying frequencies let each position be expressed as a unique combination of wave components, giving the model input that makes it easy to learn both absolute positions and relative distances.