Transformer: Beyond Next Token Prediction

Transformer: Beyond Next Token Prediction

Transformer is a sequence-to-sequence model proposed in the 2017 paper Attention Is All You Need. It is now commonly described as the backbone of LLMs, but it started with machine translation — reading a sentence in one language and converting it to another, using only attention to build an encoder-decoder architecture without any RNNs or CNNs.

The most important conceptual shift in understanding Transformer is the move from "a model that reads a sentence token by token" to "a model that lays out the entire sentence at once and computes relationships." RNNs processed words left to right, one at a time. Transformer spreads all tokens out simultaneously and computes, for each token, how much attention it should pay to every other token.

Figure 1. Transformer stacks encoder and decoder blocks built around attention.

Starting from the Translation Problem

Machine translation is a problem where input and output sequences can differ in length. For example, translating I love machine learning into Korean yields 나는 머신러닝을 좋아한다, where the word count and word order both change. The model cannot just map words one-to-one; it must capture the full meaning and grammatical structure of the sentence.

Early neural translation models used an encoder-decoder architecture. The encoder reads the source sentence and compresses it into a single representation; the decoder takes that representation and generates the translation one token at a time. Sequence to Sequence Learning with Neural Networks is a seminal example of this approach using an LSTM-based encoder-decoder.

The problem is that compressing an entire sentence into a single fixed-length vector breaks down for long sentences. As length increases, earlier information fades, and it becomes hard for the decoder to directly select which part of the input to attend to when generating a particular output word.

Figure 2. Early neural translation can be described as an encoder compressing the source and a decoder generating the translation from that compressed representation.

Why RNNs Were the Natural Starting Point

RNNs were designed to handle sequential data. A sentence is a sequence of words, and earlier words affect the meaning of later ones. RNNs process a sentence left to right, updating a hidden state at each step.

Simply put, an RNN works like this:

현재 토큰 + 이전 hidden state -> 새로운 hidden state

This structure is intuitive. Read 나는, update state; read 밥을, update state again; then read 먹었다. It mirrors how a person reads a sentence.

From a computational standpoint, however, there is a serious limitation. Each step depends on the previous step's output, so the computation is inherently sequential. GPUs excel at parallelizing large amounts of computation, but RNNs must wait for each time step in order.

There is also the long-range dependency problem. Information from early in a sentence may be needed much later, but as the hidden state is overwritten repeatedly, earlier information can degrade. LSTM and GRU were developed to mitigate this, but the sequential processing structure remains.

Figure 3. An RNN processes a sentence sequentially, passing the hidden state forward at each step.

Attention Was a Fix for the Bottleneck

Attention lets the decoder look back at all of the encoder's hidden states when generating each output token. Instead of compressing the input into a single vector, the model computes a weighted importance score over the input at each decoding step.

Neural Machine Translation by Jointly Learning to Align and Translate proposed an attention-based approach that jointly learns alignment and translation. Alignment here means which parts of the input sentence a given output word is most strongly connected to.

For example, when translating I love machine learning, at the moment of generating 좋아한다, love is the most relevant input. When generating 머신러닝, machine learning is the most relevant. Attention turns these correspondences into numbers.

This is a significant change: rather than passing a static compressed representation, the model can refer back to the input sentence at every generation step.

Figure 4. Attention lets the decoder look back at the relevant parts of the input sentence when generating each output token.

The Core Shift in Transformer

Transformer removes the RNN entirely and puts attention at the center of the architecture — which is exactly what "Attention Is All You Need" means. In earlier seq2seq models, attention was an add-on sitting on top of an RNN. In Transformer, attention itself is the primary computation that builds contextual representations.

Transformer is divided into an encoder and a decoder.

• The encoder reads the source sentence and produces a contextual representation for each token.
• The decoder attends to both the previously generated translation tokens and the encoder output to predict the next token.
• Both encoder and decoder repeatedly stack attention, feed-forward networks, residual connections, and layer normalization.

The original Transformer is a translation model. Its input is a source sentence and its output is a translation. During training, the ground-truth translation is shifted by one position and fed into the decoder, and the model is trained to predict the next token.

입력 원문: I love machine learning
정답 번역: 나는 머신러닝을 좋아한다

Decoder 입력: <BOS> 나는 머신러닝을
예측 목표:   나는 머신러닝을 좋아한다

This is teacher forcing. During training, even if the model produces a wrong token, the next input is always the ground-truth token, keeping training stable. At inference time there is no ground truth, so the model feeds its own generated tokens back as input.

Figure 5. The original Transformer was proposed as an encoder-decoder architecture for translation.

Self-Attention Computes Relationships Within the Sentence

Self-attention lets tokens within the same sentence attend to one another. In the sentence 나는 사과를 먹었다, 먹었다 needs to draw on both 나는 and 사과를. Self-attention produces a score representing how much each token should attend to every other token.

Transformer's attention is described in terms of Query, Key, and Value.

• Query is "what I am currently looking for."
• Key is "the label each token exposes for lookup."
• Value is "the actual content to retrieve."

The Query of one token is compared against the Keys of all other tokens to produce attention scores. A higher score means more of that token's Value is incorporated. Each token's final representation is a weighted sum of the other tokens' Values.

The formula is:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V

QK^T computes pairwise similarity between tokens. Dividing by sqrt(d_k) prevents the dot products from growing so large in high dimensions that the softmax becomes sharply peaked.

Figure 6. Query sets the search criterion, Key provides what each token is indexed by, and Value is the content actually retrieved.

Multi-Head Attention Looks at Multiple Perspectives Simultaneously

Using a single attention pass captures relationships through only one lens. Multi-head attention runs several attention operations in parallel, each in a different projected subspace, allowing each head to learn different types of relationships.

For example, one head might strongly capture subject-verb relationships, another might focus on modifier-noun relationships, and another might emphasize local context between nearby tokens. In practice, heads do not cleanly divide by human-labeled grammar roles, but the key point is that multiple relationship types can be learned in parallel.

The outputs from all heads are concatenated and then linearly projected. This allows Transformer to compute multiple views of intra-sentence relationships simultaneously.

Figure 7. Multi-head attention concatenates the outputs of all attention heads and applies a linear transformation.

Why Positional Encoding Is Necessary

Transformer does not read tokens in order the way an RNN does — it processes all tokens at once. This is great for parallelization, but without explicit position information, the model cannot distinguish 나는 밥을 먹었다 from 밥은 나를 먹었다.

For this reason, a positional encoding is added to the token embeddings before they are fed into the Transformer.

Transformer 입력 = token embedding + positional encoding

Sinusoidal positional encoding generates distinct sine and cosine values for each position and adds them to the token embeddings. This provides position information without any learnable parameters and makes it easy for attention to exploit relative position relationships. This connects directly to the earlier post on sinusoidal positional encoding.

Figure 8. Transformer inputs are formed by adding positional encodings to token embeddings.

The Encoder Converts the Full Source Sentence into Contextual Representations

Each encoder block has two main components:

• Multi-head self-attention
• Position-wise feed-forward network

Self-attention computes relationships among tokens in the source sentence. The feed-forward network applies a nonlinear transformation to each position's representation independently. Residual connections and layer normalization are wrapped around both, enabling deep networks to train stably.

The encoder output is a contextual representation for each token in the input sentence — not a plain word vector, but a vector that reflects the token's relationships with all surrounding words. The translation decoder uses these encoder outputs as a reference when generating the target-language sentence.

Figure 9. The encoder uses self-attention to compute relationships among source tokens.

The Decoder Generates the Translation One Token at a Time

Each decoder block has one more sub-layer than an encoder block:

• Masked multi-head self-attention
• Encoder-decoder attention
• Position-wise feed-forward network

Masked self-attention prevents the decoder from seeing tokens that come after the current position. Encoder-decoder attention determines which parts of the encoder output the decoder's current state should attend to.

During translation training, the full target sentence is known. But if the model can see future ground-truth tokens when predicting a given token, it is effectively copying the answer rather than learning. A causal mask is therefore applied so that each position can only attend to positions at or before it.

Figure 10. The decoder applies masked self-attention to prevent attending to future tokens, then attends to the encoder output.

What Masks Block

In Transformer, masks are applied to the attention score matrix. This matrix records how much each token attends to every other token. Inserting a very large negative value at specific positions drives their softmax probability to nearly zero, effectively blocking attention there.

There are two main types of masks:

• Padding mask: prevents attention to padding tokens.
• Causal mask (also called look-ahead mask): prevents the decoder from attending to future tokens.

The padding mask ignores <PAD> tokens that are inserted to make all sequences in a batch the same length. These tokens carry no meaning and should not be attended to.

The causal mask is especially important for generative models. The model should predict 좋아한다 given 나는 머신러닝을, not by peeking at the answer — letting it see future ground-truth tokens turns learning into answer-copying.

Figure 11. The causal mask blocks the upper-triangular region of the attention matrix, preventing any position from attending to future tokens.

How Masks Are Applied During Translation Training

Transformer translation training proceeds in the following order:

• Tokenize the source sentence with a tokenizer.
• Tokenize the target (ground-truth) sentence with a tokenizer.
• Shift the target sentence one position to the right to construct the decoder input.
• The decoder output predicts the next-token distribution at each position.
• Compute the cross-entropy loss between the predicted distribution and the ground-truth tokens.
• Padding positions are excluded from both the loss computation and attention.
• A causal mask is applied to decoder self-attention to hide future tokens.

In short, the decoder repeatedly solves the following problem:

<BOS> -> 나는
<BOS> 나는 -> 머신러닝을
<BOS> 나는 머신러닝을 -> 좋아한다
<BOS> 나는 머신러닝을 좋아한다 -> <EOS>

During training, all of the above inputs are processed in parallel — not sequentially like an RNN, where the second prediction must wait for the first. The causal mask ensures that each position can only attend to the ground-truth tokens that precede it. This is the core of how Transformer training works.

Figure 12. During translation training, the target sentence is shifted right to form the decoder input, and the model predicts the next token at each position.

Attention Mask vs. Masked Language Modeling

The word "mask" causes these two concepts to get conflated regularly. The attention mask inside a Transformer controls which tokens a given position is allowed to attend to. BERT's masked language modeling, by contrast, is a pre-training objective that replaces a subset of input tokens with [MASK] and trains the model to recover the original tokens. BERT used a masked language model combined with next sentence prediction to learn bidirectional encoder representations.

GPT-style causal language models use a causal mask to prevent attention to future tokens, training on next-token prediction. Improving Language Understanding by Generative Pre-Training introduced the generative pre-training and task-specific fine-tuning pipeline.

To summarize:

• Attention mask: blocks positions in the attention computation that should not be attended to.
• Causal LM: uses a causal mask to train next-token prediction.
• Masked LM: replaces a subset of inputs with [MASK] and trains the model to recover the originals.
• Translation Transformer: the decoder uses a causal mask; both the encoder and decoder use a padding mask.

Figure 13. Encoder-only, decoder-only, and encoder-decoder models differ in their masking strategies and training objectives.

Inference Uses Autoregressive Generation

During training, the ground-truth translation is shifted right and fed into the decoder. At inference time, no ground-truth sentence is available, so the model generates one token at a time:

Step 1: <BOS> -> 나는
Step 2: <BOS> 나는 -> 머신러닝을
Step 3: <BOS> 나는 머신러닝을 -> 좋아한다
Step 4: <BOS> 나는 머신러닝을 좋아한다 -> <EOS>

At each step, the encoder output remains fixed. The decoder attends to the tokens generated so far and computes the next-token probability distribution. The highest-probability token can be selected via greedy decoding, or multiple candidates can be maintained with beam search.

Beam search is commonly used in translation models. Committing to a single choice at each step can allow early mistakes to compound through the rest of the sequence. Beam search maintains several candidate sentences simultaneously and selects the translation with the best overall score.

Figure 14. Beam search expands multiple candidate sentences in parallel and selects the output with the highest score.

Why Transformer Outperformed RNN

The advantages of Transformer come down to three things.

First, parallelization. RNNs must compute in sequential order, whereas Transformers compute attention over all tokens in a sentence as a single matrix operation. This maps naturally onto GPU hardware.

Second, direct long-range dependencies. In an RNN, information from distant tokens must pass through many intermediate hidden states. Self-attention computes relationships between every pair of tokens directly.

Third, encoder-decoder structure suits translation well. The encoder converts the full source sentence into a contextual representation; the decoder generates the target-language sentence token by token. Attention lets the decoder select the relevant parts of the source at each generation step.

That said, Transformer has its own costs. Self-attention produces an attention score matrix of size n x n for a sequence of n tokens. Computation and memory usage grow large for long documents. This motivated subsequent work such as Longformer, Reformer, and FlashAttention. FlashAttention introduced an IO-aware attention computation that reduces memory access overhead.

Figure 15. RNN computes sequentially over time steps; Transformer computes token relationships as parallel matrix operations.

Structural Significance for LLMs

Transformer originated in translation, but its core insight — computing relationships between tokens via attention — is a general-purpose structure. As long as inputs and outputs are token sequences, the same basic framework applies to translation, summarization, question answering, code generation, and dialogue generation.

Encoder-dominant models excel at tasks that require sentence understanding and classification. Decoder-dominant models excel at next-token generation tasks. Encoder-decoder models are a natural fit for tasks that generate an output sequence conditioned on an input sequence.

• Encoder-only: BERT family, widely used for sentence understanding and classification.
• Decoder-only: GPT family, widely used for next-token generation.
• Encoder-decoder: T5, widely used for tasks that map an input to an output, such as translation and summarization.

T5 unified a broad range of NLP tasks under a text-to-text format, demonstrating the generality of the encoder-decoder Transformer.

Transformer Through the Lens of Graph RAG and Search

Transformer itself is not a Graph RAG algorithm, but attention can be interpreted from a graph perspective. Treating tokens as nodes and attention weights as weighted edges between them, each layer performs information propagation across the token graph.

In Graph RAG, retrieval traverses documents, entities, relationships, and queries to assemble evidence for an answer. Within that assembled context, the Transformer computes token-level relationships. Where Monte Carlo Tree Search and Legendre approximations reduce large search and computation problems approximately, Transformer attention assigns learned weights to important relationships within a given token set.

There is an important distinction. MCTS allocates its search budget by simulating candidate paths. Transformer attention computes relationship weights in a single forward pass using learned parameters. Both share the property of not treating everything equally, but their computational procedures and learning mechanisms are different.

End-to-End Summary

Transformer began as a solution to translation. RNN-based seq2seq read sentences sequentially and accumulated information in hidden states. Attention let the decoder look directly at the relevant parts of the source sentence. Transformer went further — it removed the RNN entirely and made attention the central mechanism.

The encoder converts the full source sentence into a contextual representation. The decoder attends to previously generated tokens and the encoder output to predict the next token. During training, the ground-truth sentence is shifted right and fed in, with a causal mask hiding future tokens. A padding mask excludes meaningless padding tokens from attention and loss computation.

The core idea of Transformer is simple: instead of memorizing a sentence one token at a time in sequence, lay out all tokens at once and compute their relationships. The reason this structure — born in translation — became the backbone of today's LLMs is that it fits parallel computation, long-range dependencies, and general-purpose sequence modeling all at once.