Speculative Decoding: How LLMs Generate Multiple Tokens at Once

Speculative Decoding Core Principles: Why LLMs Can Generate Multiple Tokens at Once

The most important bottleneck in LLM inference is the requirement to confirm tokens one at a time. Transformers can process multiple tokens in parallel during training, but during generation, the probability distribution over the next token can only be computed once the preceding tokens are fixed. This is called autoregressive decoding.

Speculative decoding was developed to work around this sequential bottleneck. A small predictor generates several candidate tokens first, and a larger model verifies those candidates in a single pass. When the candidates align well with the target model's distribution, the effect is equivalent to confirming multiple tokens at once. When they don't, a stochastic correction is applied to preserve the distribution that the target model would have produced on its own.

Figure 1. The basic structure: a small draft model generates candidate tokens, and a large target model verifies them.

This is Part 1. Before covering how to apply speculative decoding and the characteristics of specific models, this post focuses on exactly what problem speculative decoding solves and how it works. The core concepts come down to five words: draft, target, verify, accept, and reject.

1. Why LLMs Generate One Token at a Time

LLMs don't complete a sentence all at once. Given an existing token sequence, the model computes a probability distribution over the next token and selects one. The selected token is appended to the input, and the model uses that new prefix to compute the next token.

For example, given the prefix The weather today is, the model assigns probabilities to candidate next tokens — nice, clear, cloudy, and so on. Once one is selected and the prefix becomes The weather today is nice, only then can the model compute the candidates for the following step: and, ,, ., etc.

In this structure, generating 100 tokens requires, in principle, 100 decoding steps. Even with a KV cache eliminating the need to recompute keys and values for previous tokens, one full forward pass through the large model is still required to confirm each new token.

Attention Is All You Need, which introduced the Transformer architecture, showed how attention efficiently computes intra-sequence dependencies. But autoregressive generation cannot assume that future tokens are already known. A token only has meaning once the tokens before it have been determined.

2. The Bottleneck Is Not Just Compute

Attributing LLM inference latency purely to FLOPs misses the point. At every decoding step, a large model reads enormous weights, computes activations, and updates the KV cache. When batch sizes are small and user-facing latency matters, it's hard to keep GPU compute units saturated.

In this setting, the target model behaves like a very expensive compute device. The problem is that this expensive device is often invoked just to confirm a single token. The target model is responsible for output quality, but using the full target model for one-token verification at every step is inefficient.

This observation is the starting point for speculative decoding. Individual token generation is inherently sequential, but if multiple candidate paths can be generated ahead of time and evaluated by the large model in a single pass, the number of tokens confirmed per target model call can be increased.

3. The Question Researchers Asked

The core question is simple:

Instead of using the large model to generate one token at a time, can we make it verify multiple candidate tokens at once?

This question didn't emerge with LLMs. Blockwise Parallel Decoding for Deep Autoregressive Models explored predicting and verifying candidates at multiple positions in parallel for autoregressive models. It didn't eliminate the sequential constraint entirely, but it was an early attempt to handle multiple tokens in blocks.

As LLMs grew in scale, the problem became more pressing. Fast Inference from Transformers via Speculative Decoding formalized the structure where a small draft model generates candidates and a large target model verifies them. Accelerating Large Language Model Decoding with Speculative Sampling presented a procedure for increasing speed while preserving the target model's distribution even under sampling.

The key insight across this line of work is that this is not a simple speed-quality tradeoff. The goal is to maintain the target model's output distribution while reducing the number of target model calls, or reducing wasted work per call.

4. The Basic Structure Through a Minimal Example

The simplest form of speculative decoding uses two models:

Draft model: A small, fast model that generates several candidate tokens first.
Target model: The large model responsible for actual output quality. It verifies the candidates proposed by the draft model.

Suppose the current prefix is A. The draft model generates three candidate tokens:

prefix: A
Draft proposal: x1, x2, x3

In standard decoding, the target model would need to be called once to generate x1, once for x2, and once for x3. In speculative decoding, the target model receives the full sequence A x1 x2 x3 as input and computes the next-token distribution at each position in a single forward pass.

The target model simultaneously evaluates:

Does x1 make sense after A?
Does x2 make sense after A x1?
Does x3 make sense after A x1 x2?

Transformers compute logits at multiple positions in the input sequence in parallel. This property is what makes it possible to verify draft candidates in a single pass. Generation is sequential, but verification can be parallelized.

Figure 2. The repeating loop of candidate generation, parallel verification, accept/reject, and correction sampling.

5. Why a Draft Model Is Needed

The draft model is not a secondary model that needs to be correct. Its role is to generate plausible candidates cheaply. It proposes several tokens ahead of time much faster than the target model, so the expensive target model can focus on verifying those proposals.

A good draft model satisfies two conditions:

It is fast enough relative to the target model.
Its token distribution is close enough to the target model's.

Meeting only the first condition doesn't produce speed gains. No matter how fast the draft model is, if its proposals frequently diverge from the target model, rejections accumulate. That reduces the number of tokens confirmed per target model call.

Meeting only the second condition isn't sufficient either. If the draft model closely matches the target model in distribution but is similar in size, the cost of generating candidates is too high. The benefit of speculative decoding comes from keeping draft generation cheap while ensuring enough candidates are accepted.

6. What the Target Model Is Verifying

The target model is not just checking whether draft tokens are grammatically natural. It compares the probability distribution it would have assigned to the next token on its own against the distribution proposed by the draft model.

This distinction matters. Speculative decoding is not "a large model spell-checking a draft written by a small model." It is a procedure for deciding whether draft tokens can be accepted based on the large model's own probability distribution.

If the target model assigns high probability to a token and the draft model proposed that same token, acceptance is likely. If the draft model strongly favored a token that the target model considers unlikely, rejection is likely.

The target model is the final quality standard. The draft model does not determine the final answer. The final output is designed to follow the target model's distribution.

7. Accept Is Not Grading for Correctness

The word "accept" in speculative decoding is easy to misinterpret. It does not simply mean that the draft model predicted the target model's argmax. Under sampling, it is more precisely a stochastic acceptance procedure designed to preserve the target distribution.

Let q denote the draft model's distribution and p the target model's distribution. When the draft model proposes a token t, if the target model views that token as at least as likely as the draft model does, the token is accepted with high probability. If the draft model was overconfident but the target model assigns it low probability, the acceptance probability drops.

The key is the relationship between p and q. The closer the draft model's distribution is to the target model's, the more acceptances occur. As the two distributions diverge, rejections increase.

This procedure is what separates speculative decoding from simple approximate inference. A naive approach of just using the small model's outputs more often would pull output quality toward the draft model. Speculative decoding uses the accept/reject mechanism and correction sampling to maintain the target model's distribution.

8. What Happens When a Token Is Rejected

When a rejection occurs, all subsequent draft candidates are discarded. If one token changes, the conditional distribution over everything that follows also changes. In an autoregressive model, the distribution after A x1 and the distribution after A y1 are different problems.

After a rejection, a token is sampled from a corrected distribution: the target model's distribution adjusted to remove probability mass that the draft model had already over-represented. This correction step is what ensures the overall output has the same distribution as if the target model had sampled on its own.

The intuition is:

If the draft model proposed a token and the target model agreed, it passes through as-is.
If the draft model pushed a token too strongly but the target model didn't agree, it's rejected.
At the rejected position, the missing probability mass is redistributed according to the target model.

In this structure, rejection is not failure. It is a safeguard for preserving the target model's distribution. That said, if rejections happen too frequently, the speed benefit diminishes.

9. Greedy Decoding Makes It Simpler to See

Greedy decoding selects the highest-probability token at every step. Under greedy decoding, speculative decoding is relatively straightforward to understand. The draft model proposes several tokens, and the target model checks whether its own argmax at each position matches.

For example, if the draft model proposes x1, x2, x3 and the target model agrees that each of those is its highest-probability token at the corresponding position, all three can be accepted. If the target model's argmax differs at the second position, only the first token is accepted, and from that point the target model's choice takes over.

Under greedy decoding, outputs are deterministic, making the verification criterion easy to explain intuitively. In practice, most production systems use sampling, and in that setting, simple argmax matching is not sufficient.

10. Under Sampling, Preserving the Distribution Is What Matters

Sampling doesn't always pick the highest-probability token. Settings like temperature, top-k, and top-p introduce randomness into token selection. In this regime, speculative decoding cannot simply check whether the draft token matches the target model's most likely token.

What matters under sampling is the distribution of final outputs. The distribution over outputs when speculative decoding is applied must match the distribution that would result from the target model sampling on its own. If this condition breaks, throughput may improve but the model's behavior changes.

Speculative sampling uses the ratio of draft and target distributions to decide whether to accept a token, and samples from a corrected distribution on rejection. Because of this procedure, speculative decoding can be understood as an optimization that preserves the target model's distribution.

Higher temperature and wider top-p settings increase candidate diversity. In these cases, the gap between the draft and target distributions can be more pronounced, leading to lower accept rates. At lower temperatures or for more formulaic outputs, accept rates tend to be more stable.

11. Why It Appears to Generate Multiple Tokens at Once

Speculative decoding doesn't actually know the future or generate multiple tokens simultaneously. More precisely, a small model guesses future candidates first, and a large model verifies those guesses in parallel.

The speed benefit comes from increasing the average number of tokens confirmed per target model call. If the draft length is 4 and an average of 3 tokens are accepted, one target model call effectively confirms about 3 tokens. The actual speedup is smaller than the theoretical ratio, since the cost of running the draft model and verification overhead must be accounted for.

The key metric is the number of tokens accepted per target forward pass. If this value stays near 1, there is little difference from standard decoding. When it consistently reaches 2 or more, meaningful latency improvements become possible.

12. When This Technique Actually Delivers Speedups

Speculative decoding is not always faster. The following conditions need to hold for it to be effective:

The target model's per-step cost is high.
The draft model is significantly faster than the target model.
The draft and target model distributions are close to each other.
The output is long enough.
Rejections don't occur frequently early in generation.
The service is latency-sensitive for small batches.

Conversely, the gains shrink when the target model is already small, outputs are very short, there isn't enough memory to run a draft model, or the draft and target models use different tokenizers or chat templates.

Speculative decoding is not a toggle-it-on-and-it's-always-faster option. The number of tokens accepted per target model call must outweigh the cost of running the draft and the associated overhead.

13. What Problem Does This Technique Historically Address?

The history of speculative decoding reflects a recurring tension in LLM inference optimization. As models grow larger, quality improves — but the sequential nature of autoregressive generation remains unchanged. Rather than trying to eliminate that sequentiality entirely, researchers chose to focus on calling large models more efficiently.

Blockwise parallel decoding was an early attempt to treat multiple positions as a block. Speculative decoding and speculative sampling combined fast candidate generation from a small model with distribution-preserving verification from a large model. Later, Medusa proposed attaching multiple prediction heads instead of using a separate draft model, and EAGLE explored more efficient candidate generation through feature-level extrapolation.

The common thread across all of these is that none of them trade quality for speed through simple approximation. The goal is to keep the target model's behavior intact while getting the GPU to do more work in a single pass.

14. One Sentence for Beginners to Remember

Speculative decoding is a method in which a small predictor cheaply generates multiple token candidates, a large model verifies those candidates in parallel, and the number of decoding steps is reduced while preserving the final output distribution.

Everything essential is captured in that sentence.

Small predictor: the draft model, or whatever candidate generator plays that role.
Multiple token candidates: a few future tokens are guessed ahead of time.
Large model: the target model.
Parallel verification: exploits the Transformer's ability to compute logits at multiple positions in a single forward pass.
Preserving the output distribution: the purpose of acceptance, rejection, and corrective sampling.
Reducing decoding steps: the effect of increasing the number of tokens confirmed per target model call.

15. What Part 2 Will Cover

Part 1 focused purely on the underlying principles. In practice, how to generate candidates becomes its own problem. Options include attaching a small draft model, reusing n-grams from the prompt, adding multi-token prediction heads as in Medusa, or using feature-level drafting as in EAGLE.

Part 2 covers each of the following separately:

Attaching a small draft model
N-gram and prompt lookup approaches
Medusa-style multi-token prediction heads
EAGLE-style feature-level drafting
Model-specific characteristics: decoder-only, code, chat, multilingual, encoder-decoder, and MoE models
Practical considerations for vLLM and Hugging Face Transformers
Criteria for evaluating draft length, accept rate, and latency

Applying speculative decoding correctly requires keeping the principles and the implementation choices distinct. The principle is to increase the efficiency of target model calls through parallel verification while preserving the target model's output distribution. The implementation is a question of which candidate generator to use to put that principle into practice.