Speculative Decoding in Practice: Draft Model, Medusa, and EAGLE

Speculative Decoding Approaches: Draft Model, Medusa, EAGLE, and a Breakdown by Method

Part 1 covered the core principles of speculative decoding: a small drafter generates several token candidates, a large target model verifies them all in a single pass, and the number of decoding steps is reduced while preserving the final output distribution.

Part 2 covers how to apply these principles in practice. There is one central question:

How does speculative decoding generate candidate tokens, which models does it work well with, and what should you measure in production?

This post does not enumerate features. It traces the problems researchers encountered and the structures that emerged to solve them, in order. Speculative decoding is better understood not as a simple inference option, but as the convergence of several research threads aimed at reducing the bottleneck inherent in autoregressive generation.

Figure 1. Speculative decoding approaches differ based on where candidate tokens come from.

1. How Approaches Are Categorized

Speculative decoding approaches differ based on who generates candidate tokens, what information they use, and at what cost.

The most basic approach uses a separate small draft model. The draft model is smaller and faster than the target model, and it proposes multiple token candidates likely to be generated by the target. The target model then verifies those candidates in a single forward pass.

Other approaches do not use a separate smaller model. They instead reuse repeated token patterns within the prompt, attach multiple token-prediction heads to the target model itself, or use intermediate features such as hidden states to generate the next candidates.

There are four main approaches:

• Attaching a small draft model
• N-gram or prompt lookup
• Medusa-style multi-token prediction heads
• EAGLE-style feature-level drafting

All four target the same problem: reducing the cost of running a deep forward pass through the target model on every token. The tradeoffs differ because the location and cost of candidate generation differs.

2. Research Trajectory: From Small Models to Internal Predictors

Speculative decoding began with a structure that paired a small model with a large one. Fast Inference from Transformers via Speculative Decoding demonstrated that an approximation model could propose multiple tokens for a target model to verify, accelerating inference while preserving the target's output distribution.

Around the same time, Accelerating Large Language Model Decoding with Speculative Sampling formalized the procedure for processing multiple candidate tokens while preserving the target model's distribution through speculative sampling. A key point in this line of work is that the small model's output is not trusted blindly for the sake of speed — candidate tokens must pass the target model's verification step.

Subsequent research expanded in two directions. One direction asked how to make draft models cheaper and better-aligned. The other asked whether candidate tokens could be generated internally within the target model, without a separate draft model at all.

Medusa proposed attaching multiple decoding heads on top of the target model to predict tokens at several future positions simultaneously. The key point is that it uses the target model's internal representations to generate candidates without operating a separate draft model.

EAGLE approaches the problem by predicting the next state at the feature level rather than predicting tokens directly. It starts from the observation that operating at the level of feature uncertainty, rather than token uncertainty, is more favorable for speculative sampling.

In summary, the research trajectory moved from attaching a small auxiliary model toward more aggressively reusing internal information and intermediate features within the target model. The reason is straightforward: the draft must be cheap, it must align well with the target, and operational complexity must stay low.

3. Approach 1: Small Draft Model

The most standard structure uses a separate draft model alongside the target model.

For example, if the target is a 70B model, the draft model might be a 7B, 3B, or 1B model from the same family. The draft model quickly proposes multiple tokens, and the target model verifies those candidates.

The flow is as follows:

1. There is a prefix of tokens generated so far.
2. The draft model generates k candidate tokens for the upcoming positions.
3. The target model receives the prefix and all draft candidates as input and computes the probability at each position.
4. Candidates are accepted or rejected based on the target model's distribution.
5. When a rejection occurs, a correction token is sampled from the target model's distribution.
6. The confirmed tokens are appended to the prefix, and the next round begins.

Figure 2. The small draft model approach separates candidate generation from target verification.

The advantage of this approach is conceptual clarity. A small model drafts; a large model reviews. It is also the easiest structure to explain to someone new to the topic.

The disadvantage is operational cost. Two models are required. The draft model consumes GPU memory and adds to the deployment pipeline. The tokenizer must be compatible with the target model, and candidate quality can degrade if the model families differ.

The most important metric is accept rate. When the draft model frequently agrees with the target model's predictions, the accept rate rises. A higher accept rate means more tokens are confirmed per target model verification pass.

Conversely, when the draft model frequently disagrees with the target, the benefit shrinks. The cost of running the draft model is constant, but if the target rejects many candidates, the number of tokens confirmed per round stays low.

The small draft model approach works well under the following conditions:

• The target model is large enough that generating a single token is expensive.
• The draft model is from the same family as the target, or its distribution is well-aligned.
• Outputs are relatively predictable, leading to a high accept rate.
• There is enough GPU memory to load the draft model alongside the target.
• The serving stack can reliably handle two concurrent model executions.

It is disadvantaged under the following conditions:

• The target model is already small, so per-token cost is low.
• The draft model quality is poor and rejections are frequent.
• Memory is insufficient to load the draft model without it becoming a bottleneck.
• Request length and batch patterns are irregular, increasing scheduling overhead.

4. Approach 2: N-gram or Prompt Lookup

The n-gram or prompt lookup approach does not use a separate model. It reuses token sequences that already appear in the input prompt or prior output as draft candidates.

The idea is simple. In document summarization, code editing, and long-context question answering, expressions from the input frequently reappear in the output. Instead of running a separate neural network, this approach finds token sequences in the prompt that are likely to continue in the output and uses them as draft candidates.

For example, if the input document contains the phrase "speculative decoding uses a draft model and a target model," and the output is likely to repeat that phrasing, prompt lookup can propose the tokens following that phrase as candidates.

The advantage is very low cost. No separate draft model is needed, almost no additional GPU memory is consumed, and the implementation is relatively lightweight.

The disadvantage is narrow applicability. When the prompt contains few reusable patterns, generating good candidates is difficult. For creative generation, general conversation, and reasoning-style responses, accept rates can be low.

The Hugging Face Transformers generation strategies documentation describes prompt lookup decoding as finding overlapping n-grams in the input and using them as candidate tokens. This approach is particularly useful for tasks where the output copies or partially repeats text from the input.

The prompt lookup approach works well under the following conditions:

• Output reuses phrasing from the input document heavily, as in summarization.
• Existing code snippets are repeated, as in code generation or editing.
• There is not enough memory to load a separate draft model.
• Low operational complexity is a priority.

It is disadvantaged under the following conditions:

• The vocabulary or sentence structure differs significantly between the prompt and output.
• New sentences must be constructed continuously, as in conversational responses.
• The input is short, leaving almost no candidates to look up.
• Many repeated candidates exist but do not align well with the target's actual distribution.

N-gram-based approaches represent the simplest application of speculative decoding. The performance ceiling may be lower, but the cost is small and a failure to find good candidates does not significantly increase system complexity.

5. Approach 3: Medusa-Style Multi-Token Prediction Heads

Instead of a separate draft model, Medusa attaches multiple prediction heads on top of the target model. Each head predicts not just the immediately next token, but a token at a specific future position.

While the base language model head predicts the next token, Medusa heads work as follows:

• Head 1: predicts the candidate for position +1.
• Head 2: predicts the candidate for position +2.
• Head 3: predicts the candidate for position +3.
• Head 4: predicts the candidate for position +4.

The candidates produced by these heads can be combined into a tree. The target model verifies this candidate tree and attempts to confirm multiple tokens in a single pass.

The key point of Medusa is that no external draft model is needed. The heads share the target model's hidden state and generate candidates for multiple positions from it. This reduces the burden of serving a separate model.

However, this approach requires additional training. The multiple heads must be trained to predict future tokens accurately. Since this involves attaching heads to the target model and running a training process, it is not simply a matter of toggling a configuration flag.

The Medusa approach works well under the following conditions:

• The target model can be modified and fine-tuned.
• Operating a separate draft model is undesirable.
• Candidates should be generated using the model's own internal representations.
• A specific model is being served stably over a long period.

It is disadvantaged under the following conditions:

• The internal architecture cannot be modified, as with closed-source models.
• It is difficult to repeat head training and validation for each model.
• The operating environment requires frequent model swaps.
• Training data and an evaluation pipeline for the additional heads are unavailable.

Medusa extended speculative decoding from "attaching a small model" to "embedding a fast candidate generator inside the model itself."

6. Approach 4: EAGLE-Style Feature-Level Drafting

Instead of predicting token candidates directly, EAGLE predicts the next state at the feature level. Feature here can be understood as a representation close to the model's internal hidden state.

Tokens are discrete. At any given position, the distribution spans the entire vocabulary and carries high uncertainty. Hidden features, by contrast, hold more continuous information about the next state, which makes a different prediction path possible.

This is the problem EAGLE starts from. The hard part of speculative sampling is getting the draft to produce candidates that align well with the target. Rather than predicting tokens directly, EAGLE predicts the next feature and then derives candidate tokens from that feature.

This approach is more complex, but it has advantages. Because it leverages the target model's internal representations, it has the potential to produce better-aligned candidates than a small independent draft model. It also reduces the burden of dealing with token-level uncertainty directly at the draft stage.

The EAGLE approach works well under the following conditions:

• Internal model features are accessible.
• A high accept rate is critical.
• A candidate generator more tightly coupled to the target model is preferred over a separate draft model.
• Per-model optimization can be performed in a research or advanced serving environment.

It is disadvantaged under the following conditions:

• Implementation complexity must be kept low.
• Internal model access is restricted.
• The same approach must be applied quickly across many different models.
• Training and validating the feature predictor is operationally difficult.

EAGLE illustrates how the direction of speculative decoding research has grown increasingly fine-grained. What began as a small model writing a draft on behalf of the large model evolved toward using the target model's internal representations to generate better-aligned drafts at lower cost.

7. Model Characteristics 1: Decoder-only LLMs

Most general-purpose chat LLMs — GPT, LLaMA, Mistral families — use a decoder-only architecture. Speculative decoding research and implementations have developed primarily around this structure.

In a decoder-only LLM, generation works by appending the next token after the prefix. The target model can take the prefix and draft candidates as a single input and compute logits at every position in one pass. This structure fits naturally with speculative verification.

The key variable in decoder-only models is the KV cache. In LLM serving, the key-value pairs for already-generated tokens are cached to reduce the cost of computing each new token. Because speculative decoding verifies multiple candidate tokens in one shot, KV cache management and rollback handling become critical.

The KV cache for accepted tokens can be kept as-is. Cache entries for positions after a rejected candidate must be discarded or reconstructed. If this process is inefficient, the gains from speculative decoding shrink.

Decoder-only models are the most natural target for speculative decoding. That said, real-world performance varies significantly depending on model size, batch size, draft quality, and cache management.

8. Model Characteristics 2: Code LLMs

Speculative decoding can be particularly advantageous for code LLMs. Code has more repetitive structure than natural language, stronger syntactic constraints, and frequently recurring patterns — indentation, brackets, imports, function names, variable names.

For example, when generating Python code, the indentation of the next line, a closing parenthesis, or a repeated variable name is relatively predictable. The probability that a draft model or a prompt lookup approach produces the same candidate as the target model is higher.

In coding tasks, it is also common for parts of the input context to appear verbatim in the output. Code editing, refactoring, and test generation all reuse existing function names, types, and string literals. This makes prompt lookup a strong candidate in these settings.

That said, good results are not guaranteed for every code LLM scenario. A small deviation can cause an error. Even if the accept/reject procedure preserves the target model's distribution, from a serving-system perspective you need to manage stop sequences, indentation, streaming chunk boundaries, and tool call formats carefully.

For code LLMs, tokens per second alone is not the right metric. The following should all be measured together:

• Accept rate
• End-to-end latency
• First token latency
• Code block completion rate
• Syntax error rate
• Stop sequence handling stability
• Cache efficiency on long file contexts

Code generation is a strong use case for speculative decoding, but quality evaluation must go beyond string similarity to include executability and format stability.

9. Model Characteristics 3: Chat and Instruction Models

In chat models, alignment between draft and target is critical. Even within the same base model family, differences in instruction tuning, RLHF, or system prompt handling can shift the next-token distribution significantly.

For instance, if the target model has been trained to follow a polished style and strict safety policies while the draft model is closer to the base, the candidates will frequently diverge. This lowers the accept rate and reduces the speed benefit.

Output format also matters in chat models. Beyond plain answers, there is a large volume of structured output: JSON, tool calls, function calls, XML-like tags, and Markdown tables. When the draft gets the format right, the benefit is large; when the format is even slightly off, rejections pile up.

Sampling settings have a strong effect here as well. High temperature and wide top-p increase the diversity of possible next tokens, making it harder for the draft to predict the target's choice. Conversely, low temperature or deterministic settings tend to raise the accept rate.

When deploying speculative decoding on chat models, measure the following categories separately:

• General Q&A
• Long-document-based answers
• JSON or tool call generation
• Coding questions
• Multilingual conversations
• Requests that trigger safety policies

Looking at a single average accept rate is risky. Real-world service traffic has a different distribution across request types, and speculative decoding's effectiveness can vary dramatically depending on request type.

10. Model Characteristics 4: Multilingual Models

In multilingual models, accept rate can vary across languages. The reasons are differences in tokenization, training data distribution, and sentence structure.

English is tokenized relatively efficiently by most tokenizers. For languages like Korean, Japanese, Chinese, and Arabic, the number of tokens per sentence and subword boundaries differ depending on the tokenizer design. Even sentences with the same meaning can differ in token length and prediction difficulty across languages.

If a draft model is well-calibrated toward English, the accept rate may be high for English requests and low for others. Conversely, a draft model with strong multilingual instruction tuning may show less variance across languages.

For a language like Korean, particles, verb endings, spacing conventions, Sino-Korean vocabulary, and loanword spellings all affect candidate prediction. If the target model and the draft model favor different styles — even when the meaning is similar — token-level acceptance can be low.

The evaluations needed for multilingual models are:

• Accept rate per language
• Tokens per second per language
• Average output length per language
• Character count relative to tokenizer token count
• Separate evaluation for translation, summarization, conversation, and code-mixing requests
• Stability on requests that mix Korean and English

For multilingual models, per-language distributions matter more than overall averages. Speculative decoding may accelerate only specific languages while providing little benefit for others.

11. Model Characteristics 5: Encoder-decoder Models

In encoder-decoder models like T5 and BART, the encoder processes the input first, and the decoder generates the output tokens. Speculative decoding can be applied to the autoregressive generation on the decoder side.

In this architecture, the encoder output acts as a fixed conditioning signal. The decoder attends to both its previous output tokens and the encoder output to produce the next token. How the draft and target each use the encoder output therefore matters.

When attaching a small draft model, that draft model must also reflect the same input conditioning. If the draft fails to capture the details of the input document, its candidates for summarization or translation tasks can diverge from the target.

Encoder-decoder models are widely used for document summarization, translation, and transformation tasks. In these settings, the input representation is strongly reflected in the output, so there may be room to combine prompt-lookup-style approaches. However, the implementation path differs from that of a decoder-only LLM serving stack.

For encoder-decoder models, the following should be measured:

• Encoder output reuse cost
• How decoder speculative verification is implemented
• Accept rate per translation language pair
• Speed variation by summary length
• The fraction of total latency attributable to encoder cost when input is long

When input is very long and output is short, the perceived benefit of decoder-side acceleration can be limited. Speculative decoding benefits most when many output tokens are generated.

12. Model Characteristics 6: MoE Models

MoE models activate only a subset of experts per token. Sparse MoE models like Mixtral do not use all parameters on every step, which gives them different inference characteristics from dense models.

When applying speculative decoding to MoE models, routing becomes an additional variable. Which expert paths the draft candidates take through the target model, and how cache and expert execution are managed across accepts and rejects, are both important.

A small dense draft model can be paired with a large MoE target model. The draft generates candidates quickly, but it needs to match the target's expert routing and distribution well enough. If the target's expert selection and logits distribution for a given token do not align with the draft, the accept rate drops.

Inter-GPU communication also matters in MoE models. When expert parallelism is in use, experts can be spread across multiple devices. Even if speculative decoding reduces the number of target forward passes, complex communication patterns during candidate verification can erode the benefit.

The metrics to watch for MoE models are:

• Accept rate
• Expert load balance
• Inter-GPU communication overhead
• Throughput by batch size
• Cache memory usage
• Rollback cost after rejection

Speculative decoding is feasible in MoE models, but there are more system-level variables to manage than with dense decoder-only models.

Figure 3. The meaning of accept rate and operational variables differs depending on model type.

13. What to Look at When Applying Speculative Decoding in vLLM

The vLLM speculative decoding documentation covers several related options including draft models and n-gram proposers. The key point from a vLLM perspective is that you cannot just flip a switch — you need to look at speculative decoding in conjunction with the serving scheduler, KV cache, and batch processing.

A serving engine like vLLM handles multiple requests concurrently. Because speculative decoding generates and verifies multiple candidate tokens within a single request, it affects how batches are composed and scheduled.

A common pattern is seeing speed gains in single-request benchmarks but reduced benefit in production. The reason is that when multiple requests are mixed together, GPU utilization, memory bandwidth, cache allocation, and the distribution of request lengths all interact.

Items to verify in vLLM:

• Can the draft model fit on the GPU alongside the target model?
• Are the tokenizers of the target and draft models compatible?
• Does the n-gram proposer deliver enough benefit for the target workload?
• How does speculative overhead change as batch size grows?
• Does first token latency degrade in streaming responses?
• Does KV cache memory run short under long-context requests?

In practice, "speculative decoding is enabled" is not the end of the story. You need benchmarks that match your actual request distribution — and those benchmarks need to separate different cases: short conversations, long document summarization, code generation, JSON output, and so on.

14. What to Look at When Applying Speculative Decoding in Hugging Face Transformers

The Hugging Face assisted generation post describes the flow where a small assistant model proposes candidate tokens and a larger model verifies them. In the Transformers library, speculative decoding variants appear as assisted generation and prompt lookup decoding.

The Hugging Face ecosystem is well-suited for experimentation. You can quickly swap different model combinations and measure accept rate and latency. However, you should not assume that these results will carry over directly to production serving.

Items to verify in local experiments:

• Tokenizer compatibility between the target and assistant models
• How latency changes with different assistant model sizes
• How accept rate changes with the number of draft tokens
• The effect of different temperature and top-p settings
• The effect of varying prompt length and output length
• GPU memory usage

A combination that performs well in Hugging Face may produce different results when moved to vLLM, TensorRT-LLM, or a custom serving stack. Each serving engine has its own batching, caching, kernel, and memory allocation strategy.

15. How to Understand Configuration Values

The most commonly tuned parameter in speculative decoding is the number of draft tokens — how many candidate tokens to propose in a single round.

Increasing this value creates the possibility of confirming more tokens per target forward pass. But it also increases the chance that the draft is wrong. If a rejection occurs at an early candidate, all later candidates in that round are wasted.

A larger draft token count is therefore not unconditionally better. It can be beneficial when the accept rate is high, but causes more waste when the accept rate is low.

The key configuration parameters can be understood as follows:

• Draft token count: How many candidates to generate in one round.
• Draft model size: The trade-off between candidate generation cost and candidate quality.
• Temperature: Changes sampling diversity and directly affects accept rate.
• top-p, top-k: Change the breadth of the candidate distribution and influence how well draft and target align.
• Batch size: Changes both GPU efficiency and scheduling overhead.
• Max sequence length: Affects KV cache memory usage and long-context efficiency.

In practice, change only one of these at a time. Changing multiple values simultaneously makes it impossible to isolate which factor drove the change in speed or quality.

16. Practical Evaluation Metrics

Speculative decoding cannot be judged by tokens per second alone. Even if the number of target model forward passes decreases, first-token latency may increase, tail latency may worsen, or GPU memory may become a bottleneck.

The core metrics to track are:

• accept rate: fraction of draft candidates accepted by the target
• accepted tokens per step: average number of tokens confirmed per target verification pass
• output tokens per second: throughput measured against final output
• end-to-end latency: total time from request start to response completion
• time to first token: time until the first token appears in a streaming response
• p50, p95, p99 latency: tail latency that averages tend to hide
• GPU utilization: how effectively compute resources are actually being used
• memory usage: memory consumed by the target, draft, and KV cache
• quality metric: whether task-level output quality has changed

Quality evaluation must be defined per task. Summarization requires checking factuality and coverage; code generation requires verifying executability; JSON output requires validating schema conformance.

Even though the accept/reject procedure theoretically preserves the target distribution, real systems can have results affected by implementation details, sampling configuration, stop conditions, tokenizer handling, floating-point differences, and the serving scheduler. Final evaluation must therefore be conducted under conditions that closely match the actual deployment environment.

Figure 4. Speculative decoding must be evaluated across accept rate, latency, memory, and output quality together.

17. When to Adopt Speculative Decoding

Speculative decoding is most worth adopting under the following conditions:

• The target model is large, making single-token generation expensive.
• Output sequences are long.
• The draft aligns well with the target, yielding a high accept rate.
• The workload has repetitive patterns — code generation, summarization, document-grounded answering.
• GPU memory and the serving stack can absorb the added complexity.
• Per-request-type benchmarks and regression tests are in place.

Services that generate large volumes of long outputs are particularly likely to benefit, since a single target model pass can confirm multiple tokens at once.

Conversely, be cautious in the following situations:

• Outputs are very short.
• The target model is small.
• There is not enough memory to load a draft model.
• Requests are diverse and sampling temperature is high.
• Format constraints are strict — tool calls, JSON, safety policies — but the draft does not predict them reliably.
• The operations team cannot continuously monitor accept rate and latency.

Speculative decoding is not a universal acceleration switch. It is powerful when the conditions are right, but in the wrong conditions it only adds complexity.

18. Recommended Adoption Sequence

There is no need to jump straight to architectures like Medusa or EAGLE. In practice, a simple, incremental approach is safer.

Step 1: Measure the current serving baseline. Before enabling speculative decoding, establish latency, throughput, GPU utilization, and memory usage numbers.

Step 2: Segment requests by type. Mixing general chat, long summarization, code generation, JSON output, and multilingual requests into a single average obscures the real picture.

Step 3: Start with the lowest-complexity method. For workloads with heavy input repetition, try a prompt lookup or n-gram proposer first. If the target model is large and a compatible smaller model exists, experiment with the draft model approach.

Step 4: Make accept rate the primary metric. A low accept rate means the structural benefit of speculative decoding is small. Adjust draft model size, model family, tokenizer, and sampling settings accordingly.

Step 5: Run production-like benchmarks. Results that look good in single-request experiments can differ significantly under real batch and streaming conditions.

Step 6: Consider Medusa or EAGLE only when model internals can be modified. This step is closer to per-model optimization than to simply toggling an option.

19. Selection Criteria at a Glance

The key point of this table is not absolute ranking. Speculative decoding delivers gains only when the model, data, and serving environment align. Method selection is therefore less about raw performance numbers and more about matching conditions.

20. Summary

Applying speculative decoding is not as simple as "attach a small model." Research has progressed toward making drafts cheaper to generate, better aligned with the target, and less operationally complex.

The small draft model approach is the most fundamental structure. N-gram and prompt lookup exploit repetitive patterns without any separate model. Medusa attaches multiple token-prediction heads inside the target model. EAGLE generates drafts at the feature level to reduce token-level uncertainty.

Differences across model types are significant. Decoder-only LLMs are the most natural fit. Code LLMs can benefit from their repetitive structure. Chat models require careful attention to instruction alignment and sampling configuration. Multilingual models need per-language analysis of tokenization and style differences. Encoder-decoder and MoE models require additional consideration due to their architectural characteristics.

In practice, the critical work is not enabling a feature — it is measuring accept rate, latency, throughput, memory, and output quality broken down by request type. Speculative decoding is a technique that researchers have built up to reduce the bottleneck of autoregressive generation, and in production systems it becomes a powerful acceleration method when the right conditions are met.