Speculative Decoding Variants Compared: EAGLE, Medusa, and Lookahead

Standard Speculative Decoding has a simple structure: a small draft model proposes several tokens, and a large target model verifies them all in a single forward pass. Because the verification cost is nearly constant regardless of draft length, throughput scales sub-linearly as long as the acceptance rate is high enough.

The challenge lies in how you position the draft model. Running a separate model requires additional memory, and because that model lives in a different representation space than the target model, acceptance rates come out lower than expected. Medusa, EAGLE, and Lookahead each attack a different vertex of this triangle—draft model memory, inference overhead, and acceptance rate. Medusa eliminates inference overhead; EAGLE closes the context mismatch gap; Lookahead eliminates the draft model memory entirely. Because they target different bottlenecks, which one to choose also depends on your environment.

Medusa: Eliminate Draft Overhead by Adding Multiple Heads

Medusa's architecture is straightforward. It attaches k additional heads on top of the target model's LM head, each independently predicting the k-th next token. A single forward pass yields up to k+1 candidate tokens, and since there is no separate model forward pass for drafting, the inference overhead is effectively zero. The extra parameters are limited to the heads themselves—on an A100 80GB, this amounts to roughly 0.2–0.5 GB.

Verification is done via Tree Attention. The token combinations produced by the heads are organized into a tree, and the target model verifies the entire tree in parallel, selecting the longest accepted chain. The Medusa paper reports that Medusa-1, which freezes the backbone, achieves over 2.2x speedup, while Medusa-2, which jointly fine-tunes the backbone, achieves 2.3x–3.6x.

The story changes at larger batch sizes. Each head predicts independently, looking only at the current token's hidden state. When sequences within a batch have different contexts, per-head acceptance rates vary across sequences. At batch size 1, the tree structure works in your favor, but as the batch grows, sequences with low acceptance rates become a bottleneck for the entire tree verification step. Adding more heads widens the tree, and in batches with uneven acceptance rates, overhead can actually increase. This is why Medusa's benchmarks focus on single-request (batch=1) scenarios.

Medusa is the easiest method to attach to a custom fine-tuned model—just add the heads without any draft model training pipeline. However, acceptance rates vary significantly across models and domains, and in batch-serving environments, the gains can be smaller than expected.

EAGLE: Reuse the Target Model's Hidden States as Draft Input

The fundamental reason standard Speculative Decoding suffers low acceptance rates is context mismatch. The draft model predicts from a different representation space than the target model, so even when the direction is right, the target model rejects the tokens.

EAGLE addresses this gap by feeding the target model's last-layer hidden states directly as input to the draft model. Since the draft model receives what the target model is "seeing" as input, its predictions align much more closely with the target model's distribution. On LLaMA2-Chat, the acceptance rate α is 0.75–0.77, with an average of 3.62–3.90 tokens accepted per verification pass. This acceptance rate is what makes EAGLE 1.47x–1.60x faster than Medusa and 1.7x–2.1x faster than Lookahead.

EAGLE-2 adds an adaptive draft tree that dynamically adjusts draft length based on context, rather than using a fixed value. EAGLE-1 uses a fixed number of draft steps, which wastes compute when acceptance rates are low by generating drafts of the same length regardless. EAGLE-2 detects these low-acceptance regions and cuts the draft short. This yields an additional 20%–40% speedup over EAGLE-1—3.29x on LLaMA3-Instruct 70B and 4.26x on Vicuna 13B—with accepted tokens per cycle (τ) rising from 3.94 in EAGLE-1 to 4.98.

EAGLE-3 goes a step further, training the draft head on a mixture of hidden states from multiple intermediate layers rather than just the last layer. On chat-domain benchmarks, the acceptance rate improves 8–14% over EAGLE-2. As of vLLM 0.8+, EAGLE-3 is the currently recommended method.

This architecture does carry a memory cost. Even with hidden state sharing, the draft model's KV cache is separate from the target model's. For a 70B target, the draft model occupies an additional ~1.5–3 GB on an A100 80GB. If a pre-trained EAGLE/EAGLE-2/EAGLE-3 checkpoint exists for your model, you can use it immediately; attaching it to a custom fine-tuned model requires a separate training run of 1–2 days on 4×A100 (40G).

Lookahead Decoding: Reduce Draft Model Memory to Zero

Lookahead Decoding eliminates the draft model entirely. Based on Jacobi iteration, it predicts tokens at multiple positions simultaneously, accumulating the n-gram patterns that emerge from this process into a pool. In subsequent decoding steps, n-grams from the pool that match the current context are retrieved and used as draft chains. With no additional model, the only memory overhead is the n-gram buffer.

The key parameters are N (Jacobi iteration window size) and W (n-gram size). Larger N and W allow longer draft chains but increase the computation of the Jacobi iterations themselves, raising TTFT. Finding the optimal point between these two parameters on a single GPU requires experimentation before deployment.

Acceptance rates are heavily dependent on how repetitive the text is in terms of n-grams. On open-ended conversation tasks like MT-bench, the speedup tops out around 1.8x, but on domains with high repetition—such as code completion or document summarization—it can reach 4x in multi-GPU settings. On non-repetitive text, the n-gram pool stays empty and acceptance rates bottom out.

In single-GPU environments with no memory headroom, Lookahead is effectively the only viable option.

Comparing Acceptance Rate, Memory, and Batch Sensitivity

The table below summarizes reported figures from each paper alongside estimates for A100 80GB. These are not directly comparable measurements taken under identical conditions, so treat the batch-sensitivity comparisons as directional guidance.

The reason EAGLE has lower batch sensitivity than Medusa comes down to draft quality. A higher acceptance rate means less variance in accepted chain length across sequences within a batch. Medusa's per-head independent predictions produce high variance between sequences, and as the batch grows, that variance accumulates and erodes throughput gains.

When to Choose Which Method

Consider three scenarios.

Single-request, low-latency serving (chatbot): EAGLE-2 or EAGLE-3 is currently the best choice. Acceptance rates are stable, and at batch=1 the speedup clearly justifies the additional memory cost. The prerequisite is that a pre-trained checkpoint must exist for your model.

High-throughput batch serving (batch API, async processing): The calculus changes. Medusa shines at batch=1 but gains diminish quickly as batch size increases. EAGLE is more stable across batch sizes, but the inherent limits of speculative decoding at large batch sizes apply equally to all methods. In high-batch environments, you should measure acceptance rates empirically before committing to any variant.

Single GPU with no memory headroom: Lookahead is the only practical option. That said, you need to be serving a domain with high n-gram repetition—code generation or document summarization—to see real gains.

If measuring acceptance rates in advance isn't feasible, EAGLE provides the safest floor. Medusa climbs when the domain fits but drops sharply when it doesn't, and Lookahead becomes nearly indistinguishable from baseline on non-repetitive text.

Enabling in vLLM and SGLang

Since vLLM 0.8, the speculative decoding API is unified under --speculative-config. EAGLE-3 and n-gram mode are enabled as follows:

# EAGLE-3 (vLLM 0.8+, recommended)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{
    "method": "eagle3",
    "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-70B",
    "num_speculative_tokens": 5
  }'

# N-gram / Lookahead (no additional model required)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 5
  }'

# Standard Draft Model
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{
    "method": "draft_model",
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "num_speculative_tokens": 5
  }'

In the Python API, pass the same dictionary as LLM(..., speculative_config={...}). When aligning the draft model's tensor parallel size with the target, specify draft_tensor_parallel_size explicitly.

Medusa is not currently exposed as a named method in vLLM; SGLang supports it separately. num_speculative_tokens sets the upper bound on draft chain length—with the adaptive draft length in EAGLE-2/3, the actual length is reduced dynamically, so there is less waste than with a fixed step count. A reasonable starting point is 5; measure acceptance rates in production and tune from there.

An open question is the rise of MTP (Multi-Token Prediction). As more models like DeepSeek-V3 internalize MTP at training time, drafts can come from the target model itself without a separate draft checkpoint. How far this will displace EAGLE's position remains to be seen.