When Speculative Decoding Hurts Throughput: The Acceptance-Batch-Memory Tradeoff

Speculative Decoding Conditions That Kill Throughput: The Acceptance Rate–Batch Size–Memory Triangle

Benchmarks show a 2× speedup, but measuring on an actual serving server shows little to no difference — or even a slowdown. Any team that has deployed Speculative Decoding has run into this. The problem doesn't stem from a single variable — acceptance rate, batch size, and KV cache pressure interact to determine the final speedup.

The Draft-Verify Loop: Just the Essentials

The draft model generates γ tokens sequentially. The target model verifies all γ tokens in a single forward pass — where autoregressive decoding would have required γ separate forward passes. Accepted tokens are committed, and drafting restarts from the first rejection point.

The reason this is fast: one forward pass of the target model can process up to γ+1 tokens. The key word is up to. How many are actually accepted depends on the acceptance rate.

Acceptance Rate Varies Drastically by Task

Acceptance rate (α) is the primary variable that determines speedup. Even with the same model and the same draft length, α can vary widely across tasks.

Acceptance rate distribution by task type, using vLLM + EAGLE-2:

Code completion scores high because of fixed syntactic patterns. It's much easier for the draft model to predict what follows return. Math reasoning allows multiple paths at intermediate computation steps, and the draft model doesn't always follow the target model's chain of reasoning.

From the speedup formula in Leviathan et al., the expected number of accepted tokens per speculative step is:

E[accepted] = (1 - α^(γ+1)) / (1 - α)

With α=0.88 and γ=4, E≈4.4. With α=0.55 and γ=4, E≈2.0. A difference of 0.33 in acceptance rate more than doubles the expected token count. When acceptance rate is low, γ=4 draft steps become pure overhead — the cost of running the draft model γ times isn't recovered from a single target model forward pass.

Why Larger Batch Sizes Erase These Gains

The fact that Speculative Decoding's speedup is highest at batch=1 and decreases as batch size grows is a fundamental property of the LLM decode stage.

At small batch sizes, decoding is memory-bound. The GPU's compute units are idle, and the bottleneck is reading weights from HBM. In this regime, the cost difference between "process 1 token" and "process γ+1 tokens" in a verify pass is nearly zero — weight loading is the bottleneck, and computing attention for γ+1 tokens runs on top of the same already-loaded weights with no additional memory transfers. This is when speedup is maximized.

As batch size grows, arithmetic intensity (compute / memory movement) increases and execution enters the compute-bound regime. Now the verify pass actually costs more for γ+1 tokens versus 1. Each request occupies a long GPU cycle of "generate γ draft tokens + 1 verify pass," which degrades the overall throughput of the full batch.

Approximate speedup by batch size, with draft length γ=4 and acceptance rate α=0.75:

Sub-1× at batch=32 is the point where draft model execution cost plus verify overhead outweigh plain AR decoding. This means using the same GPU time for standard AR decoding would have processed more tokens.

A paper analyzing the interaction between batching and speculative decoding measured a 1.93× speedup at batch=4, γ=3. The consistent pattern is that this number falls as batch size increases. This is why most benchmarks measure at batch=1 or with a small fixed number of requests — those conditions are the most favorable for Speculative Decoding. In real serving environments with mixed traffic, batch sizes grow dynamically, creating a gap with benchmark numbers.

KV Cache Pressure from the Draft Stage

Memory footprint varies structurally depending on the draft approach.

Independent draft model (e.g., Llama-68M as a draft for Llama-70B): the draft model occupies its own dedicated KV cache. Llama-68M is small — roughly 0.27 GB for model weights at fp16 — but the KV cache for the entire batch is managed separately. When the draft model's KV cache eats into HBM on the same GPU as the 70B model, the number of requests that can be handled concurrently (max concurrent requests) drops. The two models share the GPU's weights and activation memory.

EAGLE / EAGLE-2: a lightweight draft head takes the target model's hidden states as input. There is no separate draft model KV cache. Instead, speculative tokens generated during the draft phase must be appended to the target model's KV cache, so the KV sequence length grows by γ before and after verification. The draft head itself is roughly 0.5–0.6 GB in parameters — far smaller than an independent draft model. As reported in the EAGLE-2 paper, it achieves 3.05×–4.26× speedup while minimizing KV memory overhead.

Medusa: multiple decoding heads are added to the target model to predict tokens at positions +1, +2, … in parallel. Tree attention evaluates multiple candidate paths simultaneously. The KV cache structure is fully shared with the target model — there is no separate draft KV cache. However, KV compute in tree attention scales with the number of candidate paths.

All three approaches split on one axis: does the draft method require its own KV cache or not? The independent draft model approach is straightforward, but having to manage two models' KV caches on the same HBM puts it at a disadvantage at large batch sizes.

Conflicts with the Continuous Batching Scheduler

From the perspective of a continuous batching scheduler like vLLM, Speculative Decoding looks like an unusual request.

In standard AR decoding, each request has "one step = one token = a short, uniform GPU occupancy." The scheduler can easily insert new requests or adjust priorities.

With Speculative Decoding enabled, a single request occupies a longer cycle of "generate γ draft tokens + 1 verify pass." Other requests wait during this cycle. This is especially problematic in mixed scenarios where only some requests in a batch use speculative decoding — long-cycle requests break scheduler fairness and increase TTFT (Time to First Token) for other requests. Even if overall throughput looks acceptable, the latency distribution degrades.

If preemption occurs mid-draft: vLLM commits only the accepted tokens and discards any draft tokens that haven't been verified yet. The compute spent generating γ draft tokens is wasted. This is why Speculative Decoding's real-world gains drop sharply in environments with heavy memory pressure or frequent request priority changes.

EAGLE vs. Medusa vs. Lookahead: Trade-offs

EAGLE-2 delivers the highest speedup ratio, but that number comes from MT-bench conditions: temperature=0 and small batch sizes. In production, as batches grow and tasks mix, the actual speedup falls.

Lookahead has lower acceptance rates but requires no additional models or training. It's worth trying first when you want to run a quick experiment.

When Not to Use It

If any of the following apply, reconsider before adopting Speculative Decoding:

• Serving environment with a sustained batch size of 8 or more — standard AR decoding may have better throughput
• Expected acceptance rate below 0.6 for your primary tasks (math reasoning, Korean free-form generation, etc.)
• Less than 20% HBM headroom — without space to load the draft model KV cache or EAGLE head, max batch size shrinks and you take a net throughput loss

The batch size condition is the most commonly overlooked. The typical failure mode: benchmark at batch=1, deploy, then see no improvement or a regression under real traffic.

When experimenting in vLLM, the most reliable approach is to directly measure tokens/s across varying batch sizes:

# independent draft model approach
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-chat-hf \
  --speculative-model facebook/opt-125m \
  --num-speculative-tokens 4 \
  --gpu-memory-utilization 0.85

# compare tokens/s per batch size with --num-speculative-tokens 4 vs 8

Increasing --num-speculative-tokens from 4 to 8 improves throughput for high-acceptance-rate tasks (code completion) but increases draft overhead and hurts throughput for low-acceptance-rate tasks (math reasoning). The numbers are only meaningful if you measure with a task mix that matches your actual traffic.