Why Larger Batches Kill Prefix Caching in LLM Serving

When you first enable --enable-prefix-caching in vLLM, the impact is immediate. For chatbots sharing a system prompt, TTFT drops 40–60%. The natural next move is to push max_num_seqs higher to squeeze out more throughput. That's where things go wrong. As you increase batch size from 32 to 64 to 128, the hit rate graph starts bending downward. What started at 60% falls below 20%, and TTFT actually gets worse.

The very act of increasing throughput kills the cache. Without understanding this feedback loop, you can't explain why batch size tuning consistently behaves counterintuitively.

One Memory Pool, Two Competitors

Prefix caching works simply: KV blocks computed for previous requests stay in GPU memory, and new requests sharing the same prefix reference those blocks without recomputation. Blocks that aren't reused are evicted via LRU policy to make room for new requests.

The structural problem starts here. The KV cache block pool is a single memory space. Blocks occupied by currently running requests and blocks that must survive for reuse compete for the same pool. As the batch grows, in-flight requests claim more blocks, leaving less room for cached blocks to survive eviction. When the pool fills up, the scheduler evicts LRU cache blocks first to accommodate new requests.

Once batch size crosses a threshold, every new request admitted immediately displaces an existing cache block. No matter how frequently a prefix is reused, if there's no space left in the block pool, the block is already gone by the time the next request arrives.

Calculating KV Cache Block Occupancy

Running the numbers for Llama-3-8B makes the scale of this competition concrete.

Spec: 32 layers, 8 GQA KV heads, head dimension 128, FP16. vLLM default block size of 16 tokens.

KV memory per block:

memory per block = num_layers × 2(K,V) × num_kv_heads × head_dim × block_size × dtype_bytes
                 = 32 × 2 × 8 × 128 × 16 × 2 bytes
                 = 2,097,152 bytes ≈ 2MB

If the average request has a 2,048-token context, it requires 2048 / 16 = 128 blocks. In-flight occupancy by batch size:

On an A100 80GB, after subtracting Llama-3-8B weights (FP16, ~16GB) and applying --gpu-memory-utilization 0.9, the KV cache pool gets roughly 56GB. That translates to 56GB / 2MB ≈ 28,000 blocks. At batch size 128, in-flight requests already hold 16,384 blocks, leaving less than half the pool available for cached blocks.

And there's an additional pressure on top of that.

Larger Batches Mean Longer Request Lifetimes

As batch size grows, the scheduler juggles more concurrent requests. The time a single request takes from start to completion — its request lifetime — also increases, because the GPU only advances to the next decode step after every request in the batch finishes the current one.

This creates a compounding effect. Longer request lifetimes mean KV blocks stay pinned in an occupied state longer. Meanwhile, new requests keep arriving while block reclamation is delayed. Together, these drain the free space in the block pool much faster than you'd expect.

At average output of 512 tokens and 2,048-token context, a batch size of 32 on an A100 runs decode steps at roughly 40ms each, giving a request residence time of 512 × 40ms ≈ 20 seconds. At batch size 128, increased KV load and memory pressure push per-step latency higher, stretching residence time to 60–80 seconds — during which 128 new requests continue consuming blocks.

Observed Pattern: The Inflection Point Falls Between Batch 32 and 64

On a chatbot workload with 80% system prompt sharing and --enable-prefix-caching enabled in vLLM, the following pattern emerges across batch sizes:

# Collect prefix cache hit rate via Prometheus metrics
curl -s http://localhost:8000/metrics | grep -E "prefix_cache_hit_rate|e2e_request_latency"

vLLM exposes the vllm:gpu_prefix_cache_hit_rate gauge, so you can track batch size changes alongside hit rate in Prometheus + Grafana.

Throughput peaks near batch size 128. Hit rate collapses between batch 32 and 64. These two points don't coincide. At maximum throughput, TTFT p99 already exceeds 16 seconds and hit rate has fallen to 18%. For any service with a p99 latency SLO, batch size 64 or above is operationally out of the question.

Preemption Causes Discontinuous Hit Rate Collapse

When the KV block pool runs dry, the vLLM scheduler preempts an in-flight request and releases all of its KV blocks at once. This is why hit rate graphs show sudden spikes downward rather than a smooth, gradual decline.

In vLLM logs, this event appears as:

INFO  ... Running 128 requests, 14 sequences in the swap out queue
WARN  ... Sequence XXX is preempted from ... to ...

Sequences accumulating in the swap out queue signal that preemption is happening. Monitoring vllm:gpu_prefix_cache_hit_rate at the same time reveals a near-simultaneous correlation between preemption events and hit rate spikes.

Preemption is a distinct, discontinuous cause, separate from steady memory pressure. When hit rate falls gradually as batch size increases and then suddenly drops sharply at a specific point, preemption has almost always begun at exactly that point. Without monitoring, it's easy to misattribute the hit rate degradation to request diversity (prefix collisions) rather than the actual cause.

A Formula for Computing the Cache-Friendly Batch Size Ceiling

For prefix cache blocks to survive at least one generation — meaning cached blocks aren't evicted while a single cohort of requests is being processed — the following condition must hold:

B_max = (M_free - overhead) / (L_avg × KV_bytes_per_token)

Variable definitions:

• M_free: GPU memory available for the KV cache after subtracting model weights
• overhead: Minimum reserved capacity of evictable blocks you want to preserve in the KV cache pool
• L_avg: Average request context length (tokens)
• KV_bytes_per_token: KV memory per token = num_layers × 2 × num_kv_heads × head_dim × dtype_bytes

Applying A100 80GB, Llama-3-8B, average context 2,048 tokens:

KV_bytes_per_token = 32 × 2 × 8 × 128 × 2 = 131,072 bytes ≈ 128KB/token
M_free             ≈ 56GB  (80GB × 0.9 - 16GB model weights)
overhead           = 9GB   (minimum reservation for prefix cache survival)
B_max              = (56GB - 9GB) / (2,048 × 128KB)
                   = 47GB / 256MB
                   ≈ 47

Push batch size above 47 and every new request evicts an existing cache block. The 9GB overhead figure depends on average system prompt length and how many generations you want to keep in cache. In practice, you control the batch size ceiling directly via vLLM's --max-num-seqs option.

Sarathi-Serve (Agrawal et al., 2024) approaches this from a different angle. By using chunked prefill to split prefill work into small token chunks interleaved between decode steps, it mitigates the burst where a single large prefill exhausts the block pool all at once. The paper reports 2.6× higher serving capacity than vLLM on Mistral-7B. That said, Sarathi-Serve doesn't eliminate the fundamental competition between batch size and cache eviction — the memory pressure is still there. It smooths out the temporal distribution of that pressure to reduce preemption spikes.

When to Sacrifice Batch Size to Preserve the Cache

The right tradeoff depends on workload type.

Comparing batch size 32 vs 64 on a chatbot workload with 80% system prompt sharing: throughput increased 37% (840 → 1,150 tok/s), but TTFT p50 degraded 145% (200ms → 490ms). Cache hit rate dropped from 60% to 31%, nearly eliminating the TTFT savings from prefix reuse. Batch 64 looks better on paper, but from a user-perceived latency standpoint it's a regression.

For workloads like one-off document summarization where every request has a completely different prefix, there's no cache benefit to begin with. Lowering --max-num-seqs to suppress batch size in that case is simply giving up throughput for nothing. If vllm:gpu_prefix_cache_hit_rate consistently stays below 20%, expanding batch size is the more practical choice for that workload than protecting prefix caching.