Prefix Caching Deep Dive: Why TTFT Drops but Throughput Doesn't

The idea behind prefix caching — pulling already-computed KV state from a previous request and reusing it for the next — is straightforward. The implementation is not particularly complex either. But once you enable it and look at the metrics, the numbers don't match intuition. TTFT drops, but throughput stays flat. The hit rate reads 80%, yet p99 latency doesn't budge. You expand the cache, and throughput actually falls.

What Gets Stored and What Gets Reused

vLLM's Automatic Prefix Caching partitions the KV cache into blocks of 16 tokens by default and assigns each block a chaining hash (SHA-256 of the parent block's hash plus the current block's token list). When a new request arrives with the same prefix, the hash chain matches and the stored KV state is fetched directly, skipping prefill.

The hit condition has a simple but important constraint: a block must be completely full to be eligible. The final partial block is always a miss. If a system prompt is 2,050 tokens, the first 128 blocks (2,048 tokens) can be hits, but the remaining 2-token block is never shared. Unless the prefix length is an exact multiple of the block size, there is always a boundary loss.

What Determines Hit Rate

Hit rate is almost entirely determined by three workload characteristics. No amount of system tuning can push past this ceiling.

System prompt sharing rate. When every request starts with the same long system prompt, prefill for that segment is skipped from the second request onward. A RAG pipeline with a fixed 6,000-token static context achieves 88–94% prefill savings. General open-ended Q&A, where each request carries a different context, lands at 0–5%.

Prefix length distribution. In multi-turn conversations, the cacheable prefix grows as conversation history accumulates. From the second turn onward, more blocks are reusable. The n4n.ai multi-turn benchmark shows second-turn TTFT dropping from 120 ms to 15 ms on a 2,000-token session. By the third turn and beyond, latency falls to network round-trip territory.

Inter-request interval and eviction. Cache blocks are evicted in LRU order under memory pressure. If requests sharing the same prefix are not arriving in rapid succession, the cache may already be empty by the time they come in. The SGLang v0.4 cache-aware load balancer addresses this at the routing level, pushing hit rate from 20% to 75% and throughput from 82,665 tok/s to 158,596 tok/s.

Expected hit rates and impact by workload type:

76% of Tool & Agent requests share more than 50% of their prefix, because tool definitions are repeatedly prepended to the system prompt.

Why TTFT Drops but Throughput Doesn't

Prefill and decode use the GPU in fundamentally different ways.

Prefill is a large matrix multiplication that processes all input tokens in parallel. GPU utilization runs at 90–95%, with arithmetic intensity around 200–400 ops/byte — it is compute-bound. Decode generates one token per step and must read the entire KV cache from HBM at every step. Tensor cores finish the computation in microseconds, but then sit idle waiting on the next memory read, driving GPU utilization down to 20–40%. Arithmetic intensity is 60–80 ops/byte — it is memory-bandwidth-bound.

Prefix caching skips the prefill stage. That is why TTFT decreases. But in the decode stage, regardless of cache hit status, the full KV state for that request must still be read from HBM, and the number of output tokens to generate does not shrink. In decode-bound serving — long outputs, saturated batches, or large models — prefix caching only affects TTFT. GPU utilization and req/s do not improve.

The Spheron 2026 benchmark illustrates this clearly. On a single H100 SXM5 80GB with Llama 3.3 70B FP8, throughput for unique prompts with no caching is 1,850 tok/s for vLLM and 1,920 tok/s for SGLang — a 4% difference. Under the condition of 80% shared prefix, 512 tokens, and 50 concurrent requests, TTFT p50 and p95 drop substantially, but throughput numbers show little change for either engine.

Conversely, in prefill-bound workloads — document classification, embedding generation, or summarization with very long inputs and short outputs — skipping prefill directly translates to saved GPU cycles, and throughput rises accordingly.

The Problems That Come from Chasing Higher Hit Rates

The amount of cache space is determined by how much GPU memory is allocated to the KV cache pool. Raising --gpu-memory-utilization from 0.85 to 0.90 in vLLM grows the KV cache pool and keeps previously evicted blocks alive longer. The trade-off is that the maximum number of requests that can be processed concurrently — the maximum batch size — decreases within the same GPU memory budget.

This trade-off can work in the wrong direction. Even if the hit rate rises, a smaller batch size means the scheduler runs under-filled batches and GPU utilization actually drops. This is especially pronounced in decode-bound workloads, where large batch sizes are required to keep the GPU sufficiently occupied. Allocating more memory to the cache lowers that ceiling and reduces throughput.

LRU eviction is also not fair. Long system prompt entries with many blocks occupy a large number of blocks at once and are evicted wholesale under memory pressure. Blocks from requests with short prefixes survive longer on an LRU basis. The net effect is that requests using long system prompts repeatedly experience cache misses.

In vLLM, monitor hit rate via the Prometheus metric vllm:prefix_cache_hits / vllm:prefix_cache_queries. If the hit rate rises while throughput starts to fall, check maximum batch size and GPU utilization alongside it.

Implementation Differences: vLLM, SGLang, and TensorRT-LLM

The three engines differ in the granularity at which they evaluate hits. A single difference in block size can produce a wide gap in effective hit rate for the same workload.

vLLM uses 16-token blocks by default, SHA-256 chaining hashes, and counts only full blocks as hits. It is enabled by default in the V1 engine; the legacy V0 requires the --enable-prefix-caching flag.

# vLLM V0 (explicit activation)
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --block-size 16

SGLang's RadixAttention manages the KV cache with a radix tree (compact prefix tree). The default page_size is 1 token, so prefix sharing requires no block-boundary alignment. When a branch occurs mid-prompt, the common prefix before the branch is maximized via node splitting. It is enabled by default and can be disabled with --disable-radix-cache. Adding --schedule-policy lpm enables Longest Prefix Match priority scheduling for even higher hit rates.

# SGLang (RadixAttention enabled by default)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.3-70B-Instruct \
  --mem-fraction-static 0.88 \
  --schedule-policy lpm

TensorRT-LLM uses a default block size of 128 tokens — 8× larger than vLLM. Kernel efficiency is higher, but reusable segments are only recognized at multiples of 128. Host memory offload (--kv_host_cache_bytes) allows blocks evicted from GPU memory to be retained in pinned memory and reused later; this is particularly effective on Grace-Hopper architectures. Allocating large pinned memory on x86 can add more than 10 seconds to initialization time.

# TensorRT-LLM build
trtllm-build --use_paged_context_fmha enable --tokens_per_block 128

# Triton server (enabled by default at runtime)
enable_kv_cache_reuse: "true"

# Host memory offload (recommended for Grace-Hopper)
--kv_host_cache_bytes 45000000000

Under the same workload — 80% shared prefix, 512 tokens, 50 concurrent requests — vLLM TTFT p50 coming in at 310 ms versus SGLang's 195 ms (a 37% gap) is the accumulated effect of block-boundary alignment loss.

How to Read the Metrics

Looking at hit rate alone gives you half the picture. A prefix_cache_hit_rate of 80% means little if the hits are concentrated on short prefixes — actual prefill savings may be small. The metric to pair with it is the effective prefill token reduction rate: the ratio of tokens actually processed in prefill to the tokens that would have been processed without caching.

The TTFT distribution is a mixture of hit and miss requests. A low p50 alongside an unmoved p99 means miss requests are pulling the tail up. If the hit rate is high but TTFT p99 won't move, suspect one of two things: long system prompt entries are being evicted, or a class of requests that will always miss — such as new user sessions — is dominating the tail.

If the hit rate is high, TTFT p50 has dropped, but throughput (req/s) is unchanged, that is a signal that serving is decode-bound.

When to Disable or Constrain It

In environments with low prefix overlap, the cache sits mostly empty while still consuming memory. For B2C services where each request carries a unique user context, short-context batch workloads, or any workload where the hit rate does not consistently exceed 60%, the reserved cache space does nothing but reduce the maximum batch size.

The same applies when memory is already tight — gpu_memory_utilization already at 0.90 or above, or when long outputs produce a large KV cache footprint. In vLLM, use --no-enable-prefix-caching; in SGLang, use --disable-radix-cache or lower --mem-fraction-static to cap the cache pool size.

Measure a baseline before disabling anything. Comparing TTFT p50/p99, throughput, and maximum batch size before and after enabling prefix caching tells you directly which direction the trade-off cuts for your workload.