Why Prefix Caching Hit Rates Disappoint in Production

Prefix Caching is widely reported to significantly reduce TTFT. In production, however, the improvement is often 10–20% at best, or barely noticeable at all. The culprit is a low cache hit rate—but the reasons why it stays low are not straightforward. Block alignment, eviction policy, request distribution, and prompt design each erode hit rate in different ways.

Block-based Hashing (vLLM) vs. Radix Tree (SGLang): Why Hit Rate Diverges on the Same Requests

The premise of Prefix Caching is simple: if the prefix of a new request matches a previous one, reuse the already-computed KV cache and skip the Attention computation. The question is how "match" is defined—and vLLM and SGLang have taken completely different approaches.

vLLM uses block-level hashing. It groups block_size tokens (default 16) into a single hash key and manages the entire KV cache at that block granularity. When a prefix aligns exactly to a block boundary, there is no problem. But if the prefix length is not a multiple of 16, the last partial block is not cached.

# Alignment loss with vLLM block_size=16
prefix_len = 100  # actual prefix token count
block_size = 16

effective_hit_tokens = (prefix_len // block_size) * block_size  # = 96
alignment_loss       = prefix_len % block_size                  # = 4

A 100-token prefix loses 4 tokens. At that length the loss is 4%—negligible. But the shorter the prefix, the worse the ratio. A 20-token prefix suffers the same 4-token loss, which is 20%. vLLM issue #40696 reports cases where a prefix shorter than block_size invalidates the cache entirely—a direct consequence of this design.

SGLang uses RadixAttention. All KV caches across all in-flight requests are managed in a single Radix Tree (Trie). When a new request arrives, the tree is traversed to find the longest common prefix path; computation begins from that point forward. Matching is token-level, so there is no block-alignment constraint. Whether the prefix is 20 tokens or 100, whatever matches is reused exactly.

The hit rate gap between the two implementations varies considerably with the request distribution. On traffic with short prefixes and a flat distribution, vLLM's alignment loss is especially pronounced, and the hit rate gap can reach 20–30 percentage points. The SGLang NeurIPS 2024 paper benchmarks show up to 6.4× throughput improvement over vLLM on high-prefix-reuse workloads—RAG and multi-turn dialogue—where token-level matching is a core contributor to the difference. On workloads with no prefix reuse at all, the performance difference between the two implementations is noise-level; RadixAttention is designed to incur no overhead on cache misses.

Three Structural Causes of Low Hit Rate

Beyond the implementation differences, three additional factors erode hit rate in real traffic.

The Long-Tail Effect of Request Distribution

Prefix Caching works well when many requests share the same prefix. A chatbot with a fixed 1,024-token system prompt and only the user input varying meets that condition. But in services where each request has a different prefix—RAG pipelines that inject raw search results into the context, or document summarization pipelines—the number of unique prefixes (U) is large, and the U/C ratio relative to cache capacity (C) is high. Once U/C exceeds 1, LRU eviction becomes frequent and a cycle begins where entries are evicted almost immediately after being inserted.

Given cache capacity C, total requests N, and unique prefix count U: if U/C < 1, all prefixes fit in the cache and the theoretical hit rate ceiling is (N - U) / N. If U/C > 1, LRU eviction kicks in and hit rate drops sharply.

Injecting Variables in the Middle of a System Prompt

Where a variable is inserted determines hit rate. Comparing two structures against the same 1,024-token system prompt makes the difference clear.

When a variable is inserted mid-prompt, all tokens after that point differ between requests. Because vLLM hashes the prefix sequentially, blocks after the injection point carry completely different hashes on every request, forcing cache misses. In a 1,024-token system prompt with a variable at position 500, the trailing 524 tokens never hit the cache.

This is a prompt design problem, not a model problem. Pushing variables to the suffix frequently doubles the number of cacheable tokens in practice.

Batch Contention and Shortened Cache Lifetime

Increasing batch size to improve throughput simultaneously reduces the benefit of Prefix Caching—a reverse trade-off. More concurrent requests intensify KV cache memory contention and make LRU eviction far more frequent. If batch size is B and each request has an average prefix of L tokens, a single step requires B×L tokens of cache space. As that value approaches total cache capacity, earlier requests' cache entries are already evicted within the same batch.

You can observe this trade-off directly by watching hit rate fall as max_num_seqs is increased in vLLM.

Eviction Policy: When LRU Becomes the Worst Choice

vLLM's default eviction policy is LRU (Least Recently Used). It seems intuitive, but it degrades badly under certain traffic patterns.

LRU preserves recently accessed entries. When the request distribution follows a power law—where a small number of popular prefixes attract most of the traffic—LRU works well: popular prefixes are accessed repeatedly and stay in cache.

The problem is high request diversity. When requests are spread evenly across prefixes, every new request evicts an older one. In this regime, LFU (Least Frequently Used) outperforms LRU—even if a prefix is old, keeping it is worth more if it has been accessed frequently. The simulation below demonstrates this directly.

from collections import OrderedDict, Counter
import random

def simulate_hit_rate(policy, capacity, requests):
    cache = OrderedDict()
    freq  = Counter()
    hits  = 0

    for req in requests:
        freq[req] += 1
        if req in cache:
            hits += 1
            if policy == 'lru':
                cache.move_to_end(req)
        else:
            if len(cache) >= capacity:
                if policy == 'lru':
                    cache.popitem(last=False)
                elif policy == 'lfu':
                    min_key = min(cache, key=lambda k: freq[k])
                    del cache[min_key]
            cache[req] = True

    return hits / len(requests)

capacity   = 20
n_requests = 10_000

# Power-law distribution (traffic concentrated on popular prefixes)
power_law_reqs = [int(random.paretovariate(1.5)) % 100 for _ in range(n_requests)]
print("Power-law distribution:")
print(f"  LRU hit rate: {simulate_hit_rate('lru', capacity, power_law_reqs):.2%}")
print(f"  LFU hit rate: {simulate_hit_rate('lfu', capacity, power_law_reqs):.2%}")

# Uniform distribution (high request diversity)
uniform_reqs = [random.randint(0, 99) for _ in range(n_requests)]
print("Uniform distribution:")
print(f"  LRU hit rate: {simulate_hit_rate('lru', capacity, uniform_reqs):.2%}")
print(f"  LFU hit rate: {simulate_hit_rate('lfu', capacity, uniform_reqs):.2%}")

Running this shows that LRU and LFU perform similarly on power-law distributions. On uniform distributions, LFU is noticeably better. The code also directly confirms that once U exceeds 2×C, LRU performance degrades sharply.

LFU has its own drawback: it retains prefixes that were heavily used in the past but are no longer active—it has no temporal locality. LRFU or TTL-based eviction policies are superior in many environments, but vLLM v1 only provides LRU.

How to Measure Hit Rate Correctly

vLLM's /metrics endpoint exposes prefix cache metrics as two Prometheus counters. The previously used gpu_prefix_cache_hit_rate_perc gauge is deprecated.

# Prometheus counters (current recommended approach)
vllm:prefix_cache_queries   # total prefix cache lookups
vllm:prefix_cache_hits      # cache hits

# Hit rate over a 5-minute window
rate(vllm:prefix_cache_hits[5m]) / rate(vllm:prefix_cache_queries[5m])

# Overall KV cache utilization (0–1)
vllm:kv_cache_usage_perc

There is a common misconception here. prefix_cache_hits is aggregated in terms of cached KV blocks—it is closer to a token-level hit rate. If a request has a 1,024-token prefix and 512 of those tokens are cached, that request counts as a 50% hit.

This is why the metric can be misleading when predicting TTFT reduction. For TTFT to drop meaningfully, request-level hit rate—"was this request's entire prefix cached?"—must be high. When only short prefixes are cached, the token-level hit rate can look acceptable while requests with long prefixes still recompute most of their prefix. Prefix length distribution must be examined alongside the hit rate to avoid this trap.

Interpreting both metrics together (vllm:kv_cache_usage_perc and hit rate):

How to Improve Hit Rate at the Serving Layer

Once the causes are understood, there are three structural levers to pull.

Prompt design. Push variables to the suffix. Dynamic values in the system prompt—user IDs, session context, search results—should be appended after the fixed prefix. This alone frequently doubles the number of cacheable tokens in practice.

Request routing. Routing requests that share the same prefix to the same server instance is essential—this is prefix-aware routing. Ray Serve's prefix-aware routing is designed for this. Applying consistent hashing keyed on the prefix hash ensures requests with the same prefix land on the same instance. Without this in a multi-instance deployment, each instance must populate its own cache independently, diluting hit rate inversely with the number of instances.

Cache capacity. Increasing capacity raises hit rate, but with diminishing returns. Once U/C drops below 1—all unique prefixes fit in the cache—additional capacity provides almost no benefit. Conversely, when U/C > 2, doubling capacity produces a large jump in hit rate. If both kv_cache_usage_perc and hit rate are high, the problem is routing or prompt design, not capacity.

Traffic Conditions Where Prefix Caching Actually Works

Four variables determine whether it is worth enabling.

When these four conditions are not met, the metrics will show it first: hit rate pinned low, or kv_cache_usage_perc high while hit rate is low. If hit rate is falling short of expectations, check which of these four variables is off before diving into the numbers.