KV Cache Isolation in Multi-LoRA Serving: Why Adapter Swaps Break Caching

Prefix Caching is enabled and vLLM is running, yet the hit-rate monitor shows 0%. If the serving stack uses Multi-LoRA, this is almost always the reason.

Why Multi-LoRA Serving Is Structurally Different

LoRA works by adding low-rank matrices A and B to the pretrained weight W. The Q, K, V projections in each attention layer become (W + AB)x, and since A and B differ per adapter, the same input token x produces different K and V values depending on which adapter is active.

The KV cache stores those K and V matrices. The moment the adapter changes, previously cached KVs no longer match what the new adapter would compute, so they cannot be reused. This breaks the core assumption behind Prefix Caching.

When serving a single base model, the cache key is determined solely by the token sequence. Two requests that share the same system prompt reuse the KV computed for that prefix. In Multi-LoRA, even an identical 1024-token system prompt produces different KVs for adapter A versus adapter B, so each adapter needs its own cache entry.

How the Prefix Caching Key Space Fragments by Adapter ID

Both vLLM and SGLang (arXiv:2312.07104) use a Radix Tree as the core of Prefix Caching. A cache block key is structured as follows:

block_hash = hash(parent_block_hash, token_ids, extra_keys)
extra_keys  = (lora_name, mm_hash, cache_salt, ...)

vLLM's kv_cache_utils.py inserts the LoRA name into extra_keys via this function:

def _gen_lora_extra_hash_keys(request: Request) -> list[str]:
    if not request.lora_request:
        return []
    return [request.lora_request.lora_name]

The consequence is straightforward. Two requests with different adapter names produce different block_hash values even when the token sequences are identical. If adapters A, B, C, and D each receive the same system prompt for the first time, that triggers four cache misses and creates four independent entries.

Under a uniform traffic distribution across N adapters, the theoretical upper bound on cache hit probability is 1/N. With 16 adapters sharing traffic equally, the Prefix Caching hit-rate ceiling is roughly 6.25%.

This 1/N limit holds even under ideal conditions — infinite cache capacity and high prompt reuse. The "cache is enabled but hit rate is zero" scenario appears precisely in workloads that approach these ideal conditions.

The True Cost of Adapter Switching

Adapter switching carries two kinds of overhead: the GPU memory load cost, and the throughput loss that comes from batch fragmentation.

Adapter Size and Memory Footprint

The VRAM consumed by a single LoRA adapter can be estimated as:

adapter_size = rank × 2 × hidden_dim × num_layers × dtype_bytes

For Llama-3-8B (hidden_dim=4096, num_layers=32, bfloat16) with a rank-16 adapter:

16 × 2 × 4096 × 32 × 2 = 8,388,608 bytes ≈ 8 MB   (per projection)

Applying LoRA to all four attention projections (q, k, v, o) multiplies that by four, giving roughly 32 MB. Including MLP layers can push it past 50 MB.

There are two main adapter switching strategies.

Full-offload: Only the base model lives on the GPU; each request loads its adapter from CPU to GPU on demand. On a PCIe system, CPU→GPU bandwidth is around 12–16 GB/s, so loading a 32 MB adapter adds roughly 2 ms of overhead. This matches the "millisecond-scale latency for on-demand adapter loading" reported by Punica (arXiv:2310.18547).

Hot-pool: Keep the K most frequently used adapters resident in GPU VRAM. Switching latency is negligible, but the adapter weights directly eat into the KV cache budget. Keeping 16 adapters of 32 MB each in a hot pool consumes 512 MB of KV cache space — a non-trivial fraction even on an A100 80GB.

S-LoRA (arXiv:2311.03285) attempted to mitigate this conflict with Unified Paging — dynamically managing adapter weights and the KV cache from a single memory pool to reduce fragmentation. It reported up to 4× throughput improvement over prior systems in scenarios scheduling thousands of adapters on a single GPU, but it does not address the cache hit-rate drop itself.

Batch Fragmentation

When serving a single base model, tens to hundreds of requests can be packed into one batch to maximize GPU tensor throughput. In a Multi-LoRA setup, requests using different adapters cannot be trivially combined, so the batch must be split by adapter. With traffic spread uniformly across N adapters, effective batch size drops to 1/N, and smaller batches reduce both Tensor Core utilization and memory bandwidth efficiency.

Punica's SGMV (Segmented Gather Matrix-Vector Multiplication) kernel targets this directly. It packs requests using different adapters into a single batch and executes per-adapter matrix operations in parallel across segments. The reported 12× throughput improvement over prior serving systems comes from reducing the batch fragmentation penalty. The cache hit-rate problem is a separate dimension.

When Shared-Prefix Optimization Actually Applies

A common question is whether system prompt KVs can be shared across adapters. The answer depends on the LoRA configuration.

The embedding layer — the first thing a token passes through — is typically excluded from LoRA targets. Its output is identical regardless of which adapter is active. However, KVs are recomputed at every Transformer layer, and once LoRA is attached to layer 1, every subsequent layer's KVs become adapter-dependent. Standard LoRA configurations (q/k/v or all-attention) apply LoRA starting from layer 1, leaving effectively zero sharable KV prefix.

With a full-rank configuration that applies LoRA to every layer, the sharable prefix is exactly zero: KVs diverge immediately after the embedding.

A niche configuration that restricts LoRA to layers before the attention mechanism — with frozen embeddings — can share some layer KVs. This configuration significantly limits fine-tuning expressiveness and is rarely used in general-purpose serving.

How Hit Rate Varies with Adapter Count and Request Distribution

Comparing a uniform distribution against a Zipf distribution (s=1.0, commonly observed skew in production traffic) yields the following. Hit rates are theoretical upper bounds assuming the cache has enough capacity to hold the shared prefix for all adapters.

The hit-rate recovery under Zipf occurs because the top one or two adapters capture more than half of all traffic. At N=4, s=1.0, using the harmonic series (H₄ ≈ 2.08), the top adapter alone handles roughly 48% of requests. Its cache fills quickly and yields a high hit rate.

The problem is the tail adapters with minimal traffic. They occupy cache slots while receiving requests too infrequently to stay warm, so their caches remain perpetually cold. The same traffic skew that boosts overall hit rate inevitably drives up tail latency for the lower-ranked adapters.

Framework Status and When to Split Instances

vLLM includes lora_name in the cache key to prevent cross-adapter cache pollution. However, as noted in issue #30931, there is a bug where stale KVs can be reused when two requests share the same lora_name but the adapter weights have changed at runtime (hot-swap). The combination of Prefix Caching and Multi-LoRA is not fully safe today. SGLang is also moving toward including adapter IDs in the Radix Tree key, but the isolation implementation is still in progress.

The decision between a single Multi-LoRA instance and per-adapter dedicated instances roughly comes down to this crossover point:

Per-adapter QPS ≥ ~60% of peak GPU throughput  → dedicated instances
Many adapters, low per-adapter traffic          → shared Multi-LoRA instance

Under strict SLAs (p99 TTFT < 500 ms), dedicated instances become the first option to evaluate once the adapter count exceeds four. When the cache hit rate falls below 25%, Prefix Caching's TTFT benefit is nearly gone, and adapter-switching overhead becomes the dominant latency contributor.