vLLM Preemption: When to Swap KV Cache vs. Recompute

LLM serving frequently covers KV cache eviction policy — which cache entries to discard. What gets less attention is the next step: once you've evicted, how do you recover when that request resumes?

When the scheduler preempts a running request, the recovery path depends on where the KV blocks go. Offloading them to CPU DRAM and reading them back later is Swap. Discarding them entirely and re-prefilling from scratch when the request resumes is Recompute. vLLM V1 defaults to Recompute. Throwing away already-computed results looks wasteful, but the actual cost structure makes this choice reasonable.

What the Scheduler Faces Under Memory Pressure

When KV cache space runs short, the scheduler has three options.

The first is waiting: stop accepting new requests and let the current batch drain naturally. This sacrifices GPU throughput but keeps the implementation simple and leaves existing KV caches untouched. The second is Swap — move the preempted sequence's KV blocks to CPU DRAM and read them back when the sequence is rescheduled. The third is Recompute — discard the blocks outright and re-prefill the sequence from scratch on resumption.

Intuition favors Swap when choosing between the two. "Recomputing is wasteful" feels right. But that intuition underestimates the cost of CPU↔GPU data transfer.

The Real Cost of Swap

Swap cost is entirely transfer time, and the transfer volume is determined by KV cache size.

KV_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × dtype_bytes

The factor of 2 accounts for storing both the K and V matrices. For Llama-3 8B — num_layers = 32, num_kv_heads = 8 (GQA), head_dim = 128, BF16 = 2 bytes — the per-token cost is 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KB.

Scaling by sequence length:

Two additional points matter here. First, actual PCIe transfers carry a fixed kernel-launch overhead on top of the raw transfer time. The smaller the data, the larger this overhead becomes relative to total transfer time — making the real numbers significantly worse than the table suggests. This is why the vLLM PagedAttention paper shows Swap efficiency collapsing for small block sizes: when transferring hundreds of small blocks, per-transfer fixed overhead becomes the bottleneck, not PCIe bandwidth. Second, preemption triggers a write (GPU→CPU) and resumption triggers a read (CPU→GPU) — each occurring separately. Round-trip transfer time for a single 4K-token sequence comes to 34 ms on PCIe 4.0.

Why Recompute Is Competitive

Recompute is just a short prefill. The preempted sequence is treated as a new prompt and run through attention and MLP from the beginning.

How fast this is depends heavily on sequence length. From a roofline perspective, the arithmetic intensity (FLOPs/byte) of a prefill kernel decreases as sequence length shrinks. The ridge point on an A100 SXM is roughly 156 FLOPs/byte for BF16; short-sequence prefills fall below this threshold and become HBM-bandwidth-bound. That means compute units are waiting on memory transfers, and actual wall-clock time is much shorter than peak FLOPs would suggest.

Batching is also a key variable. In high-throughput serving, sequences being recomputed join the next prefill batch and are processed alongside new requests. The marginal cost of adding one such sequence to the batch is far smaller than processing it in isolation. Resuming via Swap is the opposite — the CPU→GPU transfer is a serial cost that consumes PCIe bandwidth exclusively for that one sequence.

The microbenchmarks in Kwon et al., 2023 Figure 19 support this. Recompute overhead is bounded at roughly 20% slower than Swap in the worst case. At small block sizes, Recompute wins by a wide margin; at large block sizes, the two methods converge but never reverse. In other words, Recompute is at most 20% slower than Swap in the worst case and faster in most cases.

Conditions Where Swap Can Win

Swap has a meaningful advantage in two situations.

The first is when PCIe bandwidth is wide and block sizes are large enough that transfer overhead is small. For long-context workloads — document summarization or code generation at 8K+ tokens on PCIe 4.0 — a 34 ms one-way transfer can be faster than a large-scale prefill recomputation. This is especially true once attention FLOPs scale as the square of sequence length; recompute cost grows quickly for long sequences.

The second is when the same request is preempted multiple times. Recompute re-runs the full prefill on every resumption. Swap preserves the KV data in CPU memory from the first eviction onward. If a request is preempted three times, Recompute pays for three full prefills of the same sequence; Swap pays for one write and one read. In environments where requests queue for long periods under sustained load, this difference accumulates.

vLLM's Implementation and Default Choice

vLLM switches between modes via the --preemption-mode argument. Swap was used more aggressively in V0, but the V1 architecture made Recompute the default.

The reasoning behind V1's choice is tied to its architectural changes. V1 separates the Worker and Scheduler into distinct processes. Using Swap requires coordinating block movement over IPC, and that overhead offsets Swap's theoretical advantage. Recompute simply re-queues the sequence for prefill — no cross-process coordination, and the implementation stays straightforward.

To use Swap mode, --swap-space must also be specified. This space is allocated as pinned CPU memory, so setting it large noticeably increases host memory usage.

vllm serve meta-llama/Llama-3-8B-Instruct \
  --preemption-mode swap \
  --swap-space 8    # GB 단위, 기본값 4GB

SGLang takes a different approach: it minimizes preemption frequency by tuning batch size, and falls back to Recompute when preemption does occur. Both frameworks have converged on Recompute as the default recovery strategy, and the reasoning is the same in both cases — PCIe transfer overhead and cost amortization across a batch.

What to Watch in Production

vllm:num_preemptions_total, exposed via Prometheus, is the first signal for this issue. If the preemption rate relative to total requests stays within a few percent, the default configuration incurs no meaningful loss. Once it exceeds 10%, batch configuration needs attention.

Intervention order:

1. Reduce --max-num-seqs — lowering the number of concurrently processed sequences suppresses preemption at the source.
2. Increase --gpu-memory-utilization (default 0.9 → 0.95) — more space allocated to KV cache means fewer preemptions. Verify OOM headroom before increasing.
3. Quantization (AWQ / GPTQ) — reducing weights to 4-bit frees up space for KV cache allocation.
4. Add GPUs / Tensor Parallelism — distributing the model gives each GPU more KV cache capacity.

Switching to --preemption-mode swap when preemption frequency exceeds 20% is rarely the right fix. It does nothing to reduce preemption frequency itself, and depending on the environment, recovery cost may actually increase. Expanding KV cache capacity or reducing concurrency will produce more meaningful improvements first.