nvidia-smi shows 80% utilization, yet you're hitting OOM. Intuitively, 20% should still be free — but allocation fails. This is an inevitable consequence of how the CUDA memory allocator is designed.
PagedAttention is often credited with solving this problem entirely. It does effectively eliminate external fragmentation. But internal fragmentation, the reserved/allocated gap in the CUDA allocator, and misunderstandings around gpu_memory_utilization continue to eat into memory in real serving deployments.
Two Levels of Fragmentation
In LLM serving, memory fragmentation occurs independently in two distinct contexts.
External fragmentation is when total available memory is sufficient, but no contiguous free block exists to satisfy an allocation. Systems before PagedAttention allocated KV cache as per-request contiguous slabs, making this a serious problem. Kwon et al. 2023 measured memory waste of 60–80% under that scheme. PagedAttention eliminates this constraint by partitioning the KV cache into fixed-size blocks distributed across non-contiguous physical memory. Any free block can be assigned to any request, so contiguity is no longer required.
Internal fragmentation is waste within an allocated block. If the last block of a request isn't fully filled, the remaining slots cannot be used by any other request. PagedAttention didn't eliminate this — it reduced it. Previous systems pre-reserved up to the maximum sequence length (e.g., 2,048 tokens) from the start, wasting most of it when actual generation was shorter. Now the waste is confined to the last block — at most block_size - 1 slots. Depending on workload characteristics, this can still be significant.
How the CUDA Allocator Hides Fragmentation
PyTorch's CUDA caching allocator doesn't return memory to CUDA immediately after freeing, in order to amortize the cost of cudaMalloc calls. When a tensor is deleted, it disappears from the Python side but stays in PyTorch's internal cache. This is reserved memory.
x = torch.randn(1024, 1024, device='cuda') # ~4MB allocated
del x # freed in Python — not returned to CUDA
print(torch.cuda.memory_allocated()) # 0
print(torch.cuda.memory_reserved()) # ~4MB or more, held in cache
The PyTorch CUDA caching allocator satisfies allocation requests by finding a sufficiently large cached block; if none exists, it calls cudaMalloc. As a model creates and frees tensors of varying sizes, the cache accumulates fragments of different sizes. Even if the total is large, the absence of a contiguous block of the right size triggers a cudaMalloc retry — and if that fails too, you get OOM.
The number nvidia-smi reports includes this reserved memory. Three metrics each measure something different:
| Metric | What it measures |
|---|---|
nvidia-smi | Total GPU memory the process has acquired from the OS (includes cuBLAS, NCCL, etc.) |
torch.cuda.memory_reserved() | Total memory held by the PyTorch caching allocator (includes cache, excludes other libraries) |
torch.cuda.memory_allocated() | Memory actually occupied by live tensors |
If you see nvidia-smi at 78 GB, reserved at 70 GB, and allocated at 45 GB during serving, then 25 GB is trapped in the cache — some of it scattered as fragments that cannot satisfy new allocations. The wider the gap between these three values, the worse the fragmentation.
What gpu_memory_utilization Actually Does
gpu_memory_utilization=0.9 does not mean "use 90% of total GPU memory for the KV cache." It's the fraction of memory remaining after weights and CUDA overhead that gets allocated to the KV cache. vLLM's actual sequence is:
- Load model weights onto the GPU
- Run a forward pass on dummy input to profile peak memory usage
- Use
total_gpu_memory × gpu_memory_utilizationas the upper bound for KV cache allocation - Assign
(total_gpu_memory × gpu_memory_utilization) − (weights + CUDA graphs + activations)to the KV cache
Working through an example with Llama-3-8B (FP16), A100 80 GB, gpu_memory_utilization=0.9, block_size=16:
target = 80GB × 0.9 = 72GB
weights ≈ 16GB (8B params × 2 bytes, FP16)
overhead ≈ 2GB (CUDA graphs, activations, NCCL, etc.)
KV cache available = 72 − 16 − 2 = 54GB
Llama-3-8B has 32 layers, GQA with 8 KV heads, and head_dim=128. Memory per block:
bytes_per_block
= 2(K+V) × block_size × num_layers × num_kv_heads × head_dim × dtype_bytes
= 2 × 16 × 32 × 8 × 128 × 2
= 2,097,152 bytes ≈ 2MB
Number of available blocks:
num_blocks = 54 × 1024 MB ÷ 2 MB ≈ 27,648
max_tokens = 27,648 × 16 = 442,368 tokens
You can verify this directly in the vLLM startup log:
INFO ... worker.py] # GPU blocks: 27648, # CPU blocks: 0
To back-calculate the maximum number of concurrent sequences from this number: num_blocks ÷ ceil(max_seq_len / block_size). With max_seq_len=2,048, that's 27,648 ÷ 128 = 216 as a theoretical upper bound (actual numbers vary by scheduler policy).
Setting gpu_memory_utilization too high (0.97+) will cause an OOM during the profiling step. Too low, and the KV cache is undersized, hurting throughput. 0.85–0.92 is a safe starting range for most environments.
block_size and Internal Fragmentation
The last block of each request is never fully filled. Approximating expected waste (assuming uniform distribution of sequence lengths):
expected_waste ≈ (block_size − 1) / (2 × avg_seq_len + block_size − 1)
Theoretical waste rates by avg_seq_len and block_size:
| avg_seq_len | block_size 8 | block_size 16 | block_size 32 | block_size 64 |
|---|---|---|---|---|
| 32 tokens | ~10% | ~19% | ~33% | ~50% |
| 64 tokens | ~5% | ~11% | ~20% | ~33% |
| 128 tokens | ~3% | ~6% | ~11% | ~20% |
| 512 tokens | <1% | ~1.4% | ~3% | ~6% |
Large block sizes are expensive when the workload is dominated by short sequences. With avg_seq_len=32 and block_size=64, half the KV cache slots are wasted. This is why vLLM defaults to block_size=16.
For a single request, consider the worst case: with block_size=32, a sequence of length 33 occupies 2 blocks (64 slots) but fills only 33, wasting 48%. This worst case is not rare in API serving environments where short sequences are common.
Estimating Available KV Cache Blocks
To estimate block count before serving:
def estimate_kv_blocks(
total_gpu_gb: float,
gpu_memory_utilization: float,
model_weights_gb: float,
overhead_gb: float,
block_size: int,
num_layers: int,
num_kv_heads: int,
head_dim: int,
dtype_bytes: int = 2, # FP16
) -> int:
available_gb = (
total_gpu_gb * gpu_memory_utilization
- model_weights_gb
- overhead_gb
)
bytes_per_block = (
2 * block_size * num_layers * num_kv_heads * head_dim * dtype_bytes
)
return int(available_gb * (1024 ** 3) // bytes_per_block)
# Llama-3-8B, A100 80GB
n = estimate_kv_blocks(
total_gpu_gb=80,
gpu_memory_utilization=0.9,
model_weights_gb=16,
overhead_gb=2,
block_size=16,
num_layers=32,
num_kv_heads=8,
head_dim=128,
)
print(f"Estimated blocks: {n}") # ≈ 27,648
print(f"Max cached tokens: {n * 16:,}") # ≈ 442,368
If the actual block count in the vLLM log differs from this estimate by more than a few thousand blocks, the likely causes are: CUDA graph capture using more memory than expected, larger-than-expected NCCL initialization, or another process holding GPU memory. Comparing torch.cuda.memory_reserved() before and after the profiling step gives you the actual overhead.
How Fragmentation Shows Up in Tail Latency
Under memory pressure, the vLLM scheduler either swaps KV cache blocks for running requests to CPU, or evicts requests that haven't finished prefill and recomputes them later. This path leaves average latency untouched while causing tail latency to spike.
Under normal conditions, TTFT (Time to First Token) is stable. When an eviction happens under memory pressure, the affected request must re-run prefill from scratch, so TTFT jumps suddenly. If you see p50 and p95 looking healthy while p99 spikes, suspect this path.
vLLM exposes num_preempted as a Prometheus metric. If this counter is trending upward and you're simultaneously observing p99 spikes, KV cache fragmentation is actively degrading serving quality.
Operational Levers
Tune block_size: Measure your workload's avg_seq_len first, then set block_size to roughly half that value to keep internal fragmentation below 10%. block_size is fixed at server startup, so changing it requires a restart.
Cap max_num_seqs: Reducing the number of concurrent requests lowers KV cache pressure and decreases eviction frequency. This is effective when you want to prioritize latency stability over throughput.
Separate prefill and decode: Running prefill and decode on separate instances prevents the two phases from competing for the same KV cache space. Rather than reducing fragmentation directly, this isolates the sources of cache pressure.
The right lever depends on your workload's sequence length distribution. An avg_seq_len=32 chat API and an avg_seq_len=512 document summarizer have completely different optimal configurations. Before tuning anything, observe TTFT, num_preempted, and torch.cuda.memory_allocated() together in production to understand the actual pressure patterns.