GPU Memory Bandwidth Sets the Ceiling on LLM Decode Batch Size

max_num_seqs를 512로 올려도 aggregate throughput은 거의 늘지 않는데 TPOT p99는 계속 치솟는 구간이 있다. 컨텍스트 길이 2048 토큰, Llama-3-8B 기준으로 그 구간은 배치 크기 60부터 시작하며, 이 숫자는 GPU 메모리 대역폭·모델 파라미터 바이트·컨텍스트 길이 세 값으로 정확히 예측된다.

Why Decode Arithmetic Intensity Is Bounded by Batch Size

The decisive difference between prefill and decode lies in Arithmetic Intensity (AI: FLOP/byte).

In prefill, AI scales proportionally with sequence length S. Attention FLOPs are O(S²·d) while memory reads stay at O(S·d), which is exactly why prefill transitions to compute-bound at sufficiently long sequences.

In decode, each of the B requests in a batch reads its own KV cache independently. Breaking down a single decode step for one layer:

• Weight matrix operations: FLOPs ≈ 24Bd², memory ≈ 24d² bytes (weights are shared across the batch)
• KV cache attention: FLOPs ≈ 4BSD, memory ≈ 4BSD bytes (independent reads per request)

(d: hidden dim, S: average context length, B: batch size, BF16 assumed)

Overall AI = (24Bd² + 4BSD) / (24d² + 4BSD)

As B grows, the KV cache terms in numerator and denominator scale at the same rate. AI is pinned between two extremes:

• Short context, small B: AI ≈ B (weight reads dominate)
• Long context: AI ≈ 1 (KV cache reads dominate, independent of B)

The roofline ridge point on an A100 is roughly 156 FLOP/byte (312 TFLOPs ÷ 2 TB/s). At context length 512 tokens and batch size 32, measured AI for Llama-3-405B is 41.56 and for DeepSeekV3 is 89.83 — both well below the ridge point. Decode never reaches fully compute-bound territory at any batch size.

FlashAttention-2, in its analysis of attention kernel memory access patterns, notes that scattered KV cache reads during the decode phase destroy cache locality — a fundamentally different access pattern from prefill's consolidated access.

Computing the Upper Bound from GPU Memory Bandwidth

The bytes a GPU must read per decode step are:

total read bytes = model_bytes + B × kv_bytes_per_seq

model_bytes      = num_params × 2  (BF16)
kv_bytes_per_seq = num_layers × 2 × num_kv_heads × head_dim × 2 × ctx_len
                   (K and V separately, BF16)

Theoretical aggregate throughput is B × GPU_BW / total_read_bytes. At small B, model_bytes dominates the denominator, so throughput scales roughly linearly with B. As B grows, the B × kv_bytes_per_seq term takes over and the growth rate bends. The point where the two terms are equal is B_knee:

B_knee = model_bytes / kv_bytes_per_seq

For Llama-3-8B (32 layers, num_kv_heads=8 GQA, head_dim=128, BF16 16GB) and Llama-3-70B (80 layers, num_kv_heads=8 GQA, head_dim=128, BF16 140GB):

The 70B model has a much higher B_knee than 8B. With GQA keeping KV head count the same (8 heads), the model parameters are 8.75× larger but the KV cache is only 2.5× larger. B_knee is identical on H100 SXM (3.35 TB/s) and A100 (2 TB/s). Changing bandwidth does not shift the crossover point between the two terms. The H100 reduces TPOT by ~40% compared to A100, but B_knee sits at the same location.

Per-Batch TPOT and Throughput: Theoretical Values

For Llama-3-8B served on a single A100 (2 TB/s), roofline-model-based theoretical values (within 10–20% of measured results) are as follows:

At S=512, throughput keeps climbing through B=256 with only gradual TPOT degradation. B_knee is 238, so the limit hasn't been hit yet.

At S=2048, TPOT goes from 12.3 → 16.6 ms (35% increase) and tok/s from 2,602 → 3,857 (48% increase) in the B=32→64 range. There's still a good reason to increase batch size here. In the B=64→128 range, TPOT climbs another 51% while tok/s gains only 32%. This is where marginal efficiency inverts.

S=8192 is the most dramatic case. At B=8, TPOT is already 1.45× that of B=1, and at B=32 it's 2.95×. With B_knee at 15, a batch of 32 is already twice past the knee. Scaling the batch 8× from B=32 to B=256 yields only a 38% improvement in tok/s (1,270 → 1,759).

The main sources of error between theoretical and measured values are memory fragmentation (PagedAttention's block-based allocation), scheduling overhead, and actual HBM access efficiency of the attention kernel.

How Serving Engines Handle This

vLLM's --max-num-seqs is the direct upper bound on decode batch size. Continuous batching inserts new prefills into existing decode iterations while keeping the decode batch below this limit. The problem is that --max-num-seqs defaults to 256, while B_knee for an 8B model at 2048-token context is 60 — running with the default means operating at more than 4× the knee.

TensorRT-LLM puts a hard cap on total KV cache tokens via KvCacheConfig(max_tokens=N). Because this limits total KV cache size rather than batch count directly, bandwidth saturation stays consistent across varying context lengths. When used together with kv_cache_free_gpu_mem_fraction (default 0.9), the smaller of the two conditions determines the actual allocation.

SGLang's RadixAttention shares KV cache across requests with identical prefixes. In deployments where all requests share the same prefix — such as a system prompt — effective kv_bytes_per_seq shrinks and B_knee rises. In deployments with more than 60% prefix sharing, effective batch headroom can nearly double in practice.

Sarathi-Serve (OSDI 2024) uses chunked prefill to mix prefill chunks and decode steps within the same iteration. Stall-free scheduling means new requests can be admitted without pausing the decode batch, which dampens the TPOT spikes that occur when batch size suddenly jumps. On a single A100 with Mistral-7B, it reports 2.6× throughput improvement over vLLM.

The three engines take different approaches, but they're solving the same problem: reduce the total bytes the GPU must read per decode step, or expose a parameter that controls that total.

Decision Criteria for Production

B_knee can be estimated from just three values:

# B_knee estimation
model_bytes      = num_params * 2                           # BF16
kv_bytes_per_seq = num_layers * 2 * num_kv_heads * head_dim * 2 * avg_ctx_len
B_knee           = model_bytes / kv_bytes_per_seq

# Recommended max_num_seqs range: 1.0–1.5× B_knee

Llama-3-8B, average context 2048 tokens: B_knee ≈ 60 → --max-num-seqs 64.
Llama-3-8B, average context 8192 tokens: B_knee ≈ 15 → --max-num-seqs 16.

The diagnostic signal is TPOT p99. If TTFT (Time To First Token) p99 violates the SLO first, the bottleneck is prefill. If TPOT p99 starts spiking first, the decode batch has exceeded B_knee. When nvidia-smi dmon shows memory bandwidth utilization stuck above 85%, reducing batch size brings TPOT back down immediately.

One common misconception: the intuition that a smaller 8B model should handle larger batches than a 70B. As long as GQA keeps KV head count the same, a smaller model has a higher kv_bytes_per_seq to model_bytes ratio, which means a lower B_knee. Llama-3-8B's B_knee at S=2048 is 60; the 70B's is 208.