Prefill vs Decode Bottlenecks: Roofline Analysis of LLM Inference

Prefill and Decode run on the same model and the same GPU, but they saturate different resources. Prefill squeezes compute; Decode chews through memory bandwidth. Understanding that difference intuitively versus quantitatively leads to different design decisions.

Differences in Computational Structure

Prefill processes the entire input token sequence at once. Looking at a single linear layer: a weight matrix W ∈ R^{d_out × d_in} is multiplied against an input matrix X ∈ R^{BS × d_in} for B sequences of S tokens — a matrix-matrix multiply (GEMM). The GPU's Tensor Cores can maximize parallelism on large matrix multiplications, and all SMs (Streaming Multiprocessors) share the work.

Decode reads the already-generated KV cache and produces one next token. At each step, each sequence processes exactly one new token, so the input shrinks to X ∈ R^{B × d_in} and the multiply against the weight matrix collapses into a matrix-vector multiply (GEMV). GEMM and GEMV are both matrix multiplications, but the ratio of GPU resources they actually use is completely different.

Arithmetic Intensity: Quantifying the Gap Between the Two Phases

Williams et al.'s Roofline paper defines Arithmetic Intensity (AI) simply as:

AI = FLOPs / Memory_Traffic  (units: FLOPs/Byte)

It's the ratio of how many floating-point operations are performed per byte pulled from memory. Higher AI means compute is the bottleneck; lower AI means memory bandwidth is.

Let's compute AI for a single linear layer. The weight matrix W is stored in FP16, so the load cost is 2 × d_in × d_out bytes. With T tokens being processed:

FLOPs  = 2 × T × d_in × d_out
Memory = 2 × d_in × d_out  Bytes  (weight load)

AI = FLOPs / Memory = T  (FLOPs/Byte)

The d_in × d_out terms cancel out. AI equals the number of tokens processed, T. Since T = B × S in Prefill and T = B × 1 in Decode, the ratio of their AIs scales exactly with sequence length S.

Concretely, for the Query projection in LLaMA-3 8B (d_in = d_out = 4096):

Both phases pay the same cost to load the same weight matrix (32 MB), but Prefill extracts 32,768× more compute from that load. It's all about how much work you get out of each weight read.

Roofline Model: Plotting Both Phases Against GPU Specs

The Roofline Model plots AI on the x-axis and achievable performance (FLOPS) on the y-axis, with two "roofs": the GPU's peak compute (horizontal line) and memory bandwidth × AI (diagonal line sloping up to the right). Actual performance cannot exceed whichever roof is lower.

The point where the two roofs meet is the Ridge Point:

Ridge Point = Peak FLOPs / Memory Bandwidth

Specs and Ridge Points for the A100 SXM 80GB and H100 SXM 80GB:

Prefill (B=64, S=512) with AI=32,768 exceeds the Ridge Point by tens of times on both GPUs — it sits in the compute-bound region on the far right of the Roofline graph, under the horizontal roof. Decode (B=1) with AI=1 sits at 1/156 of A100's Ridge Point and 1/295 of H100's.

The H100's higher Ridge Point is actually a disadvantage for Decode. Because the compute increase (3.17×) outpaces the memory bandwidth increase (1.68×), low-AI workloads like Decode see relatively lower utilization per dollar on H100. The expectation that "buying an H100 will make serving faster" is hard to meet for Decode latency (TPOT) — you can't expect more than the 1.68× bandwidth increase.

Can Scaling Batch Size Push Decode into Compute-Bound Territory?

Since AI = B for Decode, scaling B should theoretically push past the Ridge Point. The threshold batch size is straightforward:

B_threshold ≈ Ridge Point

A100: B_threshold ≈ 156
H100: B_threshold ≈ 295

The problem is the KV cache. LLaMA-3 8B uses GQA (8 KV heads), so the KV cache per token is:

KV/token = 2(K,V) × 8(KV heads) × 128(d_head) × 32(layers) × 2(FP16)
         = 131,072 Bytes  ≈  128 KB/token

On an A100 80GB, the model weights (FP16) occupy 16 GB, leaving 64 GB of HBM. Serving B=156 requests with a 4096-token context each requires:

156 × 4096 × 128 KB ≈ 83.9 GB  →  OOM

Even cutting context to half (2048 tokens) costs roughly 41.9 GB — and once you add activations, scheduler overhead, and fragmentation headroom, the practically achievable batch size is far smaller.

HBM runs out before you reach the compute-bound threshold. This is one reason throughput doesn't scale linearly with batch size. Once KV cache contention kicks in, Decode stays memory-bound while being constrained by an entirely different bottleneck.

This Asymmetry Is the Common Root of Serving Design Decisions

Disaggregated Prefill, Chunked Prefill, and Speculative Decoding look like they solve different problems, but all of them start from the same AI asymmetry.

Disaggregated Prefill physically separates the AI asymmetry at the GPU level. Prefill (compute-bound) is routed to a compute-dense GPU pool; Decode (memory-bound) goes to a high-bandwidth GPU pool, with KV caches transferred between them via P2P. This AI asymmetry is precisely why vLLM separates P-nodes and D-nodes — mixing both workloads on the same GPU means you can't optimize either one.

Chunked Prefill splits long Prefill sequences into smaller chunks and interleaves them with Decode batches. For a chunk of size C, Prefill AI ≈ C (per sequence). On A100, C=256 exceeds the Ridge Point (156) and enters compute-bound territory; on H100, the Ridge Point is ≈295, so C=256 is still memory-bound — you need C=512 to reliably land in the compute-bound region. This is why the optimal chunk size for Chunked Prefill needs to be tuned per GPU.

Speculative Decoding has a draft model generate k candidate tokens at once, which the target model verifies in parallel. The target model's verification step bundles what would be multiple B=1 GEMVs into a length-k sequence, turning them into a GEMM closer to Prefill — a direct attempt to raise AI. With k=4, AI jumps 4×, but it still sits in Decode territory. This is why a high acceptance rate from the draft model is essential; a low acceptance rate just adds overhead.

All three strategies reduce to one of: "raise Decode's AI," "reduce the proportion of Decode," or "physically separate the two phases by their AI."

Profiling Prefill and Decode Separately in Practice

The fastest way to observe both phases separately during LLaMA-3 8B inference is dcgmi. It outputs SM utilization (field 1002) and DRAM utilization (field 1005) at 100 ms intervals:

# GPU 0 기준, 100ms 간격으로 SM 활용률과 DRAM 활용률 출력
dcgmi dmon -e 1002,1005 -d 100

For a per-phase timeline, combine Nsight Systems with NVTX markers:

nsys profile \
  --trace=cuda,nvtx \
  --gpu-metrics-device=0 \
  --output=llama3_profile \
  python run_inference.py

import nvtx

with nvtx.annotate("prefill", color="blue"):
    output = model.forward(input_ids, use_cache=False)

with nvtx.annotate("decode", color="red"):
    for _ in range(max_new_tokens):
        output = model.forward(next_token, use_cache=True, past_key_values=kv)

The measured numbers align well with theory. Prefill: SM utilization ~80–90%, DRAM BW utilization ~25–35%. Decode: SM utilization ~10–20%, DRAM BW utilization ~85–95%. The two numbers flip between phases. With shorter sequences or smaller batches, SM utilization drops during Prefill, while Decode stays bandwidth-saturated regardless of batch size.

Caveats to the Simplified Model

With short prompts (S < 64), Prefill can also fall below the Ridge Point. At S=32, B=1, AI=32 — about 1/5 of A100's Ridge Point (156). In environments like RAG pipelines or tool calling, where prompts are short and batches are small, Prefill can also be bottlenecked by memory bandwidth.

Flash Attention complicates this analysis. Because Flash Attention tiles the computation within SRAM rather than writing intermediate attention matrices to HBM, it dramatically reduces HBM traffic compared to a naive implementation. The effective AI of the attention layer is higher than the analysis above suggests. For linear layers — Q, K, V, O projections, FFN — weight loading still dominates and the calculations above apply directly. But the attention computation itself is pulled closer to compute-bound territory by Flash Attention.

Keeping these two caveats in mind, you can use "Prefill = compute-bound, Decode = memory-bound" as your default mental model, while knowing to check dcgmi numbers first for short-sequence workloads or architectures with a high proportion of attention computation.