A100 GPU running LLM decode shows SM (Streaming Multiprocessor) utilization hovering around 10–20%. Hardware rated at 312 TFLOPS appears to be using less than 5% of its compute capacity. Many engineers look at this number and conclude that "something isn't optimized," but that intuition is wrong. The actual bottleneck during the decode phase isn't compute — it's memory bandwidth.
Without understanding this structural reason, you'll end up spending time on optimizations that don't help. That's exactly why Arithmetic Intensity and the Roofline Model matter here.
Arithmetic Intensity: The Single Ratio That Determines the Bottleneck
Arithmetic Intensity (AI) is the ratio of floating-point operations performed per byte of data read from memory.
AI = FLOPs / bytes_moved
A high value means the kernel does a lot of computation per byte read (compute-bound); a low value means memory reads take longer than the computation (memory-bound).
The Roofline Model, introduced by Williams et al. (2009), captures both limits in a single graph. The x-axis is AI (FLOPs/Byte), the y-axis is achievable performance (TFLOPS), and the point where the two bounds meet is the Ridge Point.
Achievable performance = min(AI × BW_peak, FLOPs_peak)
For the NVIDIA A100 80GB:
- Peak BF16 Tensor Core: 312 TFLOPS
- HBM2e Memory Bandwidth: 2,039 GB/s (≈ 2 TB/s)
- Ridge Point: 312 TFLOPS ÷ 2 TB/s = 156 FLOPs/Byte
Only operations with AI ≥ 156 can saturate the Tensor Cores. Below that, memory bandwidth is the first thing to hit the ceiling.
Applying the Same Formula to Prefill and Decode
Let's apply this directly to the two phases of LLM inference. We'll use a single Linear layer from Llama-3 8B with hidden size 4096 (4096→4096, BF16).
Decode — batch size 1, generating 1 token
This is a matrix–vector operation.
FLOPs = 2 × 1 × 4096 × 4096 ≈ 33.5 MFLOP
Bytes read = 4096 × 4096 × 2 ≈ 33.5 MB (weight matrix, BF16)
AI = 33.5M / 33.5M = 1 FLOPs/Byte
That's 1/156th of the ridge point.
Prefill — sequence length 512
The input grows to (512, 4096), making this a matrix–matrix operation.
FLOPs = 2 × 512 × 4096 × 4096 ≈ 17.2 GFLOP
Bytes read = 4096 × 4096 × 2 ≈ 33.5 MB (same weight matrix)
AI = 17.2G / 33.5M ≈ 512 FLOPs/Byte
We're reading the same weight matrix, yet AI reaches 512 — well past the ridge point of 156, firmly in compute-bound territory.
The reason prefill has high AI is straightforward: 512 tokens share the same weight matrix read. The 33.5 MB of weights is read once, but 512× the computation is performed on top of it. In decode, those same weights are read for just one token, so there's no reuse. Even though the matrix size and the bytes read are identical, the amount of computation on top scales with batch size — so AI rises linearly with batch size.
Where the Two Phases Sit on the Roofline Graph
Plotting these two points on the Roofline graph makes the difference immediately clear.
- Decode (B=1): AI ≈ 1, far to the left of the ridge point. Achievable performance = 1 FLOPs/Byte × 2 TB/s = 2 TFLOPS — 0.6% of peak.
- Prefill (S=512): AI ≈ 512, to the right of the ridge point. Achievable performance ≈ 40–50% of peak.
In terms of MFU (Model FLOPs Utilization), on the same model and the same A100, prefill (S=512) reaches 40–50% MFU, while decode (B=1) sits at 1–3%.
The key point: low MFU during decode is not inefficiency. Decode is structurally bandwidth-bound, so the right utilization metric is MBU (Memory Bandwidth Utilization). Measuring memory throughput during decode actually yields 1.6–1.8 TB/s — 80–90% of the A100's theoretical maximum of 2 TB/s. SM utilization is 15% while memory is 85% saturated. This is the normal operating state for decode; the observation that "SMs are idle" is a misreading of where the bottleneck lies.
Why Batch Size Changes Arithmetic Intensity
Running decode with batch size B makes the operation approach a matrix–matrix multiply, and AI scales proportionally with B.
| Batch size | AI (FLOPs/Byte) | Status |
|---|---|---|
| 1 | ~1 | memory-bound |
| 16 | ~16 | memory-bound |
| 64 | ~64 | memory-bound |
| 256 | ~156 | approaching ridge point |
| 512 | ~312 | entering compute-bound |
You need B=256 just to reach the ridge point. Below that, doubling the batch nearly doubles throughput, because bandwidth is the bottleneck and the speed at which weights are read determines processing speed.
This is exactly why LLM serving schedulers, including Sarathi-Serve, continuously try to pack requests into the decode batch. The goal is to raise effective batch size and push AI higher — not to "process more" in a vague sense, but to physically shift the hardware utilization limit.
Which Axis Each Optimization Targets on the Roofline
There are two ways to improve the bottleneck on the Roofline: shift the ridge point left (reduce bytes needed), or push the operation's AI right (increase batch size, increase reuse). Here's how the major optimizations map to each axis:
| Technique | Axis targeted | Mechanism | Most effective when |
|---|---|---|---|
| W4 quantization | Reduce bandwidth pressure | Half the weight bytes → 2× AI | Small batch (B=1–16) |
| Continuous batching | Directly increase AI | Higher B → proportionally higher AI | When throughput is the target |
| Speculative decoding | Increase effective AI | Verify N tokens → AI×N per forward pass | Draft acceptance rate ≥ 80% |
| Tensor Parallel (N GPUs) | Distribute bandwidth | Each GPU handles 1/N of the load | When latency SLO is tight |
These techniques target different axes. With decode at B=1 for a single request, W4 quantization raises AI from 1 to 2 — still 78× short of the ridge point. In that situation, scaling to 4-way Tensor Parallel reduces the bandwidth load per GPU but doesn't raise AI itself. Latency drops, but throughput doesn't change much.
Conversely, once you're already at B=256 or higher and push the batch further, you enter compute-bound territory and the gains from larger batches saturate. At that point, quantization and Tensor Parallel play a different role.
Applying techniques without diagnosis leads to exactly this kind of wasted effort — or, in the case of speculative decoding, draft overhead that exceeds the acceptance rate benefit, actually increasing latency.
Measuring It Directly: nvidia-smi dmon
The fastest way to see what's actually happening during decode is nvidia-smi dmon.
# Monitor SM utilization and memory controller busy ratio at 500ms intervals
nvidia-smi dmon -s um -d 500
Example output during a decode window:
# gpu sm mem enc dec jpg ofa
0 15 88 0 0 0 0
0 14 91 0 0 0 0
0 16 87 0 0 0 0
sm is SM utilization (%), mem is memory controller busy ratio (%). mem at 88 means 88% of memory bandwidth is in use; sm at 15 means only 15% of SMs are active.
Reading both numbers together immediately reveals the bottleneck. High mem with low sm → memory-bound. High sm with low mem → compute-bound. The former is expected during decode.
Nsight Systems lets you break this down per kernel — memory throughput (GB/s) and SM occupancy on a timeline — making it easy to directly compare the characteristics of prefill kernels vs. decode kernels. You'll see prefill kernels saturating the SMs while decode kernels saturate memory bandwidth, side by side on the same timeline.
Classify Your Workload First
Before choosing an optimization technique, determine whether your workload is prefill-heavy or decode-heavy.
Prefill-heavy workloads have long inputs and short outputs (document summarization, classification). The compute-bound phase dominates, so compute optimizations like Flash Attention and kernel fusion are effective.
Decode-heavy workloads have short inputs and long outputs (code generation, chat). The memory-bound phase is overwhelmingly dominant. The right directions are increasing batch size (continuous batching), reducing weight size (quantization), or processing multiple tokens per forward pass (speculative decoding).
If increasing batch size isn't an option — real-time response required, requests arriving sparsely — W4 quantization is the most direct lever. Raising AI from 1 to 2 at B=1 nearly doubles throughput, because the same computation completes using half the bandwidth.
If you can sustain B=64 or higher, speculative decoding is worth evaluating first. Effective throughput varies significantly with the draft model's acceptance rate, but above 80% acceptance it's one of the rare techniques that improves both latency and throughput simultaneously.