FP8 serving being faster than BF16 comes with conditions: large batch sizes, long input sequences, and a GPU with FP8 Tensor Cores like the H100. If any of these conditions is missing, the ranking can flip.
In the batch=1 decode regime, W4A16 (AWQ/GPTQ) can actually outperform per-tensor FP8. The intuition that "lower precision is faster" holds for prefill but breaks down for decode. Here's why that reversal happens, starting from the hardware execution path.
Hardware Generation Changes the Premise
FP8 Tensor Cores first appeared with H100 (Hopper). The lowest-precision Tensor Cores on A100 are INT8. Running an FP8 model on A100 gives you no FP8 Tensor Core acceleration path — without Transformer Engine, execution falls back to CUDA cores. Benchmark results from an A100 environment cannot be directly applied to H100, and vice versa.
Per the NVIDIA H100 datasheet, H100 FP8 dense FLOPS is approximately 1,979 TFLOPS vs. 989 TFLOPS for BF16 — exactly a 2× theoretical ratio. For FP8 Tensor Cores to engage, matrix dimensions must be multiples of 16, and input tensors must be in E4M3 (for weights and activations) or E5M2 (for gradients) format. E4M3 has 4 exponent bits and 3 mantissa bits with a representable range of [-448, 448]; E5M2 has 5 exponent bits and 2 mantissa bits with a wider range. In practice, E4M3 is the standard for both weights and activations at serving time.
In compute-heavy regions like prefill, measured performance gets reasonably close to this theoretical ratio. The problem shows up in decode.
Arithmetic Intensity: Why Decode Can't Take Advantage of FP8
LLM inference splits into prefill and decode. Prefill processes the entire input in a single matrix operation, so every token in the batch participates in one GEMM. When matrices are large, GPU FLOPS become the actual bottleneck — the workload is compute-bound.
Decode is different. Tokens are generated one per step, and with small batches, the data volume per matrix operation drops dramatically. Arithmetic intensity is FLOP divided by bytes moved; in batch=1 BF16 decode it falls to around 1 FLOP/byte. The threshold at which H100 shifts into compute-bound territory is roughly 295 FLOP/byte for BF16 — batch=1 decode sits at just 0.3% of that threshold.
When memory bandwidth is already the bottleneck, doubling FLOPS via FP8 does nothing for throughput. This is where W4A16 wins. Storing weights at 4 bits reduces the bytes that need to be read from memory to one-quarter of BF16. Since bandwidth is the bottleneck, cutting weight transfer volume directly translates to proportionally higher throughput.
How Granularity Cuts Into Decode Throughput
Even where W4A16 has an advantage in decode, granularity is a variable.
AWQ- and GPTQ-based W4A16 typically uses per-group quantization (group size=128). Each group stores a separate FP16 scale and zero-point, and a dequantization pass runs before the FP16 MMA. Per-tensor FP8 uses a single scale for the entire tensor. The dequantization cost difference between the two is significant.
Per-group W4A16 incurs additional memory reads and computation proportional to the number of groups. In a 70B model with group size=128, each weight tensor requires loading hundreds of separate scale vectors. This overhead becomes pronounced in decode, which is already memory-bound — per-group scales and zero-points add more memory traffic on top of that. At low concurrency (batch=1~4), per-group W4A16 can end up slower than per-tensor FP8.
In prefill the situation reverses. When matrices are large enough, the bandwidth savings from 4-bit weights overwhelm the dequantization overhead.
Per-channel quantization (per-channel scale, no zero-point) sits between the two. Its overhead is lower than per-group, and its accuracy is also lower. In batch=1 serving, per-channel can offer a better trade-off than per-group.
The Ranking Changes as Batch Size Changes
From the NVIDIA TensorRT-LLM benchmark (H100 80GB, 1024 input tokens, 128 output tokens, speedup vs. FP16):
| Format | LLaMA-v2-7B batch=1 | LLaMA-v2-7B batch=8 |
|---|---|---|
| FP8 | 1.51× | 1.40× |
| INT8 SQ | 1.47× | 1.32× |
FP8 speedup is 1.51× at batch=1 but drops to 1.40× at batch=8. As the share of output token generation increases, the memory-bound portion of execution grows and the FLOPS advantage of FP8 is diluted.
Peak throughput tells a different story. For GPT-J 6B at batch=64, H100 FP8 reached 10,907 tok/s vs. 3,679 tok/s for A100 FP16. With TTFT capped at 500ms, FP8 achieved a 2.3× speedup over FP16 at batch=16 — this is the regime where FP8 Tensor Core gains show up in measured results, driven by long inputs where prefill dominates.
The efficiency gap widens further for large models. From the "Give Me BF16 or Give Me Death" paper (2024), synchronous serving of a 405B model on H100:
| Format | Latency | Cost Efficiency |
|---|---|---|
| BF16 | 15.0s | 5 queries/USD |
| FP8 | 7.8s | 19 queries/USD |
| INT4 | 7.8s | 37 queries/USD |
FP8 and INT4 have identical latency, but INT4 nearly doubles the cost efficiency. At the same latency, INT4's memory savings allow more replicas to be deployed.
Combining KV Cache Quantization with Weight Quantization
Applying both weight FP8 and KV cache FP8 simultaneously in vLLM:
vllm serve meta-llama/Llama-3-8B \
--quantization fp8 \
--kv-cache-dtype fp8
Combining AWQ (W4A16) weights with an FP8 KV cache:
vllm serve meta-llama/Llama-3-8B \
--quantization awq \
--kv-cache-dtype fp8
Applying both quantizations together gives multiplicative memory savings. Relative to BF16 weights + BF16 KV cache (roughly 140GB for a 70B model), switching to FP8 weights + INT8 KV cache can cut total GPU memory footprint to 50–55%, since both weights and KV cache are halved.
Errors also compound. Weight quantization error tends to concentrate in early layers near the embedding and late layers near the output. KV cache quantization error is most pronounced where attention scores are peaked, and gets worse with fewer GQA heads and longer context. Which layers see overlapping error depends on the model's GQA ratio and layer depth.
vLLM provides an option to exclude specific layers from KV cache quantization:
vllm serve <model> \
--kv-cache-dtype fp8 \
--kv-cache-dtype-skip-layers 0 1 23
Keeping a few early and late layers on BF16 KV cache preserves most of the memory savings while minimizing accuracy loss. For reference, FP8 weight quantization incurs an MMLU accuracy loss of 0.21%–0.89% under TRT-LLM, while AWQ (INT4) incurs 0.85%–2.11%.
Which Format to Try First
Start with hardware. On A100, there are no FP8 Tensor Cores, so using FP8 for weight serving yields no FLOPS benefit. On A100, INT8 (W8A8) is the practical starting point for quantization acceleration.
On H100 or newer, identify where your service's bottleneck is. Services with tight TTFT SLAs — long-document summarization, RAG pipelines — have prefill as the critical path. With large batches and long inputs, FP8 Tensor Cores are actually engaged and throughput gains show up in measured results. Low-latency chat with tight TPOT SLAs has decode as the critical path and low concurrency — W4A16 is the better fit for that scenario.
For large models (70B+) where GPU memory is tight, using INT4 (AWQ/GPTQ) to run more replicas is a more direct way to increase throughput for the same infrastructure cost. The 405B serving result showing INT4 achieving nearly 2× the throughput-per-dollar of FP8 supports this.
H200 has the same Tensor Core specs as H100 but adds HBM3e for higher memory bandwidth. In decode-bound scenarios, H200's increased bandwidth reduces the relative advantage of W4A16, so the gap between W4A16 and FP8 is likely narrower on H200 than on H100.