Why W4A16 Falls Behind W8A8 at Large Batch Sizes

"Which quantization is fastest?" — any benchmark post that answers this without conditions is hiding its assumptions. Posts claiming W4A16 is faster assume small decode batches; posts claiming FP8 is better assume large-scale prefill workloads. Even on the same model and the same GPU, changing the batch size alone can flip which method wins. This post explains that reversal through the lens of Arithmetic Intensity.

Arithmetic Intensity: Why Decode and Prefill Hit Opposite Bottlenecks

Arithmetic Intensity (AI) is FLOPs performed divided by bytes moved.

AI = FLOPs / bytes_accessed

A GPU is bound by whichever saturates first: compute units (TFLOPS) or memory bandwidth (HBM GB/s). If AI is below the GPU's ridge point (= peak TFLOPS ÷ peak BW), execution is bandwidth-bound; above it, compute-bound. The H100 SXM's ridge point is approximately 295 FLOP/byte.

During the decode phase (bs=1), a Transformer linear layer reads the entire weight matrix from HBM but uses each weight exactly once. With BF16 weights, AI ≈ 1 FLOP/byte — about 0.3% of the ridge point. It is completely bandwidth-bound.

Prefill is the opposite. When processing an input of sequence length S, the same weights are reused S times, so AI ≈ 2S FLOP/byte. At S=2048, AI ≈ 4,096 — 14× above the ridge point. Same model, same GPU, but prefill is compute-bound.

As batch size B increases, decode AI rises too, since the weights are loaded once and reused across B sequences. For a single linear layer, the formula is straightforward:

AI(decode, bs=B) ≈ 2B / bytes_per_weight

Plugging in bytes_per_weight for each weight format and the ridge point (295) yields the batch size threshold at which execution transitions to compute-bound.

W4A16: Strong at Small Decode Batches, Weakens as Batch Size Grows

W4A16 stores weights in INT4 and dequantizes them to FP16 just before the GEMM, which then runs on FP16 Tensor Cores. This reduces storage to 0.5 bytes per weight.

In the bandwidth-bound regime (small decode batches), this design is optimal. When the bottleneck is the rate at which weights are read from HBM, compressing weights by 4× reduces read time proportionally. The dequantize kernel adds overhead, but since compute is not yet the bottleneck, that overhead is hidden. Per Lin et al. (AWQ, 2023), decode speedups reach 1.89× on LLaMA2-7B, 2.41× on LLaMA2-13B, and 1.96× on LLaMA3-8B.

At larger batch sizes, the picture changes. Substituting bytes_per_weight = 0.5 into the AI formula, the batch size threshold at which W4A16 transitions to compute-bound on an H100 SXM is approximately 74.

Past bs=74, W4A16 enters the compute-bound regime. The dequantize cost moves onto the GEMM critical path, and while the bandwidth savings from weight compression have already been realized, there is no compute acceleration — so the advantage over FP16 erodes quickly.

Prefill (sequences in the hundreds or more) is compute-bound from the start. W4A16 only adds a dequantize kernel before an otherwise unaccelerated FP16 GEMM. Saving memory in a regime where memory is plentiful and compute is the bottleneck does nothing for prefill throughput — and the dequantize overhead can actually make it slower than BF16.

W8A8 and FP8: The Conditions for a Reversal at Large Prefill Batches

W8A8 (INT8) and FP8 reduce both weights and activations to lower precision and execute the GEMM directly on INT8 or FP8 Tensor Cores — no upcasting to FP16.

In the compute-bound regime, these formats have two advantages. Weight bytes are halved, reducing HBM traffic, and INT8/FP8 Tensor Cores deliver 2× the throughput of FP16 at the same clock. Since compute is the bottleneck, the GEMM itself runs faster, which directly benefits prefill.

SqueezeBits benchmarks (H100-PCIe, LLaMA-3.1-8B, vLLM v0.6.2, prefill-heavy workload) put numbers to the reversal:

At small batches, W4A16 leads by nearly 2×. At large batches, W8A8 overtakes it. While W8A8 holds a steady ~40% gain regardless of batch size, W4A16's advantage shrinks to 10%.

In decode-heavy workloads the numbers differ. W8A8 shows ~40% over FP16 at bs=1, dropping to 15–20% at bs=256. The attention layers remain in FP16, and at small decode batches the weight savings (2×) are smaller than W4A16's (4×), yielding less bandwidth benefit. For pure small-batch decode, W4A16 still wins.

FP8 extends this advantage further on H100. The E4M3 format has a wider dynamic range than INT8, making it easier to handle activation outliers, and the vLLM documentation cites up to 1.6× throughput improvement over FP16.

The Feedback Loop When Combining KV Cache Quantization

W4A16 reduces the model parameter footprint to one-quarter of BF16. For LLaMA-3-8B, that is roughly 16 GB in BF16 → ~5 GB in W4A16. The freed VRAM accommodates more KV cache entries and raises the maximum batch size ceiling.

Adding FP8 quantization on top of the KV cache pushes that ceiling even higher. The problem is that a higher ceiling means larger actual operating batches, and larger batches drive prefill deeper into the compute-bound regime.

Running the full capacity enabled by W4A16 + KV FP8 causes prefill TTFT (Time To First Token) to grow. Throughput increases, but tail latency worsens. Even if the memory savings raise the batch ceiling by N%, the throughput gain is smaller than N% once the prefill compute bottleneck intensifies.

This combination pays off in offline batch processing, where there is no latency SLA and only throughput matters — W4A16 + KV FP8 becomes the best choice for maximizing tokens per GPU. For online serving with a tight latency budget, even if the batch ceiling rises, the actual operating batch size must be kept within SLA bounds, which limits the additional benefit of KV quantization.

Commands to configure the main combinations in vLLM:

# W4A16 (AWQ) — optimized for small-batch decode latency
vllm serve meta-llama/Llama-3-8B-Instruct \
  --quantization awq

# W4A16 + KV cache FP8 — push past memory limits, maximize throughput
vllm serve meta-llama/Llama-3-8B-Instruct \
  --quantization awq \
  --kv-cache-dtype fp8_e5m2

# FP8 W8A8 — large-batch throughput on H100/RTX 4090
vllm serve meta-llama/Llama-3-8B-Instruct \
  --quantization fp8

Hardware Support Matrix and Practical Selection Criteria

Even with the same FP8 configuration, there is a significant performance difference between native W8A8 computation and a weight-only fallback, depending on the GPU.

H100 (Hopper, sm90) — The GPU where FP8 has the most impact. It includes the Transformer Engine, and FP8 Tensor Core throughput is approximately 3,958 TFLOPS — twice BF16 and six times A100 FP16. Both E4M3 and E5M2 formats are supported natively, and FP8 incurs less accuracy loss than INT8 on models with many activation outliers.

A100 (Ampere, sm80) — vLLM has no native FP8 W8A8 support. Loading an FP8 model falls back to weight-only FP8 (W8A16) via the FP8 Marlin kernel, with activations remaining in FP16. For W8A8 on A100, INT8 SmoothQuant is the practical choice; A100 INT8 Tensor Cores deliver 2× over FP16. For small-batch decode, W4A16 is still the best option on A100.

RTX 4090 (Ada Lovelace, sm89) — Ada Lovelace includes FP8 Tensor Cores, enabling native FP8 W8A8 computation. Among consumer GPUs, it is the only option that can leverage FP8 W8A8 comparably to H100. The 24 GB VRAM limit is a hard constraint — 70B-class models do not fit on a single card without W4A16.

The decision order is GPU first, serving pattern second. Confirm whether the GPU natively supports FP8 W8A8, then measure the average concurrent request count in your actual serving environment. On H100, W4A16 wins on throughput below bs=74; FP8 wins above it. This threshold varies by GPU model and parameter count, so direct profiling is required for accurate numbers.