How Arithmetic Intensity Shifts with Batch Size in LLM Serving: 7B, 13B, and 70B Models Compared

When first operating vLLM, most engineers run into the same question: what should max_num_seqs be set to? And can you use the same value for a 7B model and a 70B model on the same A100?

The short answer is: no. And the reason is far more specific than "70B uses more memory" — it also depends on context length.

Why decode is memory-bound: one equation

In the Roofline model (Williams et al., 2009), whether the bottleneck is memory or compute comes down to a single number: arithmetic intensity (AI) — FLOPs/byte. If AI is below the GPU's ridge point (= peak compute / memory bandwidth), you're memory-bound; above it, compute-bound.

For the A100 80GB, ridge point = 312 TFLOPS (BF16 TC) / 1.94 TB/s ≈ 161 FLOPs/byte.

For a decode step processing a batch of B sequences, the work breaks down as:

• FLOPs ≈ 2BP (P = number of parameters)
• Bytes loaded = weights W + KV cache B × kv_per_seq

At B=1, AI ≈ 2P / (W + kv_per_seq). In BF16, W = 2P, so AI ≈ 1 — two orders of magnitude below the ridge point of 161. Decode stays in the memory-bound regime at any batch size — no matter how large the batch, Llama-3 8B decode on an A100 never crosses into compute-bound territory.

How arithmetic intensity shifts as batch size grows

Scaling to batch size B:

AI(B) = 2BP / (W + B × kv_per_seq)

FLOPs scale with B, but the weight footprint W is fixed regardless of batch size. KV cache bytes grow proportionally with B. In the limit B → ∞, AI(∞) = 2P / kv_per_seq. For Llama-3 8B, this works out to roughly 30 FLOPs/byte — still below the ridge point (161).

The reason to increase batch size anyway becomes clear when you look at how TPOT (time per output token) behaves:

TPOT(B) = (W + B × kv_per_seq) / (B × BW)
         = W/(B × BW)  +  kv_per_seq/BW

The first term decreases as B grows — B requests share the cost of loading weights. The second term, kv_per_seq/BW, is a constant independent of B, and represents the theoretical floor on TPOT.

The saturation batch size B* is defined as the point where the two terms are equal:

B* = W / kv_per_seq

Below B*, increasing batch size meaningfully improves TPOT. At B*, the two terms are equal and TPOT is exactly 2× the floor. Beyond B*, the per-request improvement from adding more sequences drops off sharply — you've entered the practical throughput saturation regime.

Computing B* for different models

Spelled out concretely:

W(BF16) = num_parameters × 2 bytes
kv_per_seq = 2 × num_kv_heads × head_dim × num_layers × seq_len × 2 bytes

For Llama-3 8B (BF16), 8 KV heads, head_dim 128, 32 layers:

kv_per_token = 2 × 8 × 128 × 32 × 2 bytes = 131,072 bytes = 128 KB/token
kv_per_seq(512t)  = 128 KB × 512  = 64 MB
kv_per_seq(2048t) = 128 KB × 2048 = 256 MB
kv_per_seq(8192t) = 128 KB × 8192 = 1,024 MB

B*(512t)  = 16 GB / 64 MB   = 256
B*(2048t) = 16 GB / 256 MB  = 64
B*(8192t) = 16 GB / 1,024 MB = 16

With W4A16 quantization applied to Llama-3 8B, the weight footprint drops to 4 GB, so B*(512t) = 4 GB / 64 MB = 64. The KV cache size is unchanged, so quantization moves B* down.

Applying the same analysis to Llama-3 70B, Llama-2 13B (MHA architecture), and Mixtral 8x7B:

One number goes against intuition: Llama-2 13B (MHA) has B*(512t) = 65, which is 4× lower than Llama-3 8B (GQA) at 256. Despite having more parameters, its saturation batch size is lower. MHA retains all 40 KV heads, so kv_per_token is 800 KB — 6.25× that of Llama-3 8B (128 KB). The KV cache grows much faster than the weight footprint (13/8 ≈ 1.6×), which is what drives B* down.

The 70B vs. 8B comparison follows the same logic. The parameter ratio is 70/8 ≈ 8.75×, but the B* ratio is only 875/256 ≈ 3.4×. The 70B model also has 8 KV heads, but its layer count goes from 32 to 80, pushing kv_per_token to 320 KB — 2.5× larger. W grows by 8.75× while kv_per_token grows by 2.5×, so the ratio comes out to 3.4×. "It's a 70B, so target 10× the batch size of the 7B" doesn't hold.

Mixtral's high B* comes from its MoE architecture: the full weight matrix (~93 GB in BF16) must reside in memory, but the attention architecture uses the same 8 KV heads as Llama-3 8B, keeping kv_per_token at 128 KB. The numerator in B* is total weight bytes, not active-parameter bytes (~13B).

Three regimes of the TPOT curve during a batch sweep

Running Llama-3 8B on an A100 80GB with vLLM, fixed context of 512 tokens, sweeping max_num_seqs from 1 to 256, the TPOT curve falls into three distinct regimes.

Regime 1 (B ≤ ~30): TPOT drops rapidly. Weight loading dominates — doubling the batch roughly halves TPOT. At B=1, TPOT ≈ W/BW = 16 GB / 2 TB/s ≈ 8 ms/token.

Regime 2 (B ~ 30 to B)**: Improvement rate slows. The KV cache term starts catching up to the weight term. At B = B(512t) ≈ 256, TPOT = (16 GB + 256 × 64 MB) / (256 × 2 TB/s) ≈ 64 μs/token — exactly 2× the theoretical floor (kv_per_seq/BW = 32 μs).

Regime 3 (B > B_mem): TPOT starts increasing. Once the KV cache exhausts GPU memory, the vLLM scheduler begins preempting sequences or swapping them to CPU, and that cost shows up directly in TPOT.

The maximum memory-feasible batch size B_mem is determined by the KV cache pool remaining after the model weights are loaded:

B_mem = (GPU_mem × utilization - W) / kv_per_seq
      = (80 GB × 0.9 - 16 GB) / 64 MB  ≈  900   # Llama-3 8B, 512 tokens

At 512 tokens: B* = 256, B_mem = 900. The range 256–900 is the zone where gains are marginal but risk is growing. There's no reason to push max_num_seqs that high.

Monitoring HBM bandwidth utilization with nvidia-smi dmon: in regimes 1–2, utilization saturates around 1.7–1.9 TB/s. Once you enter regime 3, the GPU idles waiting on swaps, bandwidth numbers drop, and TPOT spikes. The point where bandwidth is saturated but TPOT stops improving is the practical B*. When bandwidth drops and TPOT starts rising, you've crossed B_mem.

At 8192-token context, the picture changes significantly. B* = 16, B_mem ≈ (80 × 0.9 - 16) GB / 1 GB = 56. Regimes 1–2 compress into the range below B=16, and pushing past B=32 already puts you in the danger zone. A batch size of 512 that was safe at short context will start causing preemptions above 32 at 8192 tokens.

How context length pulls B* down

Reading across the table makes it clear. For Llama-3 8B:

• 512 → 2048 tokens: B* drops from 256 → 64 (one quarter)
• 512 → 8192 tokens: B* drops from 256 → 16 (one sixteenth)

Since kv_per_seq = kv_per_token × seq_len and B* = W / kv_per_seq, quadrupling context length cuts B* by exactly a factor of four. This inverse relationship holds regardless of model size.

For Llama-3 70B + 8192 tokens on 4× A100 with tensor parallelism: B* = 56, B_mem = (320 × 0.9 - 140) GB / 2.5 GB ≈ 59. B* ≈ B_mem. Push the batch past 60 and preemptions begin immediately. The same convergence happens at short context (512 tokens) where B*(875) ≈ B_mem(925) — for 70B models, memory and bandwidth limits bind simultaneously.

For Llama-3 8B + 512 tokens, there's a wide gap between B*(256) and B_mem(900). Pushing batch size to 900 in this range yields marginal throughput improvement but makes tail latency hard to manage. I call this the "diminishing returns, rising risk" zone.

How far GQA/MQA raises B*

If Llama-3 8B used MHA, kv_per_token = 2 × 32 × 128 × 32 × 2 = 524 KB, and B*(512t) ≈ 16 GB / 262 MB ≈ 61. GQA (4 groups) pushes B* up to 256 — roughly a 4.2× gain.

This gain scales proportionally with context length — GQA reduces kv_per_token by the group ratio, and B* rises by the same factor. However, as context grows, B_mem also shrinks, and once B* ≈ B_mem, the operational benefit of GQA's higher B* is hard to realize. For Llama-3 8B at 8192 tokens, GQA gives B* = 16, but B_mem is only 56. Without GQA, B* would be 4 — but in both cases memory is the binding constraint.

Operations: how to set max_num_seqs

Computing B* and B_mem is straightforward:

def bstar(model_bytes, num_kv_heads, head_dim, num_layers, seq_len, dtype_bytes=2):
    kv_per_seq = 2 * num_kv_heads * head_dim * num_layers * seq_len * dtype_bytes
    return int(model_bytes / kv_per_seq)

def b_mem(gpu_mem_gb, model_bytes, num_kv_heads, head_dim, num_layers, seq_len, util=0.9, dtype_bytes=2):
    kv_pool = gpu_mem_gb * 1e9 * util - model_bytes
    kv_per_seq = 2 * num_kv_heads * head_dim * num_layers * seq_len * dtype_bytes
    return int(kv_pool / kv_per_seq)

# Llama-3 8B, 512 tokens, single A100 80GB
bstar(16e9, 8, 128, 32, 512)           # → 256
b_mem(80, 16e9, 8, 128, 32, 512)       # → 900

# Llama-3 70B, 2048 tokens, 4× A100 80GB
bstar(140e9, 8, 128, 80, 2048)         # → 219
b_mem(320, 140e9, 8, 128, 80, 2048)   # → 231

When B* ≤ B_mem, use max_num_seqs = B* as a starting point. To validate with an empirical sweep:

for bs in 1 4 8 16 32 64 128 256; do
  python benchmarks/benchmark_serving.py \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --max-num-seqs $bs \
    --input-len 512 --output-len 64 \
    --num-prompts 2000 \
    2>&1 | grep -E "mean_tpot_ms|output_throughput"
done

The point where TPOT improvement drops below 5% compared to the previous step is the practical B*. Set max_num_seqs to that value.

When B* ≈ B_mem (long context + large model), set the limit conservatively at 80–90% of B_mem. A single preemption event can add tens of milliseconds of latency, so from a tail latency perspective, memory headroom matters more than throughput headroom.

vLLM V1's default max_num_seqs is 1024. That number is only meaningful for small models at short context. On 70B + 8192 tokens, that default will trigger an immediate preemption storm. max_num_seqs is a parameter you need to recompute every time your model or context length distribution changes.