Why Adding Nodes Hurts LLM Inference Throughput

Adding nodes means more GPUs, and more GPUs should mean faster serving — that intuition is natural. Yet it's not uncommon to scale from a single-node 8-GPU setup to a 2-node 16-GPU setup and watch throughput drop. There is measured data showing that moving Qwen2.5-72B from TP=8 on a single node to TP=16 across two nodes reduced decode throughput by 66%. Twice the GPUs, one-third the performance.

The Bandwidth Cliff at the Node Boundary

Within a single node, GPUs are connected via NVLink. On H100, NVLink 4.0 provides 900 GB/s bidirectional per GPU, and NVSwitch gives all eight GPUs a full-mesh direct connection. AllReduce completing in microseconds is a direct consequence of this topology.

Cross the node boundary and that path disappears. InfiniBand or RoCE/Ethernet takes over instead.

Compared to NVLink, IB HDR delivers 18× lower unidirectional bandwidth. This single number is the structural reason that throughput hits a ceiling in multi-node LLM serving.

Why AllReduce Slows Down Across Nodes

Tensor Parallel inference passes through an AllReduce synchronization point at every transformer layer. Each GPU computes its shard of the Attention and MLP outputs, and those partial sums must be reduced before the next layer can proceed. The communication volume for Ring AllReduce is 2 × (N-1)/N × message size.

A 70B BF16 model has a hidden dimension of 8192. With a batch size of 32 and a single decode step, the layer output tensor is 32 × 8192 × 2 bytes = 512 KB. With TP=8 on a single node, Ring AllReduce transfers 2 × 7/8 × 512 KB ≈ 896 KB per GPU — completing in microseconds over NVLink.

Moving to TP=16 across two nodes routes all AllReduce traffic over IB HDR (25 GB/s). A 70B model has 80 transformer layers, each triggering one Attention AllReduce and one MLP AllReduce. That means 160 cross-node collective operations per generated token. The 3× jump in TTFT — from 19 ms on a single node to 58.5 ms with cross-node TP — comes directly from this structure.

Why Communication Hiding Doesn't Work During Decode

In theory, you can overlap AllReduce with computation by starting the next layer's work before the current AllReduce finishes. This works reasonably well during prefill, where large batches provide enough compute to hide behind.

Decode is different. At each step, every sequence in the batch generates exactly one token. With a sequence length of one, the matrix operation time on each GPU is inherently short. Higher TP degree also means each GPU handles a smaller matrix. At TP=16, each GPU does half the compute of TP=8, leaving even less time for communication to hide within.

The fact that 32-way TP shows 33% MFU and 16-way TP shows 46% MFU follows directly from this. As TP degree increases, communication overhead grows as a fraction of total time, and GPUs spend more time sitting idle. The decode regime with batch sizes of 1–8 is the most vulnerable.

Does Mixing in Pipeline Parallelism Help?

When you have to go multi-node, naively scaling TP to match the node count is worse in practice than combining intra-node TP with inter-node PP.

PP partitions layers across nodes: each node processes only its assigned layer range and passes the result to the next node. The only data transmitted between nodes is the single activation tensor at each layer boundary. Unlike TP, there is no all-GPU synchronization at every layer, so the total communication volume is much lower.

The downside is pipeline bubbles. The smaller the batch, the higher the bubble ratio. For a 405B model configured under various parallelism strategies, the theoretical comparison looks like this:

A TP=8+PP=2 configuration (intra-node TP, inter-node PP) shows nearly 2× the throughput of pure cross-node TP=16 (see measured results below). The hybrid approach clearly wins, but the optimal split depends on batch size and model architecture — it cannot be determined without benchmarking. In vLLM, tensor_parallel_size and pipeline_parallel_size can be specified independently to run combination experiments.

Measured Throughput by Configuration

The table below compares decode throughput for Qwen2.5-72B under a single-request workload. A single request means batch size 1 — the worst-case condition where communication overhead is most visible.

The TP=16 configuration, despite having 2× the GPUs, is 66% slower than single-node TP=8. The cost of 160 AllReduces per token over an IB link overwhelms any gain from compute parallelism. Increasing batch size to 32–128 reduces the proportion of time spent waiting on communication and narrows the absolute gap, but the ranking between configurations does not change.

When Scaling to More Nodes Actually Helps

For multi-node scaling to improve throughput, three conditions must hold simultaneously.

The model must not fit in a single node's memory. At a scale like Llama-3.1 405B, which cannot fit in 8×H100 (640 GB), distributed deployment is mandatory. Qwen2.5-72B in BF16 is roughly 144 GB and fits in 8×H100. There is no good reason to split that model across two nodes.

The network must be IB HDR or better. Running cross-node TP over RoCE 100G (12.5 GB/s) cuts AllReduce throughput in half again compared to IB HDR. Well-tuned RoCE 400G approaches IB NDR, but stock 100G RoCE creates a genuine bottleneck for LLM AllReduce. The reason Meta can operate LLMs at thousands of GPUs on production RoCE clusters is that those clusters run 400G or faster backends.

Batch size must be large enough to hide communication behind computation. In the batch 1–8 decode regime, the exposed communication window is longer than the compute window.

Before making a scaling decision, verify three numbers: (1) model parameter count × 2 (BF16 bytes), (2) effective network bandwidth, and (3) the serving batch size distribution.

Measuring the Communication Bottleneck

# Measure single-node vs. multi-node AllReduce bandwidth with nccl-tests
# Single node (8 GPUs)
./build/all_reduce_perf -b 512M -e 4G -f 2 -g 8

# 2-node 16-GPU (using MPI)
mpirun -np 16 -H node1:8,node2:8 \
  ./build/all_reduce_perf -b 512M -e 4G -f 2 -g 8

The busbw column in nccl-tests output is the right metric for comparison. If a single node shows hundreds of GB/s, the multi-node run will show 20–50 GB/s, reflecting the IB link bandwidth. The larger this drop, the more clearly cross-node AllReduce is the bottleneck.

Distinguishing communication bottlenecks from compute bottlenecks requires checking SM utilization alongside throughput.

# Log NCCL AllReduce time
NCCL_DEBUG=INFO NCCL_ALGO=Ring python serve.py 2>&1 | grep AllReduce

# Measure GPU SM activity
dcgmi dmon -e 1001,1002,1003

If SM utilization is low (60–70%) and throughput is also low, communication stalls are the cause. If SM utilization is above 90% and throughput is still low, compute itself is the bottleneck — upgrading the network will not help. Failing to distinguish these two cases means spending money on an IB upgrade and seeing no improvement.