When AllReduce Becomes the Bottleneck in Tensor Parallelism

Many posts explain that Tensor Parallel (TP) fails to scale throughput linearly due to "communication overhead." But few actually show, with equations and numbers, how large that overhead is and under what conditions it exceeds compute time. The answer depends on the product of three variables: batch size, GPU count, and interconnect bandwidth.

Communication TP Introduces: Two AllReduces per Layer

The TP pattern defined in the Megatron-LM paper (Shoeybi et al., 2019) is still used as-is by vLLM, SGLang, and TensorRT-LLM. Each Transformer layer triggers two AllReduces.

In the MLP block, the gate/up projections are split column-wise and the down projection row-wise. After each GPU computes its portion of the down projection, an AllReduce sums the results. The attention block follows the same pattern: the QKV projection is partitioned, and an AllReduce follows the output projection. For a 70B model with 80 layers, a single forward pass generates 160 AllReduces.

A Formula for AllReduce Cost

In the Ring-AllReduce algorithm, the total data each GPU sends and receives is:

bytes transferred = 2 × (N-1)/N × M

where N is the number of GPUs and M is the size of the tensor being reduced (in bytes). As N grows, the coefficient approaches 2. For N=2 it is 1.0, N=4 gives 1.5, N=8 gives 1.75, and N=16 gives 1.875. Each time you double the GPU count, the incremental increase in this coefficient shrinks.

Concrete Numbers for a 70B Model

Using a 70B LLaMA-2-class model with hidden_size=8192, BF16 (2 bytes), TP=4, batch=1:

M = 8192 × 1 × 2 = 16,384 bytes = 16 KB
bytes transferred = 16,384 × 2×(4-1)/4 = 24,576 bytes ≈ 24 KB

Looking at raw bandwidth on NVLink at 300 GB/s unidirectional (A100 SXM):

t = 24,576 / (300 × 10⁹) ≈ 0.08 μs

That number should not be taken at face value. NCCL AllReduce carries kernel launch and synchronization overhead, which means actual latency on small tensors routinely exceeds 10–30 μs. In batch=1 decode, this latency floor dominates.

Scaling to batch=128 changes the picture:

M = 8192 × 128 × 2 = 2,097,152 bytes = 2 MB
bytes transferred = 2 MB × 1.5 = 3 MB
t = 3 × 10⁶ / (300 × 10⁹) ≈ 10 μs

Once tensors reach the multi-MB range, bandwidth becomes the real bottleneck. That works out to ~10 μs on NVLink and ~93 μs on PCIe (unidirectional ~32 GB/s) for the same transfer.

AllReduce Transfer Volume by TP Degree (7B, hidden_size=4096)

Going from TP=2 to TP=4 increases transfer volume by 1.5×. Compute is halved, but communication increases. At TP=8 the coefficient is 1.75, so transfer volume is 17% higher than TP=4 while compute is halved again.

Compute Time vs. Communication Time: The Breakeven Point

During decode (batch=1, seq=1), the GEMM each GPU handles has the shape [1, hidden_size] × [hidden_size, hidden_size/TP]. With a single-row input vector, this operation is not compute-bound — it is memory-bound. The time to read the weight matrix from HBM dominates.

For a 7B model at TP=4 on an A100 (HBM bandwidth ~2 TB/s), reading the MLP down projection weights [4096, 1024] in BF16 takes:

t_gemm = 4096 × 1024 × 2 / (2 × 10¹²) ≈ 4.2 μs

The empirical NCCL AllReduce latency floor is 10–20 μs. At batch=1, communication is already longer than compute. Increasing TP degree shrinks the partitioned GEMM further while increasing AllReduce data volume, so the communication fraction only grows. At TP=8 the GEMM drops to ~2 μs, but the AllReduce still takes 10 μs or more.

In prefill (seq=2048, batch=1) the situation reverses. The GEMM tensor grows to [2048, 4096] × [4096, 1024], approaching compute-bound territory. At A100's 312 TFLOP/s, the theoretical time for this GEMM is in the hundreds of microseconds. A 10 μs AllReduce is just a few percent of that.

Overlap Strategies and Their Limits

The idea behind overlap is straightforward: launch the AllReduce asynchronously on a separate CUDA stream while the next layer's GEMM runs concurrently. If AllReduce time < next-layer GEMM time, the communication is effectively free.

This condition holds during prefill. GEMMs are long enough that the AllReduce hides completely behind them. In decode at batch=1, the GEMM is 4–10 μs, but the NCCL kernel overhead alone already exceeds 10 μs. When communication is longer than the compute you are trying to hide it behind, overlap is not a free lunch — it is just waiting.

Experiments with MSCCL++ (2025) report an average 1.11× improvement in Llama3-70B decode latency, which implies that with standard NCCL, AllReduce accounts for roughly 10% of decode step time. TokenWeave (2025) pursues fine-grained overlap to reduce this overhead further.

NVLink vs. PCIe vs. RDMA

NVLink 4th gen (H100 SXM) delivers ~450 GB/s unidirectional; 3rd gen (A100 SXM) delivers ~300 GB/s. H100 PCIe inter-GPU bandwidth is ~64 GB/s — roughly 7× lower than SXM.

On a PCIe server, there is a real batch size threshold below which TP=4 actually yields lower throughput than TP=2. Around batch=8–16, AllReduce cost already exceeds GEMM time. Even though compute is halved, if communication is 4–5× more expensive over PCIe, there is no net gain. On PCIe, the only valid reason to increase TP is that the model does not fit on a single GPU.

Even on NVLink, diminishing returns set in beyond TP=8. Going from N=8 to N=16 halves compute but raises the Ring-AllReduce coefficient from 1.75 to only 1.875 — a 7% increase in transfer volume. Cross-node communication is a different matter entirely. InfiniBand HDR unidirectional bandwidth is ~25 GB/s, more than 10× lower than NVLink. Cross-node TP causes AllReduce latency to jump from tens of microseconds to hundreds, making it counterproductive for nearly every workload.

Measuring Communication Fraction Directly

The criterion for choosing TP degree should not be "does the model fit on one GPU?" but rather "what fraction of step time is communication?" The fastest way to measure it directly:

# Enable NCCL communication logging
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=COLL

# Extract NCCL kernel timing with nsys
nsys profile --trace=cuda,nvtx \
  --output=profile_tp4 \
  python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-70b-hf \
    --tensor-parallel-size 4

# Analyze the profile (look at ncclKernel_* kernel times)
nsys stats profile_tp4.nsys-rep --report gputrace

Dividing the time spent in ncclKernel_* by total active GPU time gives the communication fraction. If that number exceeds 15%, consider reducing TP or upgrading your interconnect.

Recommended TP ranges by model size and environment:

Running a 7B model at TP=8 on an 8-GPU NVLink system is one of the most common mistakes. Applying TP to a model that fits on a single GPU increases decode latency — the AllReduce overhead outweighs the reduction in compute.

The remaining open question is where exactly communication and compute cross as batch size grows during decode. That breakeven batch size varies with model architecture, GPU generation, and NCCL version, and is difficult to estimate without profiling. Whenever you introduce new hardware or change TP degree, running an nsys profile once is the most reliable approach.