GPU를 4장에서 8장으로 늘렸는데 throughput이 1.5배 수준에 그쳤다는 얘기를 현장에서 자주 듣는다. 텐서 병렬의 이론 기대치는 GPU 수만큼 연산이 쪼개지니 거의 선형에 가까운 스케일링인데, 왜 실제는 다를까.
Going from 4 to 8 GPUs and seeing only a 1.5× throughput gain is a common complaint in production. In theory, tensor parallelism splits computation proportionally across GPUs, so scaling should be nearly linear — but in practice it isn't.
The reason: an All-Reduce is required at the end of every Transformer layer. That communication doesn't disappear as you add GPUs, and its cost varies by an order of magnitude depending on the interconnect. This is why throughput doesn't scale linearly with GPU count.
Where All-Reduces Occur: Exactly 2 Per Layer
In the tensor-parallel design proposed by Megatron-LM (Shoeybi et al., 2019), each Transformer layer contains two blocks, each of which produces one All-Reduce.
The MLP block is split into a column-parallel linear transform followed by a row-parallel linear transform. Each GPU holds a subset of the weight columns and independently computes its partial output; the row-parallel results must then be All-Reduced so the next layer receives the correct activations. The Self-Attention block works the same way: Q/K/V projections are distributed across GPUs by head, and the output projection results are gathered via All-Reduce.
Two All-Reduces per layer. Llama-3 70B has 80 layers, so decoding a single token triggers 160 All-Reduces. Prefill processes the entire sequence in one forward pass — still 160 All-Reduces — but decode repeats those 160 All-Reduces for every token generated. Generating 1,000 tokens means 160,000 All-Reduces.
Communication Volume: A Function of TP Degree
For a ring All-Reduce, the bytes actually transferred per GPU are:
bytes transferred = 2 × (TP-1)/TP × batch_size × seq_len × hidden_dim × sizeof(dtype)
In the decode phase, seq_len=1. Plugging in batch=1, FP16 (2 bytes), and Llama-3 70B (hidden_dim=8192):
| TP | (TP-1)/TP | All-Reduce transfer per layer | 80 layers × 2 total |
|---|---|---|---|
| 2 | 0.500 | 16 KB | 2.56 MB |
| 4 | 0.750 | 24 KB | 3.84 MB |
| 8 | 0.875 | 28 KB | 4.48 MB |
Going from TP=2 to TP=8, the per-layer transfer only grows from 16 KB to 28 KB — a 1.75× increase. The raw numbers aren't large. The problem is that those 160 All-Reduces execute serially. The next layer's computation cannot start until the previous All-Reduce completes, so per-iteration latency accumulates into a genuine bottleneck.
Effective Bandwidth by Interconnect Topology
NCCL defaults to ring All-Reduce for intra-node communication. Ring requires 2(N−1) send/recv steps across N GPUs, which means the full aggregate bandwidth is never realized. You can measure this gap with nccl-tests all_reduce_perf:
# 4-way GPU all-reduce, message sweep from 1MB to 256MB
./build/all_reduce_perf -b 1M -e 256M -f 2 -g 4
On an A100 SXM 8-way system (NVLink Gen 3, 600 GB/s aggregate theoretical), measured bus bandwidth for a 256 MB message is roughly 200–225 GB/s. Point-to-point tests on the same hardware yield 430–520 GB/s, so ring All-Reduce effective throughput is about half of P2P.
PCIe 4.0 is an entirely different world. When GPUs are connected indirectly through the CPU, 4-way all-reduce algorithm bandwidth drops to 13.6 GB/s and bus bandwidth falls to around 20 GB/s — less than one-tenth of NVLink. For multi-node setups, InfiniBand HDR provides 200 Gb/s (≈25 GB/s), but since the outermost hop in a multi-node ring All-Reduce traverses IB, the IB bandwidth becomes the global bottleneck regardless of the intra-node NVLink bandwidth.
Translating the theoretical transfer latency of a 24 KB All-Reduce (TP=4, batch=1) across environments:
| Interconnect | Effective bus bandwidth | 24 KB theoretical latency | 1 token (160 All-Reduces) cumulative |
|---|---|---|---|
| NVLink A100 SXM | ~200 GB/s | ~0.12 µs | ~19 µs |
| PCIe 4.0 (via CPU) | ~20 GB/s | ~1.2 µs | ~192 µs |
| InfiniBand HDR | ~25 GB/s | ~0.96 µs | ~154 µs |
These are lower-bound estimates covering only raw transfer time. Real NCCL All-Reduce also incurs kernel launch overhead, protocol handshaking, and synchronization waits — all of which inflate latency further for small messages like 24 KB. For Llama-3 70B at TP=4, batch=1, a single decode step takes roughly 15–20 ms (dominated by reading ~35 GB of weights per GPU from HBM at ~2 TB/s). In a PCIe environment, All-Reduce overhead becomes a non-trivial fraction of that total step time.
To inspect the All-Reduce intervals directly, profile with nsys:
nsys profile --trace=cuda,nccl --output profile_tp4 python run_inference.py
In the Nsight Systems GUI, placing the NCCL kernel timeline alongside CUDA operations makes the gap between each All-Reduce and the start of the next GEMM immediately visible. Seeing it directly is more convincing than any calculation.
Communication-to-Compute Ratio Through the Lens of Arithmetic Intensity
The decode phase is memory-bandwidth-bound. At batch=1, seq_len=1, the arithmetic intensity of each GEMM is roughly 1–2 FLOP/Byte. For Llama-3 70B at TP=4, each GPU holds about 35 GB of weights (70B parameters × 2 bytes ÷ 4). A batch=1 decode reads those weights once and performs 35G FLOPs, giving AI ≈ 1 FLOP/Byte. The A100 roofline crossover is 312 TFLOPS ÷ 2 TB/s = 156 FLOP/Byte, so decode runs at roughly 1/150th of that crossover point — compute units are almost entirely idle.
Larger batches change the picture. At batch=32, the same weights are read while performing 32× more computation, pushing AI to ≈32 FLOP/Byte. Still memory-bound, but the actual compute time per layer increases significantly. The absolute All-Reduce latency doesn't change, so as GEMM time grows, communication occupies a smaller relative fraction of the step.
In low-batch, latency-sensitive serving, increasing TP makes the communication overhead proportionally larger. In high-batch throughput serving, the benefit of distributing HBM pressure outweighs the communication cost. The regime where TP pays off shifts with batch size.
Practical Guidance: When Increasing TP Hurts
| Model size | Batch | Interconnect | Recommended TP | Rationale |
|---|---|---|---|---|
| 70B | 1–4 | NVLink | 4 | Higher comm ratio vs. TP=8, negligible latency gain |
| 70B | 1–4 | PCIe | Minimum required | All-Reduce dominates decode latency |
| 70B | 32+ | NVLink | 8 | HBM distribution benefit > communication overhead |
| 70B | 32+ | PCIe | ≤4 | PCIe comm cost > memory distribution gain |
| ≤13B | Any | Single node | 1 (if possible) | Eliminate All-Reduce entirely |
| 140B+ multi-node | Any | InfiniBand | Minimize intra-node | IB bottleneck; consider combining with PP |
TP=8 on a PCIe server is almost always a loss. Even on NVLink hardware, for batch 1–4 real-time serving, measured throughput between TP=8 and TP=4 can be nearly identical or even favor TP=4. Unless the model doesn't fit on fewer GPUs and you have no choice, start at the minimum TP degree that fits the model and verify with nsys before going higher.
Practical Limits of Async Overlap
Both vLLM and Megatron-LM attempt to overlap communication and computation using async All-Reduce. This works well in the prefill phase: long sequence lengths mean GEMM time per layer stretches to several milliseconds, providing ample room to hide the All-Reduce.
Decode is structurally different. At batch=1, GEMM time per layer is only tens to hundreds of microseconds. On NVLink, the All-Reduce itself is so fast that whether it overlaps hardly matters. On PCIe, the All-Reduce is long enough that there isn't enough computation to hide it behind. Research like TokenWeave (2025), which fine-grained decomposes operations to improve decode overlap efficiency, exists precisely because of this structural constraint.