Drag each answer to line up with its matching prompt
PCIe Gen5 x16
Workload that mostly only needs PCIe for streaming weights in and tokens out.
NVLink 4 (H100 SXM)
GPU to GPU interconnect at roughly 900 GB/s aggregate per card.
Tensor parallelism all-reduce
Standard CPU to GPU bus at roughly 64 GB/s per direction (~128 GB/s bidirectional).
Single GPU inference
Cross-GPU traffic that only scales well when GPUs are NVLinked, not PCIe bridged.
NVLink is the fast GPU-to-GPU fabric (H100 SXM ~900 GB/s, B200 ~1.8 TB/s). PCIe is the slower CPU-to-GPU bus (Gen5 x16 ~64 GB/s per direction).
Imagine a kitchen with several chefs. Each chef has a private station, and they all need ingredients from the pantry. The hallway between the pantry and the stations is the regular bus that everyone uses. Now imagine the chefs sometimes need to pass ingredients directly to each other to finish a dish together. If the chefs are connected by a tiny door between stations, ingredients fly across in seconds. If they have to walk all the way back through the pantry hallway to hand each other things, the dish takes ten times longer. NVLink is the tiny door between GPU stations. PCIe is the hallway. Workloads that need a lot of GPU-to-GPU traffic run fast on NVLink and slowly on PCIe.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-GPU inference servers have two completely different bandwidth concerns, and most performance bugs at this layer come from confusing them. HBM memory bandwidth is how fast each GPU can read its own weights, measured in TB/s. Interconnect bandwidth is how fast chips talk to each other, measured in GB/s for PCIe and TB/s for NVLink. The two numbers cap different workloads, and picking the wrong topology for the workload silently cuts the throughput you expected.
Single-GPU inference is fine on PCIe. Tensor parallelism needs NVLink-class fabric because it runs an all-reduce in every layer. Multi-node clusters add InfiniBand as a third, slower tier.
PCIe: the universal CPU/GPU bus
PCIe (Peripheral Component Interconnect Express) is the standardized bus every modern server uses to connect the CPU to peripherals: GPUs, network cards, NVMe SSDs, accelerators of all kinds. It is point to point and full-duplex, and each generation has roughly doubled bandwidth.
The practical numbers at x16 (the width that GPUs use) per direction:
- PCIe Gen3: ~16 GB/s, legacy systems
- PCIe Gen4: ~32 GB/s, A100-era servers
- PCIe Gen5: ~64 GB/s, H100-era servers
- PCIe Gen6: ~128 GB/s, the generation after Gen5
These are theoretical maxima. Real workloads typically achieve 80-90 percent of these numbers. The link is full-duplex, so doubling the per-direction number gives the bidirectional bandwidth. But most workloads have asymmetric traffic, and the per-direction number is what binds them.
What does PCIe carry in a typical inference deployment? It carries the weight load once at startup, and input prompts that arrive from a network card and are copied to GPU memory. It carries output tokens that flow back from GPU memory to the network card. In some long-context configurations, it carries KV cache offload to host memory. It also carries storage I/O for things like LoRA adapter swapping. For single-GPU inference workloads, this is all the inter-chip traffic there is, and PCIe Gen5 is comfortably sufficient.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | PCIe Gen5 x16 | NVLink 4 (H100) | NVLink 5 (B200) |
|---|---|---|---|
| Bandwidth per direction | ~64 GB/s | ~450 GB/s per GPU | ~900 GB/s per GPU |
| Aggregate per GPU | ~128 GB/s (both directions) | ~900 GB/s (18 links) | ~1.8 TB/s (multiple links) |
| Connects | CPU to GPU, peripherals | GPU to GPU | GPU to GPU |
| Available on | All GPUs | H100 SXM (PCIe cards: 600 GB/s bridges) | B200 SXM/HGX |
| Tensor parallelism | Bottleneck | Fits the workload | Comfortable |
Real products, models, and research that use this idea.
- NVIDIA DGX H100 systems use NVLink 4 plus NVSwitch internally for the 8-GPU fabric (900 GB/s GPU-to-GPU) and ConnectX-7 InfiniBand at up to 400 Gb/s between nodes.
- Lambda Cloud, CoreWeave, and other GPU providers list SXM (NVLinked) and PCIe variants of H100 at different prices.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy exactly does tensor parallelism need so much more bandwidth than data parallelism?
Tensor parallelism splits each layer's weights across GPUs, so each forward pass needs an all-reduce of partial activations between GPUs. The all-reduce volume per token scales with hidden dim and batch size. It happens twice per layer (after attention and after the feed-forward block) on every decode step. Data parallelism only synchronizes gradients once per backward pass during training, never during inference.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing memory bandwidth (HBM to compute units, in TB/s on one GPU) with interconnect bandwidth (between GPUs or between CPU and GPU). They cap completely different workloads.
60 second bullets to scan on the way to the call.
Roughly which bus connects CPU to GPU vs GPU to GPU
Bandwidth numbers for PCIe Gen4, Gen5, and Gen6 at x16
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.