Inside a typical AI server you have two completely different kinds of bandwidth, and senior engineers care about both. **Memory bandwidth** is how fast bytes move between a GPU and its own HBM, measured in TB/s. **Interconnect bandwidth** is how fast bytes move between chips: CPU to GPU, or GPU to GPU. That second category is where NVLink and PCIe show up.
**PCIe** is the general-purpose CPU-to-GPU bus that every server has. A modern PCIe Gen5 x16 link delivers about 64 GB/s per direction (so 128 GB/s bidirectional). This is plenty for streaming weights into a GPU once at startup, pulling input batches in, and sending output tokens back to the CPU. It is also how the GPU talks to the rest of the system: storage, network cards, the host operating system.
**NVLink** is NVIDIA's purpose-built GPU-to-GPU fabric. On an H100 SXM card, NVLink 4 delivers about 900 GB/s aggregate per card (split across multiple links to neighboring GPUs). On the newer B200, NVLink 5 reaches roughly 1.8 TB/s. That is more than 10x what PCIe Gen5 offers, and it exists for one reason: when you split a large model across multiple GPUs, those GPUs need to exchange activation tensors at the speed of compute, not at the speed of the storage bus.
**PCIe** is the standardized CPU-to-GPU and peripheral bus. Generation 5 at x16 width gives roughly 64 GB/s per direction (~128 GB/s bidirectional). It connects every GPU to the host CPU, system memory, network cards, and storage. Single-GPU inference workloads mostly only need PCIe: weights load once over PCIe at startup, inputs and outputs flow across it per request, and that is the end of the story.
**NVLink** is NVIDIA's proprietary GPU-to-GPU interconnect. The generations roughly track GPU generations: NVLink 3 on A100 SXM at ~600 GB/s aggregate, NVLink 4 on H100 SXM at ~900 GB/s, NVLink 5 on B200 at ~1.8 TB/s. NVLink only exists on SXM-form-factor cards; PCIe-form-factor variants of the same GPU silicon either omit NVLink entirely or expose only a limited subset. NVSwitch chips inside DGX and HGX boxes wire all NVLinks into a fully-connected fabric so any GPU can talk to any other at full NVLink speed.
The workload that cares most about this distinction is **tensor parallelism**. When a model is too big to fit on one GPU, you split each weight matrix across multiple GPUs. Every layer's forward pass requires an all-reduce of partial results, which is a collective communication pattern with bandwidth proportional to (model dim) x (batch x sequence). On NVLink this all-reduce takes microseconds. On PCIe it takes milliseconds, dominating decode latency and erasing the throughput gain from parallelism. This is why production large-model serving uses NVLink-connected GPUs and not PCIe-bridged ones.
Quick reference
TL;DR
NVLink is the fast GPU-to-GPU fabric (H100 SXM ~900 GB/s, B200 ~1.8 TB/s); PCIe is the slower CPU-to-GPU bus (Gen5 x16 ~64 GB/s per direction).
Imagine a kitchen with several chefs. Each chef has a private station, and they all need ingredients from the pantry. The hallway between the pantry and the stations is the regular bus that everyone uses. Now imagine the chefs sometimes need to pass ingredients directly to each other to finish a dish together. If the chefs are connected by a tiny door between stations, ingredients fly across in seconds. If they have to walk all the way back through the pantry hallway to hand each other things, the dish takes ten times longer. NVLink is the tiny door between GPU stations. PCIe is the hallway. Workloads that need GPU-to-GPU chatter cook fast on NVLink and slowly on PCIe.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-GPU inference on a model that fits in one HBM No inter-GPU traffic is needed; PCIe is plenty for the once per startup weight load and per-request input/output.
Embarrassingly parallel data-parallel inference with no parameter synchronization Each GPU runs independently after weight load; interconnect bandwidth never binds, so PCIe is fine.
Pipeline-parallel inference where activations flow between stages bursting Pipeline traffic is less constant than tensor-parallel all-reduce, so PCIe can sometimes carry it, though NVLink is always nicer.
Quick scan · 2–4 min
Everything important, quickly.
3 min: PCIe vs NVLink role + per-generation bandwidth numbers + tensor-parallelism all-reduce as the canonical NVLink-bound workload + NVSwitch fabric inside DGX + why PCIe-form-factor GPUs are unsuitable for tensor parallelism.
Side-by-side
Property
PCIe Gen5 x16
NVLink 4 (H100)
NVLink 5 (B200)
Bandwidth per direction
~64 GB/s
~450 GB/s per GPU pair
~900 GB/s per GPU pair
Aggregate per GPU
~64 GB/s (one link)
~900 GB/s (multiple links)
~1.8 TB/s (multiple links)
Connects
CPU to GPU, peripherals
GPU to GPU
GPU to GPU
Available on
All GPUs
SXM-form H100 only
B200 SXM/HGX
Tensor parallelism
Bottleneck
Fits the workload
Comfortable
Real-world examples
Real products, models, and research that use this idea.
1NVIDIA DGX H100 systems use NVLink 4 plus NVSwitch internally for the 8-GPU fabric and 400 Gb/s InfiniBand between nodes.
2Lambda Cloud, CoreWeave, and other GPU providers explicitly differentiate SXM (NVLinked) and PCIe variants of H100 at different price points.
3Llama 4 Maverick inference at production scale uses 8-way tensor parallelism within a single NVLink-connected node to stay under the bandwidth wall.
4Grace Hopper GH200 systems wire a Grace CPU directly to an H100 GPU over NVLink-C2C at ~900 GB/s, eliminating the PCIe bottleneck for CPU/GPU data movement.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy exactly does tensor parallelism need so much more bandwidth than data parallelism?
Sign in or go Pro to see the approach for this follow-up.
QHow does NVLink-C2C in Grace Hopper change the inference architecture story?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the bandwidth picture look like across a multi-node cluster, not just within one DGX?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing memory bandwidth (HBM to compute units, in TB/s on one GPU) with interconnect bandwidth (between GPUs or between CPU and GPU). They cap completely different workloads.