Compare tensor, pipeline and expert parallelism for inference serving.

A frontier model is often too big for one GPU, so you split it across several. There are three main ways to split, and they differ in what they cut up and how much the GPUs have to talk to each other. **Tensor parallelism** cuts each layer's matrix multiply into slices, one per GPU. Every GPU computes part of the same layer, then they combine results. Because they combine on every single layer, they must exchange data constantly, so they need a very fast link like NVLink. The payoff is low latency for one request. Pipeline parallelism instead gives each GPU a different range of layers, like stations on an assembly line. A token passes from station to station. The GPUs only talk at the handoff between stages, so the link can be slower. The downside is that the start of generation is slow while the pipeline fills up. Expert parallelism only applies to mixture-of-experts models, which contain many specialist sub-networks. You put different experts on different GPUs and route each token to the few experts it needs. The main risk is imbalance: if most tokens pick the same expert, that GPU is swamped while the rest idle. In production you usually combine these rather than pick one.

These three strategies shard a model along different axes, and each buys a different thing. **Tensor parallelism** splits each individual matmul across GPUs, by attention head or feature dimension. Every transformer layer ends with an all-reduce to recombine partial outputs. That means heavy per-layer traffic, so it needs NVLink or NVSwitch. The reward is the lowest single-stream latency, since work inside a layer runs in parallel. Returns fade past TP=8 on NVLink because communication grows with the GPU count. Pipeline parallelism assigns layer ranges to different GPUs, communicating only at stage boundaries. That traffic is light, so it tolerates slower cross-node links. It raises aggregate throughput at large batch sizes. The cost is the pipeline bubble: idle stages while the pipe fills and drains, which hurts time to first token. Expert parallelism applies only to mixture-of-experts models. It places different experts on different GPUs and routes each token to its chosen few. The cost is not bandwidth but load balance. Skewed routing leaves some GPUs idle while others saturate. In practice you combine them. Dense 70B-plus models use tensor parallel inside a node and pipeline parallel across nodes. MoE models like DeepSeek V4 pair tensor with expert parallelism.