Match each multi-GPU parallelism strategy to its defining property

When a model is too big for one GPU, or you want it faster, you split the work across several GPUs. There are four common ways, and they differ by WHAT they split. **Tensor parallelism** splits each individual matrix multiply across GPUs. The weight matrix is cut into shards, each GPU does part of the multiply, and the partial results are combined every layer. It is fast for a single request but needs heavy GPU-to-GPU traffic. Pipeline parallelism splits the model into ranges of layers. GPU 0 holds the first layers, GPU 1 the next, and so on. A token flows down the pipeline like an assembly line. This gives good throughput but the first token is slow because of pipeline startup gaps called bubbles. Expert parallelism only applies to Mixture-of-Experts models. Each GPU holds some of the experts, and tokens are routed to whichever GPU owns the expert they need. Here the hard part is keeping load balanced. Data parallelism does not split the model at all. It copies the whole model onto several GPUs, and each replica serves different requests independently. This scales throughput but does not help a model that does not fit on one GPU.

**Tensor parallelism** shards a single matmul across GPUs. Each device holds a slice of the weight matrix. Every layer needs an all-reduce to recombine partial outputs. That is low latency but bandwidth-hungry, so it wants NVLink or NVSwitch. Pipeline parallelism assigns contiguous layer ranges to different GPUs. Tokens flow stage to stage like an assembly line. It needs far less bandwidth, so it tolerates slow interconnects. The cost is pipeline bubbles that hurt time to first token. Expert parallelism applies only to Mixture-of-Experts models. Each GPU owns a subset of experts. The router sends each token to its chosen experts. Throughput now depends on load balance, not raw compute symmetry. Data parallelism replicates the entire model across GPUs. Each replica serves independent requests. It scales throughput linearly but cannot fit a model that overflows one GPU. These compose. A frontier MoE model on a large node typically runs tensor parallel within a node, expert parallel across the experts, and data parallel across nodes for replica scaling.