Same topic, related formats. Practice these next.
Tensor parallelism splits each matmul across GPUs, pipeline parallelism splits layer ranges into stages, expert parallelism shards MoE experts, and data parallelism replicates the whole model across replicas.
Imagine a restaurant with too many orders for one cook. The first way is four cooks chopping the same vegetable together: fast, but they keep passing bowls back and forth. The second way is an assembly line: one fries, the next plates, the next garnishes, so the first dish is slow but many dishes flow steadily. The third way is specialist stations, where each order goes only to the cook who handles that dish, so you must keep every cook equally busy. The fourth way is opening four identical kitchens, each cooking separate orders start to finish. Each layout trades something different: how much the cooks chatter, how fast the first plate appears, and how many plates per hour come out.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: name the unit each strategy splits, then the communication pattern of each, then latency versus throughput posture, then how a frontier MoE stack composes all four.
| Strategy | Unit split | Communication | Best for |
|---|---|---|---|
| Tensor parallel | Single matmul | All-reduce every layer (high) | Low latency single stream |
| Pipeline parallel | Layer ranges (stages) | Activation handoff per stage (low) | Throughput, slow interconnect |
| Expert parallel | MoE experts | All-to-all token routing | Large Mixture-of-Experts models |
| Data parallel | Nothing (replicates) | None during inference | Scaling throughput once it fits |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating the four strategies as interchangeable. They split different units, carry different communication patterns, and the right pick depends on whether you optimize latency or throughput and on inter GPU bandwidth.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.