Drag each answer to line up with its matching prompt
Tensor parallelism (TP)
Routes MoE experts to different GPUs: essential for MoE; load balance matters more than compute balance
Pipeline parallelism (PP)
Assigns layer RANGES to different GPUs: good throughput, bubble overhead hurts TTFT
Expert parallelism (EP)
Throughput oriented batch serving where pipeline depth >> bubble overhead and inter GPU bandwidth is limited (no NVLink)
Use TP when…
Shards a single matmul (Q, K, V, or MLP projection) across GPUs: low latency, high NVLink/NVSwitch traffic per step
Use PP when…
Latency sensitive single stream serving where every layer must finish fast
Tensor parallelism splits each matmul across GPUs, pipeline parallelism splits layer ranges into stages, expert parallelism shards MoE experts, and data parallelism replicates the whole model across replicas.
Imagine a restaurant with too many orders for one cook. The first way is four cooks chopping the same vegetable together: fast, but they keep passing bowls back and forth. The second way is an assembly line: one fries, the next plates, the next garnishes, so the first dish is slow but many dishes flow steadily. The third way is specialist stations, where each order goes only to the cook who handles that dish, so you must keep every cook equally busy. The fourth way is opening four identical kitchens, each cooking separate orders start to finish. Each layout trades something different: how much the cooks chatter, how fast the first plate appears, and how many plates per hour come out.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-GPU parallelism is the part of inference engineering where candidates most often blur four distinct ideas into one fuzzy notion of "using more GPUs." The interviewer is checking whether you know that tensor parallelism, pipeline parallelism, expert parallelism, and data parallelism split fundamentally different units of work, carry different communication patterns, and sit at different points on the latency versus throughput curve.
The organizing question is always the same: what gets split? Tensor parallelism splits a single matrix multiply. Pipeline parallelism splits the stack of layers into contiguous stages. Expert parallelism splits the experts of a Mixture-of-Experts layer. Data parallelism splits nothing about the model; it replicates the whole thing and splits the request stream instead.
Once you anchor on the unit being split, every other property follows. The communication pattern, the interconnect requirement, the effect on time to first token, and whether the strategy can even help with a model that does not fit on one device all fall out of that one fact. This deep dive walks through each strategy, then shows how production stacks compose them.
Tensor parallelism: splitting the matmul
Tensor parallelism shards an individual matrix multiply across GPUs. Take the MLP up-projection or the combined Q, K, V projection. Its weight matrix is cut column-wise or row-wise, each GPU multiplies the input by its slice, and the partial outputs are recombined.
The standard Megatron scheme is clever about minimizing the recombination. The first MLP matrix is split column-wise so each GPU produces a slice of the hidden activation with no communication, then the second matrix is split row-wise so the outputs become partial sums. Attention is split by heads: each GPU owns a subset of the heads and computes them independently. In both cases the GPUs cannot proceed to the next block until the partials are summed across all of them.
That summation is an all-reduce, and it happens twice per transformer layer, once after attention and once after the MLP. With many layers, that is a relentless stream of collectives sitting directly on the critical path of every token.
This is why tensor parallelism is a low-latency strategy that demands very high bandwidth. All GPUs work the same token in lockstep, so a single request finishes quickly and per-GPU memory drops because each device holds only its shard of the weights. But the per-layer all-reduce only pays off when the interconnect is fast, which in practice means NVLink or NVSwitch inside a single node. Stretch tensor parallelism across nodes on slower links and the all-reduce dominates the step time, which is why degrees above 8 are rare in production.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | Unit split | Communication | Best for |
|---|---|---|---|
| Tensor parallel | Single matmul | All-reduce every layer (high) | Low latency single stream |
| Pipeline parallel | Layer ranges (stages) | Activation handoff per stage (low) | Throughput, slow interconnect |
| Expert parallel | MoE experts | All-to-all token routing | Large Mixture-of-Experts models |
| Data parallel | Nothing (replicates) | None during inference | Scaling throughput once it fits |
Real products, models, and research that use this idea.
- vLLM and SGLang expose tensor_parallel_size and pipeline_parallel_size flags so operators pick the split per node topology in 2026.
- DeepSeek V4 is served with expert parallelism across GPUs because its MoE expert weights dominate the parameter count.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does tensor parallelism need an all-reduce on every layer while pipeline parallelism does not?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the four strategies as interchangeable. They split different units, carry different communication patterns, and the right pick depends on whether you optimize latency or throughput and on inter GPU bandwidth.
60 second bullets to scan on the way to the call.
Which unit each of the four strategies splits across GPUs
Why tensor parallelism demands NVLink or NVSwitch bandwidth
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.