Compare tensor parallelism, pipeline parallelism, and expert parallelism along three axes: what they shard, their inter GPU communication pattern per token, and what they cost in TTFT vs throughput. For each, state when you would deploy it.
Tensor parallel shards each matmul and cuts latency but needs NVLink; pipeline parallel splits layer ranges for throughput with a bubble tax; expert parallel routes mixture-of-experts experts and lives on load balance.
Imagine a kitchen too big for one cook, so you split the work three ways. The first way: several cooks chop the same vegetable together, then compare piles every few seconds. They finish fast, but only if they stand close enough to pass bowls instantly. The second way is an assembly line: one cook preps, the next fries, the next plates. The line hums once it fills, but the first plate is slow because each station waits for the one before it. The third way is a food court where different stalls make different dishes, and a host sends each order to the right stall. It works great until everyone wants pizza, so one stall is swamped while the rest sit idle. Real kitchens mix all three to keep every cook busy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When a model outgrows a single GPU, you must split it. But splitting is not one decision; it is three orthogonal axes, and serious interview answers grade each axis on what it shards, how the GPUs communicate per token, and what it sacrifices. The wrong split turns a fast model into a slow one, or a cheap deployment into one that burns idle GPUs.
The three axes are tensor parallelism, pipeline parallelism, and expert parallelism. They are not competing answers to the same question; they trade away different scarce resources. Tensor parallelism spends interconnect bandwidth to buy latency. Pipeline parallelism accepts a startup bubble to buy throughput. Expert parallelism tolerates routing imbalance to fit a mixture-of-experts model that otherwise will not load.
This deep dive walks through each axis, its communication signature, and its cost. Then it shows how production stacks compose all three into a 3-D scheme, with tensor parallel inside a node, pipeline parallel across nodes, and expert parallel layered in for mixture-of-experts models. By the end you should be able to look at a model size, an interconnect topology, and a latency target, and name the right parallel configuration on a whiteboard.
Tensor parallelism: shard the matmul, pay the all-reduce
Tensor parallelism splits a single weight matrix across GPUs. For attention you shard by head, giving each GPU a subset of the heads to compute. For the feed-forward block you shard the hidden dimension, so the first matmul projects into a sharded intermediate and the second projects back. Each GPU computes a partial output for the same layer, then the GPUs combine partials with a collective.
The defining cost is that combination. A column-parallel matmul followed by a row-parallel matmul ends each transformer block with an all-reduce over the participating GPUs. You pay that collective once per layer, dozens of times per forward pass, for every token. The volume per all-reduce scales with the hidden size and the batch, so the traffic is substantial and recurs constantly. Crucially this is on the critical path: the next layer cannot start until the all-reduce completes, so the collective latency adds directly to per-token decode time.
That communication pattern is why tensor parallelism lives and dies on interconnect. Inside an NVLink or NVSwitch node the bandwidth is high enough that the all-reduce overlaps cheaply with compute. Push the same scheme over PCIe or Ethernet and the all-reduce dominates the step, so the GPUs spend more time talking than computing.
The practical ceiling is roughly TP=8 within an NVLink node and TP=4 over PCIe, and tensor parallel across nodes is a mistake for any latency-sensitive workload. The upside is real: tensor parallelism is the only axis that lowers single-stream latency, because it puts every GPU to work on the very same token rather than on different tokens or different layers. That is exactly what an interactive endpoint needs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | Shards what | Communication per token | Latency vs throughput | Deploy when |
|---|---|---|---|---|
| Tensor parallel | Each matmul (head or feature dim) | All-reduce every layer (heavy) | Lowest latency; caps near TP=8 | Latency-sensitive serving inside an NVLink node |
| Pipeline parallel | Contiguous layer ranges | Activation handoff at stage boundary (light) | High throughput; bubble hurts TTFT | Batch or offline, or the outer axis across nodes |
| Expert parallel | MoE experts across GPUs | All-to-all dispatch by routing | Fits MoE; load balance sets the ceiling | Any mixture-of-experts model |
Real products, models, and research that use this idea.
- DeepSeek V4 serves its mixture-of-experts model with combined tensor and expert parallelism, using expert-parallel all-to-all dispatch across GPUs.
- vLLM and SGLang both expose tensor-parallel size and pipeline-parallel size as deployment flags, defaulting to tensor parallel within an NVLink node.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does tensor parallelism cap out around TP=8 even with NVLink?
Each layer's all-reduce volume scales with hidden size, and you pay it once per layer per token. As the GPU count rises, the all-reduce latency and the participant count grow, so communication eats an ever larger share of the step. NVLink raises the ceiling but does not remove the linear growth.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the three as interchangeable speedups. They optimize different things: tensor parallel trades bandwidth for latency, pipeline parallel trades a startup bubble for throughput, and expert parallel trades balance to fit experts in memory.
60 second bullets to scan on the way to the call.
What each strategy shards: matmul, layer range, or expert set
Per-token communication pattern for each of the three axes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.