Describe how tensor parallelism splits a single weight matrix across GPUs, what communication operation is needed to reassemble the output, and the concrete cost this imposes on each decode step. State the interconnect requirement that makes or breaks the strategy.
Free · 2 AI evals / day
Tensor parallelism is a way to run one model across several GPUs by splitting each weight matrix among them. Instead of every GPU holding a full copy of the layer, each GPU holds only a slice of the rows or columns and does the math on its slice.
The standard recipe inside a transformer block uses two splits in a row. The first linear layer is sharded by columns, so every GPU produces a vertical band of the output activation. The second linear layer is sharded by rows, so every GPU naturally consumes its matching band and produces a partial sum of the final output. Those partial sums then have to be added together across the GPUs to recover the true result.
The operation that does the adding is called an all-reduce. It fires once for each transformer block in the model, which means once for every layer of every forward pass. During decode, where the model produces one token at a time, that all-reduce runs across all the GPUs in the tensor parallel group for every single output token.
The trade is straightforward. You shrink the weights each GPU has to read from memory by the number of GPUs, which directly speeds up bandwidth-bound decode. In exchange you pay a small communication cost per layer. That trade is only profitable when the GPUs are connected by a very fast fabric like NVLink, which sits inside a single server. Across slower links the all-reduce becomes the slowest step and tensor parallelism turns into a loss.
Tensor parallelism splits each weight matrix across the GPUs in the tensor parallel group, so every device computes on a slice and holds only a slice in HBM. Inside a transformer block the standard pattern is column-then-row sharding. The first projection (the up-projection in the MLP, or the QKV projection in attention) is split column-wise: each GPU computes a vertical band of the intermediate activation independently, no communication required. The second projection (the down-projection, or the output projection) is split row-wise: each GPU multiplies its matching column-band of the intermediate against its row-band of the weights and produces a partial sum of the layer output.
An all-reduce then sums those partial sums across the group so every GPU ends up holding the full activation, ready for the next layer's residual add and normalization. That all-reduce fires once per transformer block. During decode, where one forward pass produces one output token, the model runs through every layer in series, so the per-token cost is N_layers all-reduces back to back.
The gain is real. Each GPU now reads only 1/N of the weight matrix from HBM per layer, which is the dominant cost of bandwidth-bound decode. So tokens-per-second scales nearly linearly with the tensor parallel size, up to the point where the all-reduce starts to bite.
When it bites depends entirely on the interconnect. On NVLink and NVSwitch inside an 8-GPU node you get roughly 900 GB/s between GPUs, which is fast enough that the all-reduce hides under the weight read and TP=8 typically nets 6-7x speedup on decode. On PCIe at roughly 64 GB/s the all-reduce dominates every layer and TP becomes a net loss even at TP=2. Across nodes the gap is worse. That is why tensor parallelism is almost always capped at the 8-GPU NVLink island and longer-range scaling moves to pipeline or expert parallelism, which need far less cross-GPU bandwidth.
Quick reference
TL;DR
Tensor parallelism shards each weight matrix column-wise then row-wise across N GPUs; an all-reduce after every layer sums the partial outputs, paid once per layer per decode step, and only pays off on NVLink-class
Imagine a long math problem split across four friends. The first friend does the left quarter of the multiplication, the second friend the next quarter, and so on. Each friend ends up with only a piece of the answer. Before they can do the next problem together, they have to pass their pieces around and add them up so everyone has the full answer. If the friends are sitting at the same table and can hand papers across instantly, the sharing is cheap. If they are in different buildings and have to mail letters, the sharing takes longer than the math itself. That is the trick behind splitting one giant multiplication across several GPUs, and it is the reason this only pays off when the GPUs sit next to each other on a fast fabric like NVLink.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single small model that fits comfortably on one GPU with bandwidth headroom There is no capacity pressure and no decode-bandwidth saving big enough to amortize the all-reduce; TP=1 wins.
GPUs connected only over PCIe or across separate hosts The per-layer all-reduce latency dominates and erases the weight-bandwidth saving; the workload runs slower than TP=1.
Scaling past the local NVLink island for very large models Crossing nodes pushes the all-reduce onto InfiniBand or Ethernet, where round-trips multiply across layers; pipeline parallelism is the right primitive between nodes.
MoE models where expert parallelism already gives a sparse-routing win Sharding experts across GPUs (EP) needs far less per-layer bandwidth than TP; mixing them within an island is fine but EP alone often suffices.
Quick scan · 2–4 min
Everything important, quickly.
4 min: column-then-row sharding inside the block, why the first matmul needs no communication, the all-reduce closing each block, the per-layer per-token cadence, and the interconnect-class binary that decides whether TP wins or loses.
Side-by-side
Aspect
Tensor parallelism
Pipeline parallelism
Data parallelism
What is sharded
Each weight matrix (intra-layer)
Layer groups across stages
Batch across replicas
Communication primitive
All-reduce per layer
Point-to-point at stage boundary
All-reduce of gradients (training only)
Communication frequency
Every layer every forward pass
Once per pipeline stage
Once per training step
Bandwidth need
Very high (NVLink class)
Modest (Ethernet OK)
Modest
Decode latency impact
Each token waits on N_layers reductions
Adds pipeline-fill latency at batch 1
No effect on single-request latency
Typical scope
Inside one node
Across nodes
Across replicas for throughput
Real-world examples
Real products, models, and research that use this idea.
1Megatron-LM introduced and standardized the column-then-row sharding pattern that every modern serving stack inherits.
2vLLM, TensorRT-LLM, and SGLang all expose --tensor-parallel-size as their canonical multi-GPU knob and cap it at the local NVLink group by default.
3Llama 3 70B is typically served at TP=4 or TP=8 on an H100/H200 NVLink node; the same model on PCIe-only servers runs noticeably slower at TP=2 than at TP=1.
4NVIDIA GB200 NVL72 extends the NVLink island from 8 GPUs to 72, letting frontier models like Llama 4 Maverick or DeepSeek V4 run with much larger tensor-parallel groups while staying NVLink-resident.
5DistServe and similar disaggregated serving stacks keep TP inside each prefill or decode pool and use lighter-weight pipeline parallelism between pools to avoid stretching the all-reduce across the network.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the column-then-row pattern need exactly one all-reduce per block instead of two?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes for the attention block versus the MLP block?
Sign in or go Pro to see the approach for this follow-up.
QHow does TP combine with FP8 or FP4 weights to multiply the decode win?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating tensor parallelism as a free capacity expander. The cross-GPU all-reduce fires every layer every decode step, so on PCIe-only or cross-node links it costs more wall-clock than the weight-bandwidth saving buys you.