Tensor Parallelism
Also known as: TP, Megatron-style parallelism
Split each layer's tensors across GPUs and synchronize after each op; enables single-model multi-GPU inference.
A model-parallel strategy that splits individual layer tensors (e.g. attention heads, MLP rows) across multiple GPUs, with collective communication after each split operation. Enables serving models too large for one GPU.
In practice
How 70B+ models fit on one node. Senior infra interviews probe vs pipeline parallelism and the all-reduce cost.
How it compares
Tensor parallelism splits within a layer (high comms); pipeline parallelism splits across layers (lower comms, bubble overhead).
Related topics
Questions that mention this term
- NVLink vs PCIe: match each interconnect to its bandwidth tier in a modern multi-GPU server.
- Prefill and decode: name the two phases of LLM inference and say which one is compute bound
- Match each multi-GPU parallelism strategy to its defining property
- Walk through how tensor…
- Compare tensor, pipeline and expert parallelism for inference serving.
- Match multi-GPU training strategy to what it shards
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.