Tensor Parallelism
Also known as: TP, Megatron-style parallelism
Split each layer's tensors across GPUs and synchronize after each op; enables single-model multi-GPU inference.
A model-parallel strategy that splits individual layer tensors (e.g. attention heads, MLP rows) across multiple GPUs, with collective communication after each split operation. Enables serving models too large for one GPU.
In practice
How 70B+ models fit on one node. Senior infra interviews probe vs pipeline parallelism and the all-reduce cost.
How it compares
Tensor parallelism splits within a layer (high comms); pipeline parallelism splits across layers (lower comms, bubble overhead).
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.