Pipeline Parallelism
Also known as: PP
Split the model across GPUs by layer groups, pipeline micro-batches through them; lower bandwidth needs than TP.
A model-parallel strategy that places consecutive groups of layers on different GPUs and pipelines micro-batches through them. Lower communication than tensor parallelism but introduces pipeline bubbles.
In practice
Standard with TP for multi-node serving. Senior interviews probe the bubble math and GPipe/PipeDream variants.
How it compares
Pipeline parallelism splits across layers (whole layer per GPU); tensor parallelism splits within each layer.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.