Pipeline Parallelism
Also known as: PP
Split the model across GPUs by layer groups, pipeline micro-batches through them; lower bandwidth needs than TP.
A model-parallel strategy that places consecutive groups of layers on different GPUs and pipelines micro-batches through them. Lower communication than tensor parallelism but introduces pipeline bubbles.
In practice
Standard with TP for multi-node serving. Senior interviews probe the bubble math and GPipe/PipeDream variants.
How it compares
Pipeline parallelism splits across layers (whole layer per GPU); tensor parallelism splits within each layer.
Related topics
Questions that mention this term
- Your indexing pipeline tokenizes 10M docs in 18 hours. What single change gives the biggest speedup?
- Lay out the audio front end for transcribing hour long sales calls at scale
- Match each multi-GPU parallelism strategy to its defining property
- Compare tensor, pipeline and expert parallelism for inference serving.
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.