At very long context (1M tokens), the binding constraint on a GPU is the **KV cache**. Each token contributes K and V vectors at every attention layer, so the KV cache size grows linearly with context length. At 1M tokens for a 70B-class model, the KV alone is hundreds of GB, far more than one GPU has.
The question is which parallelism axis actually distributes this KV across GPUs. The four kinds of parallelism each split a different dimension of the model and the data:
- **Tensor parallelism (TP)** splits the model's hidden dimension across GPUs. Each GPU holds a shard of the attention heads and a shard of the FFN matrices. KV cache shrinks proportionally because each GPU only stores K and V for its head shard, but it still scales linearly with context length on each GPU. So TP helps a little, but does not solve long context.
- **Pipeline parallelism (PP)** splits the model by layers. GPU 0 holds layers 0-19, GPU 1 holds layers 20-39, and so on. Each layer's KV lives on the GPU that owns that layer. PP does nothing for the per-GPU KV footprint along the sequence dimension; one GPU still holds the full sequence's K and V for its share of layers.
- **Expert parallelism (EP)** splits MoE experts across GPUs. Only useful for the FFN routing decision in MoE models. Does not touch the KV cache.
- **Sequence parallelism / context parallelism (SP / CP)** splits the **sequence dimension** across GPUs. GPU 0 owns tokens 0 to 250k, GPU 1 owns 250k to 500k, and so on. Each GPU holds the K and V for only its slice of tokens. This is the only parallelism that scales the per-GPU KV with the number of GPUs.
The correct answer is sequence / context parallelism, and in practice production deployments stack it on top of TP (SP + TP) to get the best of both.
**The decisive axis is the one that grows with context length.** KV cache size = `2 * B * L * N_layers * H_kv * d_head * bytes`. The terms TP shrinks (H_kv via head-sharding) are not the binding term at 1M context; L is. The only parallelism that splits L is sequence / context parallelism.
**How SP / CP works.** Each GPU owns a contiguous slice of the sequence and holds K and V for only that slice. During the attention op, attention is global across all tokens, so the K and V slices must be exchanged. Two common communication patterns:
- **All-to-all**: every GPU sends its KV to every other GPU during attention, then results are reduced. Simple but bandwidth-heavy.
- **Ring attention**: KV slices rotate around the GPUs in a ring, with each GPU computing partial attention on the slice it currently holds, then rotating to the next. Bandwidth-efficient at long context, this is the dominant pattern in production.
**Why TP alone is not enough.** TP shrinks KV per GPU by the TP-degree (the number of K-heads each rank holds), but the KV per GPU still scales with the full sequence length. At 1M context that scaling overwhelms any TP-degree you can practically run on a single node.
**Why PP and EP do not help.** PP splits layers, not sequence; each GPU still holds the full sequence's KV for its share of layers. EP only routes MoE experts and does not touch attention.
**Production stacking.** Real long-context deployments combine TP + SP (sometimes called 'sequence parallel TP' or 'context parallel'). TP handles the model-size dimension, SP handles the sequence dimension, and the per-GPU footprint becomes manageable. Megatron-LM, DeepSpeed, and recent serving frameworks all support this stack. Adding pipeline parallel on top is sometimes useful for very large models, but PP is orthogonal to the long-context KV problem itself.
Quick reference
TL;DR
Only sequence / context parallelism splits along the dimension that grows with context length, so at 1M tokens it is the parallelism axis that actually shrinks the per-GPU KV cache. TP, PP, and EP shard other dimensions.
Think about which way you slice a giant pizza. The cached conversation history at 1M words is a pizza that is enormous along one specific direction: how many words long it is. Splitting the math across GPUs by columns is like slicing the pizza into thinner and thinner radial wedges; that helps with width but not length. Splitting by stacking layers across GPUs is like piling pizzas in a tower (one per layer); it does not split any single pizza at all. The right move is to cut the pizza crosswise along its length, so each diner gets a shorter section of the long pizza. Splitting a long input across helpers is the only cut that reduces how much of the long axis any one diner has to hold.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short context (under 16k tokens) TP alone or even single-GPU serving is sufficient; the SP communication overhead is not worth it.
Decode at batch 1 with long context but no concurrency SP introduces per-step communication that can hurt single-stream latency. For very latency-sensitive single-user long-context decode, smaller-model alternatives or MLA may be preferable.
MoE models where expert parallelism is already saturating the inter-GPU bandwidth Layering SP on top of an already communication-heavy EP topology can lead to bandwidth contention. Profile carefully before stacking.
Quick scan · 2–4 min
Everything important, quickly.
30 sec: KV cache shape and which axis is too big at 1M (it is L). 1 min: each parallelism splits a different axis (TP heads, PP layers, EP experts, SP sequence). 1 min: only SP / CP attacks L. 1 min: explain ring attention or all to all communication during the attention op. 30 sec: production stack is TP + SP.
Side-by-side
Parallelism axis
What it shards
KV-cache effect at long context
When it helps
Tensor (TP)
Heads, hidden dim, projection matrices
Constant-factor shrink (by TP degree); does not scale with L
Model too large for one GPU; modest context
Pipeline (PP)
Layers (depth)
No help; each layer's KV still full sequence on its GPU
Very deep models; weight budget bottleneck
Expert (EP)
MoE FFN experts
Zero effect on attention KV
MoE FFN distribution only
Sequence / Context (SP / CP)
Token sequence dimension (L)
Linear shrink in L; scales with context length
Long context (the canonical answer)
Data (DP)
Independent samples across replicas
Zero effect on per-sample KV
Throughput scaling, not memory
Real-world examples
Real products, models, and research that use this idea.
1Llama-3.1's 128k context support was enabled by training with context parallelism in Megatron-LM, not by tensor parallelism alone.
2Ring Attention (Liu, Yan, Abbeel 2023) demonstrated 1M+ context training on Llama-class models by ring-rotating KV blocks across GPUs.
3DeepSpeed-Ulysses provides an all to all sequence-parallel attention implementation used in long-context training pipelines.
4Gemini 1.5 / 2 with 1M to 2M context is widely believed to use ring-style attention across many TPU pods.
5Anthropic's Claude long-context handling at 200k+ context likely uses some form of sequence partitioning in serving; the exact mechanism is undisclosed.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf a 70B-class model has KV cache of ~327 KB/token, what topology serves it at 1M context with reasonable headroom?
Sign in or go Pro to see the approach for this follow-up.
QWhy is ring attention bandwidth-efficient compared to all to all sequence parallel?
Sign in or go Pro to see the approach for this follow-up.
QDoes sequence parallelism help with prefill too, or only decode?
Sign in or go Pro to see the approach for this follow-up.
QHow does MLA (DeepSeek's multi-head latent attention) change the long-context picture?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing tensor parallelism solves long-context memory. TP shards heads and projection weights, not the sequence dimension. At long context the KV per TP rank still scales linearly with context length and runs out.