For 1M-token context decode, which parallelism axis actually relieves KV pressure?

At very long context (1M tokens), the binding constraint on a GPU is the **KV cache**. Each token contributes K and V vectors at every attention layer, so the KV cache size grows linearly with context length. At 1M tokens for a 70B-class model, the KV alone is hundreds of GB, far more than one GPU has. The question is which parallelism axis actually distributes this KV across GPUs. The four kinds of parallelism each split a different dimension of the model and the data: - **Tensor parallelism (TP)** splits the model's hidden dimension across GPUs. Each GPU holds a shard of the attention heads and a shard of the FFN matrices. KV cache shrinks proportionally because each GPU only stores K and V for its head shard, but it still scales linearly with context length on each GPU. So TP helps a little, but does not solve long context. - **Pipeline parallelism (PP)** splits the model by layers. GPU 0 holds layers 0-19, GPU 1 holds layers 20-39, and so on. Each layer's KV lives on the GPU that owns that layer. PP does nothing for the per-GPU KV footprint along the sequence dimension; one GPU still holds the full sequence's K and V for its share of layers. - **Expert parallelism (EP)** splits MoE experts across GPUs. Only useful for the FFN routing decision in MoE models. Does not touch the KV cache. - **Sequence parallelism / context parallelism (SP / CP)** splits the **sequence dimension** across GPUs. GPU 0 owns tokens 0 to 250k, GPU 1 owns 250k to 500k, and so on. Each GPU holds the K and V for only its slice of tokens. This is the only parallelism that scales the per-GPU KV with the number of GPUs. The correct answer is sequence / context parallelism, and in practice production deployments stack it on top of TP (SP + TP) to get the best of both.

**The decisive axis is the one that grows with context length.** KV cache size = `2 * B * L * N_layers * H_kv * d_head * bytes`. The terms TP shrinks (H_kv via head-sharding) are not the binding term at 1M context; L is. The only parallelism that splits L is sequence / context parallelism. **How SP / CP works.** Each GPU owns a contiguous slice of the sequence and holds K and V for only that slice. During the attention op, attention is global across all tokens, so the K and V slices must be exchanged. Two common communication patterns: - **All-to-all**: every GPU sends its KV to every other GPU during attention, then results are reduced. Simple but bandwidth-heavy. - **Ring attention**: KV slices rotate around the GPUs in a ring, with each GPU computing partial attention on the slice it currently holds, then rotating to the next. Bandwidth-efficient at long context, this is the dominant pattern in production. **Why TP alone is not enough.** TP shrinks KV per GPU by the TP-degree (the number of K-heads each rank holds), but the KV per GPU still scales with the full sequence length. At 1M context that scaling overwhelms any TP-degree you can practically run on a single node. **Why PP and EP do not help.** PP splits layers, not sequence; each GPU still holds the full sequence's KV for its share of layers. EP only routes MoE experts and does not touch attention. **Production stacking.** Real long-context deployments combine TP + SP (sometimes called 'sequence parallel TP' or 'context parallel'). TP handles the model-size dimension, SP handles the sequence dimension, and the per-GPU footprint becomes manageable. Megatron-LM, DeepSpeed, and recent serving frameworks all support this stack. Adding pipeline parallel on top is sometimes useful for very large models, but PP is orthogonal to the long-context KV problem itself.