Ring attention shards the sequence across GPUs. Q stays put; K, V chunks circulate around the ring and partial attention composes via online softmax.
Picture a million word book that no single reader can hold all at once. Handing each reader a different book does not help, because the task is to understand this one book. Sit the readers in a circle and split the book into chunks, one chunk per reader. Each reader keeps their own chunk in their lap and starts passing extra chunks around the circle. As a chunk drifts past, every reader compares it to the part in their lap and adds a few notes. After the chunks have travelled all the way around, each reader has compared their chunk to the whole book, and their notes stitch together into a full understanding.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Ring attention is the canonical implementation of sequence parallelism, the fourth parallelism axis in modern LLM training and the only one that helps when a single training sequence does not fit on a single GPU. It is the technique behind much of the million token training work shipped from 2024 onward, including Gemini class long context pipelines and the wave of frontier models that exposed 1M-token context windows in 2025-2026.
The sections below explain why the other parallelism axes fail to address long context, walk the ring algorithm step by step, dig into the online softmax composition that makes correctness work, show how ring composes with tensor and pipeline parallelism, explain the crossover regime where ring overhead is worth it, and discuss the causal mask load balancing issue that Striped Attention fixes.
Why the other parallelism axes do not help
Data parallelism gives every GPU a different sample, but each GPU still holds a full sequence. That is useless when the problem is that one sequence is too big.
Tensor parallelism (Megatron style) shards the hidden dimensions. It helps with parameter and activation size scaling along d, splitting a 4096-wide projection across 8 cards into 512-wide slices. But it does not fundamentally bound memory by sequence length; each shard still sees the full n in its slice of d.
Pipeline parallelism splits layers across stages. The peak memory profile improves a little because each stage only holds its own activations, but at the moment each stage runs, it still needs the full sequence for its layers. Bubble overhead also hurts long context throughput because of the time skew between stages.
The missing axis is sequence parallelism: shard the n dimension across devices so each device only holds n/N tokens. Ring attention is the canonical attention layer implementation, and the FFN layers (which are pointwise in n) parallelize trivially across the same shard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Parallelism | Shards across | Memory benefit | Helps long context? |
|---|---|---|---|
| Data parallelism | Batch samples | None per-sequence | No |
| Tensor parallelism | Hidden dimensions | Activation/param weight scaling | Partial (constants) |
| Pipeline parallelism | Transformer layers | Layer-wise weights | Partial (depth) |
| Sequence (ring attention) | Sequence positions | Activations O(n/N) | Yes, primary axis |
Real products, models, and research that use this idea.
- Gemini 3.1 Pro's million token context is widely understood to use ring attention style sequence parallelism in training.
- DeepSpeed Ulysses implements sequence parallelism with similar ideas.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does ring attention specifically require FlashAttention's online softmax?
Each GPU computes attention between its Q_i and a different K_j, V_j at each step. These partial softmax results have different per-row maxima and denominators. Online softmax maintains running max and denominator across iterations, rescaling the previous partial output by exp(old_max - new_max) when a new max appears. Without it, you'd have to either materialize the full attention matrix or accept a different (non-equivalent) approximation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying ring attention 'speeds up' attention or replaces softmax, it does neither. It enables sequences too long for one GPU by sharding sequence dimension, with the same exact math.
60 second bullets to scan on the way to the call.
Place sequence parallelism alongside data, tensor, and pipeline parallelism
The ring rotation: K and V circulate, Q stays local
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.