Modern LLM training uses several kinds of parallelism. Data parallelism gives every GPU a copy of the model. Tensor parallelism shards hidden dimensions. Pipeline parallelism splits layers across GPUs.
None addresses the long context problem. Q, K, V activations scale as `n × d_model × heads × bytes` per layer. At n = 1M tokens, that exceeds 80 GB. Data parallelism does not help: every GPU still tries to hold a full sequence. The problem is that one sequence is too big for one device.
**Ring attention** targets this case. Arrange N GPUs in a ring; split the sequence into N chunks. GPU i holds `Q_i, K_i, V_i`, computes attention against its current K, V chunk, then passes that chunk to the next GPU. After N-1 rotations every Q has seen every K, V, and FlashAttention's online softmax composes partials into a bit exact final result.
Activation memory per GPU drops from `O(n)` to `O(n/N)`, and communication hides behind compute when the ring is balanced. This is the missing parallelism axis behind million token training.
Ring attention does not speed gradient sync, replace softmax, or eliminate tensor parallelism. It is orthogonal sequence parallelism.
**Ring attention** shards the sequence dimension across GPUs arranged in a logical ring. Each GPU keeps Q for its slice while K, V chunks circulate around the ring. At every step the local Q meets a fresh K, V pair, and FlashAttention's online softmax composes the partial outputs into a bit exact final result.
Activation memory per GPU drops from `O(n)` to `O(n/N)`. That is exactly what is needed when a single training sequence is too long for a single device, the case where data parallelism (which only replicates the model) is no help. Ring attention sits alongside data, tensor, and pipeline parallelism as the fourth axis (sequence parallelism), and production stacks combine all four for million token training.
The correctness argument depends on the online softmax. Each rotation rescales the running output by `exp(old_max - new_max)` when a new per-row maximum appears, so the final result equals a single pass softmax up to FP reduction order. Striped scheduling fixes the causal mask load imbalance issue that naive ring attention has. Megatron-LM context parallelism, DeepSpeed Ulysses, and Gemini class long context training all implement variants of this design.
Quick reference
TL;DR
Ring attention shards the sequence across GPUs. Q stays put; K, V chunks circulate around the ring and partial attention composes via online softmax.
Picture a million word book that no single reader can hold all at once. Handing each reader a different book does not help, because the task is to understand this one book. Sit the readers in a circle and split the book into chunks, one chunk per reader. Each reader keeps their own chunk in their lap and starts passing extra chunks around the circle. As a chunk drifts past, every reader compares it to the part in their lap and adds a few notes. After the chunks have travelled all the way around, each reader has compared their chunk to the whole book, and their notes stitch together into a full understanding.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short context where a single sequence already fits comfortably on one GPU Ring rotation communication is pure overhead at small n; per-step compute does not hide the inter-GPU traffic and single-GPU FlashAttention is faster end to end.
Bandwidth starved interconnects (no NVLink, slow Ethernet) The ring depends on tight K, V transfer between adjacent GPUs each step; on weak links the rotation stalls and dominates wall clock.
Throughput bound training where batch size is the real lever Ring attention enables longer sequences, not more samples per second; if you can solve the problem with smaller context and bigger batch, data parallelism is cheaper.
Causal mask workloads without load balanced scheduling Naive ring with a causal mask leaves some ranks idle during half the rotation; you want a striped or zig zag schedule, otherwise utilization tanks.
Inference at small batch on a single accelerator PagedAttention plus FlashAttention already handles long context decode locally; ring attention's coordination cost is not worth it for one sequence inference.
Quick scan · 2–4 min
Everything important, quickly.
Walk the algorithm (N-1 rotations, online softmax composition), give memory and comm scaling, explain orthogonality to tensor and pipeline parallelism, mention Gemini 1.5 and Striped Attention, and note that ring attention is exact, not an approximation.
Side-by-side
Parallelism
Shards across
Memory benefit
Helps long context?
Data parallelism
Batch samples
None per-sequence
No
Tensor parallelism
Hidden dimensions
Activation/param weight scaling
Partial (constants)
Pipeline parallelism
Transformer layers
Layer-wise weights
Partial (depth)
Sequence (ring attention)
Sequence positions
Activations O(n/N)
Yes, primary axis
Real-world examples
Real products, models, and research that use this idea.
1Gemini 3.1 Pro's million token context is widely understood to use ring attention style sequence parallelism in training.
2DeepSpeed Ulysses implements sequence parallelism with similar ideas.
3Megatron-LM's context parallelism feature is essentially ring attention integrated with tensor and pipeline parallelism.
4Striped Attention refines ring attention for better causal mask balance and is now standard practice in 2026 long context stacks.
5Long context Llama 4 Maverick and DeepSeek V4 training pipelines use ring or context parallelism to push past what single-GPU FlashAttention can hold.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does ring attention specifically require FlashAttention's online softmax?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the communication cost of ring attention vs the compute cost?
Sign in or go Pro to see the approach for this follow-up.
QHow does ring attention compose with tensor parallelism?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying ring attention 'speeds up' attention or replaces softmax, it does neither. It enables sequences too long for one GPU by sharding sequence dimension, with the same exact math.