Inside a standard 7B decoder-only transformer block (multi-head attention + a GLU-style MLP), which sub-layer dominates the FLOP budget at short context (say 2k tokens)? Does the answer flip at long context (say 128k tokens), and why?
Free · 2 AI evals / day
At short context (around 2k tokens), the **MLP** layer eats more FLOPs than attention. At long context (128k tokens), attention takes over. The flip happens because attention has O(n^2) terms while the MLP stays strictly O(n).
**The MLP cost.** For a transformer with `d_model = 4096` and a SwiGLU FFN with hidden width `d_ff = 11008` (the Llama recipe), the MLP performs roughly `2 * n * d_model * d_ff` operations per token-projection pair. The cost grows linearly with sequence length n.
**The attention cost.** Attention has both O(n) and O(n^2) parts. The Q, K, V, and W_O projections each cost `n * d_model^2`, linear in n. But the score matrix QK^T is `n by n` and costs `n^2 * d_head` per head; the attention output (weights @ V) is also `n^2 * d_head` per head. These two terms scale quadratically with n.
**At n = 2048**, the n^2 term is small. The MLP wins because its per-token work is constant and that constant is large. Empirically, MLP is roughly 2/3 of block FLOPs for 7B at 2k context.
**At n = 128k**, the n^2 terms dominate. Attention overtakes MLP and the gap widens fast. The crossover depends on the exact configuration but sits in the 8-16k range for typical 7B and 8B models. This is exactly why FlashAttention matters.
**MLP** wins at short context; **attention** wins at long context. The crossover is around 8-16k for typical 7B configurations.
**MLP cost is O(n).** A Llama-style block has two matmuls in the FFN: one projects `d_model` to `d_ff` (typically about 4 * d_model or 2.7 * d_model in SwiGLU), one projects back. Per-token work is roughly `2 * d_model * d_ff` FLOPs. Total MLP FLOPs scale as `n * d_model * d_ff`.
**Attention has both O(n) and O(n^2) parts.**
- O(n) part: Q, K, V projections (3 * n * d_model^2) and W_O projection (n * d_model^2).
- O(n^2) part: score matrix QK^T costs `n^2 * d_model` total across heads. Weighted-sum (weights @ V) costs another `n^2 * d_model`.
So attention FLOPs are roughly `4 * n * d_model^2 + 2 * n^2 * d_model`.
**At n = 2k**, plug in d_model = 4096, d_ff = 11008. MLP work per block is about `2k * 4096 * 11008 * 2 = 360 GFLOPs`. Attention O(n) part is about `4 * 2k * 4096^2 = 137 GFLOPs`. Attention O(n^2) part is about `2 * 2k^2 * 4096 = 34 GFLOPs`. Total attention is about 171 GFLOPs vs 360 for MLP. MLP wins about 2:1.
**At n = 128k**, attention O(n^2) becomes `2 * 128k^2 * 4096 = 134 TFLOPs`. MLP is `128k * 4096 * 11008 * 2 = 23 TFLOPs`. Attention dominates about 6:1.
**Crossover** depends on the d_ff to d_model ratio and head config. Small d_ff (more attention-heavy) flips earlier; large d_ff (more MLP-heavy) flips later. For Llama-3-8B with SwiGLU, the crossover sits in the 8-16k range.
This is exactly the regime FlashAttention targets, the O(n^2) attention work is what grows with context and what serving stacks must optimize.
Quick reference
TL;DR
MLP dominates at short context (about 2/3 of FLOPs at 2k). Attention's O(n^2) terms overtake at long context, crossover near 8-16k for typical 7B configs.
Picture two workers handling letters in a mailroom. The first worker sorts each letter by recipient, that work grows in step with how many letters there are. Ten letters means ten sorts, a hundred letters means a hundred sorts. The second worker checks every letter against every other letter to find duplicates, so a hundred letters means ten thousand checks. With only a few letters, the first worker is way busier because there is more to do per letter. Once the stack gets tall enough, the second worker takes the lead because their workload grows much faster. MLP and attention split the work the same way: MLP is the per-letter sorter, attention is the pairwise checker.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Citing FLOP analysis to predict serving latency in the decode phase Decode is usually bandwidth-bound, not FLOP-bound. FLOPs predict prefill cost well; decode latency depends on KV cache memory traffic.
Comparing a dense 7B with an MoE model on raw FLOP shares MoE's effective per-token MLP cost is much lower than the dense equivalent because only a subset of experts activates per token; the FLOP share comparison is not apples to apples.
Treating attention as the only optimization target for long-context throughput KV cache memory and bandwidth often bottleneck before raw FLOPs do; PagedAttention and KV quantization are usually higher-leverage than attention FLOP optimizations.
Quick scan · 2–4 min
Everything important, quickly.
6-8 min: per-block FLOP breakdown + concrete numbers at 2k and 128k for 7B + crossover length analysis + FlashAttention as memory optimization + MoE wrinkle + decode-phase bandwidth story.
Real-world examples
Real products, models, and research that use this idea.
1Llama-3-8B (d_model = 4096, d_ff = 14336, 32 layers) shows MLP about 2/3 of FLOPs at 2k context, attention dominating past about 12k.
2Mistral 7B with d_ff = 14336 has a similar crossover near 12k.
3Llama 4 Maverick MoE shifts the crossover earlier because the activated experts reduce effective per-token MLP cost.
4DeepSeek V4 with MLA (Multi-head Latent Attention) compresses the KV side so the n^2 attention term has a smaller effective constant.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the crossover shift with d_ff / d_model ratio?
Sign in or go Pro to see the approach for this follow-up.
QWhy does FlashAttention not change the FLOP count?
Sign in or go Pro to see the approach for this follow-up.
QHow does decode-phase serving change the FLOP analysis?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming attention is always the dominant cost. At short context the MLP wins comfortably; attention only overtakes once the n^2 terms grow past the linear MLP work.