In a 7B transformer, do attention or MLP layers eat more FLOPs, and does context length change the answer?

Short answer·Medium·4.0 · 0·~3 min

Attempt it

Inside a standard 7B decoder-only transformer block (multi-head attention + a GLU-style MLP), which sub-layer dominates the FLOP budget at short context (say 2k tokens)? Does the answer flip at long context (say 128k tokens), and why?

Free · 2 AI evals / day

At short context (around 2k tokens), the **MLP** layer eats more FLOPs than attention. At long context (128k tokens), attention takes over. The flip happens because attention has O(n^2) terms while the MLP stays strictly O(n). **The MLP cost.** For a transformer with `d_model = 4096` and a SwiGLU FFN with hidden width `d_ff = 11008` (the Llama recipe), the MLP performs roughly `2 * n * d_model * d_ff` operations per token-projection pair. The cost grows linearly with sequence length n. **The attention cost.** Attention has both O(n) and O(n^2) parts. The Q, K, V, and W_O projections each cost `n * d_model^2`, linear in n. But the score matrix QK^T is `n by n` and costs `n^2 * d_head` per head; the attention output (weights @ V) is also `n^2 * d_head` per head. These two terms scale quadratically with n. **At n = 2048**, the n^2 term is small. The MLP wins because its per-token work is constant and that constant is large. Empirically, MLP is roughly 2/3 of block FLOPs for 7B at 2k context. **At n = 128k**, the n^2 terms dominate. Attention overtakes MLP and the gap widens fast. The crossover depends on the exact configuration but sits in the 8-16k range for typical 7B and 8B models. This is exactly why FlashAttention matters.

**MLP** wins at short context; **attention** wins at long context. The crossover is around 8-16k for typical 7B configurations. **MLP cost is O(n).** A Llama-style block has two matmuls in the FFN: one projects `d_model` to `d_ff` (typically about 4 * d_model or 2.7 * d_model in SwiGLU), one projects back. Per-token work is roughly `2 * d_model * d_ff` FLOPs. Total MLP FLOPs scale as `n * d_model * d_ff`. **Attention has both O(n) and O(n^2) parts.** - O(n) part: Q, K, V projections (3 * n * d_model^2) and W_O projection (n * d_model^2). - O(n^2) part: score matrix QK^T costs `n^2 * d_model` total across heads. Weighted-sum (weights @ V) costs another `n^2 * d_model`. So attention FLOPs are roughly `4 * n * d_model^2 + 2 * n^2 * d_model`. **At n = 2k**, plug in d_model = 4096, d_ff = 11008. MLP work per block is about `2k * 4096 * 11008 * 2 = 360 GFLOPs`. Attention O(n) part is about `4 * 2k * 4096^2 = 137 GFLOPs`. Attention O(n^2) part is about `2 * 2k^2 * 4096 = 34 GFLOPs`. Total attention is about 171 GFLOPs vs 360 for MLP. MLP wins about 2:1. **At n = 128k**, attention O(n^2) becomes `2 * 128k^2 * 4096 = 134 TFLOPs`. MLP is `128k * 4096 * 11008 * 2 = 23 TFLOPs`. Attention dominates about 6:1. **Crossover** depends on the d_ff to d_model ratio and head config. Small d_ff (more attention-heavy) flips earlier; large d_ff (more MLP-heavy) flips later. For Llama-3-8B with SwiGLU, the crossover sits in the 8-16k range. This is exactly the regime FlashAttention targets, the O(n^2) attention work is what grows with context and what serving stacks must optimize.

Quick reference

TL;DR

MLP dominates at short context (about 2/3 of FLOPs at 2k). Attention's O(n^2) terms overtake at long context, crossover near 8-16k for typical 7B configs.

Memory aid

Easy to grasp

Picture two workers handling letters in a mailroom. The first worker sorts each letter by recipient, that work grows in step with how many letters there are. Ten letters means ten sorts, a hundred letters means a hundred sorts. The second worker checks every letter against every other letter to find duplicates, so a hundred letters means ten thousand checks. With only a few letters, the first worker is way busier because there is more to do per letter. Once the stack gets tall enough, the second worker takes the lead because their workload grows much faster. MLP and attention split the work the same way: MLP is the per-letter sorter, attention is the pairwise checker.

Key concepts

MLP (feed-forward) layer Attention FLOPs FlashAttention Long-context scaling

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6-8 min: per-block FLOP breakdown + concrete numbers at 2k and 128k for 7B + crossover length analysis + FlashAttention as memory optimization + MoE wrinkle + decode-phase bandwidth story.

Real-world examples

Real products, models, and research that use this idea.

Llama-3-8B (d_model = 4096, d_ff = 14336, 32 layers) shows MLP about 2/3 of FLOPs at 2k context, attention dominating past about 12k.
Mistral 7B with d_ff = 14336 has a similar crossover near 12k.
Llama 4 Maverick MoE shifts the crossover earlier because the activated experts reduce effective per-token MLP cost.
DeepSeek V4 with MLA (Multi-head Latent Attention) compresses the KV side so the n^2 attention term has a smaller effective constant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the crossover shift with d_ff / d_model ratio?

QWhy does FlashAttention not change the FLOP count?

QHow does decode-phase serving change the FLOP analysis?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming attention is always the dominant cost. At short context the MLP wins comfortably; attention only overtakes once the n^2 terms grow past the linear MLP work.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.