In a 7B transformer, do attention or MLP layers eat more FLOPs, and does context length change the answer?

Short answer·Medium·4.0 · 0·~3 min·Asked atAdobe Neo4j Sierra·Relevant atAi4bharat Cerebras Deepseek Meta

Attempt it

Inside a standard 7B decoder-only transformer block (multi-head attention + a GLU style MLP), which sub-layer dominates the FLOP budget at short context (say 2k tokens)? Does the answer flip at long context (say 128k tokens), and why?

Free · 2 AI evals / day

At short context (around 2k tokens), the **MLP** layer eats more FLOPs than attention. At long context (128k tokens), attention takes over. The flip happens because attention has O(n^2) terms while the MLP stays strictly O(n). **The MLP cost.** For a transformer with `d_model = 4096` and a SwiGLU FFN with hidden width `d_ff = 11008` (the Llama recipe), the MLP performs roughly `2 * n * d_model * d_ff` operations per token-projection pair. The cost grows linearly with sequence length n. **The attention cost.** Attention has both O(n) and O(n^2) parts. The Q, K, V, and W_O projections each cost `n * d_model^2`, linear in n. But the score matrix QK^T is `n by n` and costs `n^2 * d_head` per head; the attention output (weights @ V) is also `n^2 * d_head` per head. These two terms scale quadratically with n. **At n = 2048**, the n^2 term is small. The MLP wins because its per-token work is constant and that constant is large. Empirically, MLP is roughly 2/3 of block FLOPs for 7B at 2k context. **At n = 128k**, the n^2 terms dominate. Attention overtakes MLP and the gap widens fast. The crossover depends on the exact configuration but sits in the 8-16k range for typical 7B and 8B models. This is exactly why FlashAttention matters.

**MLP** wins at short context; **attention** wins at long context. The crossover is around 8-16k for typical 7B configurations. **MLP cost is O(n).** A Llama-style block has two matmuls in the FFN: one projects `d_model` to `d_ff` (typically about 4 * d_model or 2.7 * d_model in SwiGLU), one projects back. Per-token work is roughly `2 * d_model * d_ff` FLOPs. Total MLP FLOPs scale as `n * d_model * d_ff`. **Attention has both O(n) and O(n^2) parts.** - O(n) part: Q, K, V projections (3 * n * d_model^2) and W_O projection (n * d_model^2). - O(n^2) part: score matrix QK^T costs `n^2 * d_model` total across heads. Weighted-sum (weights @ V) costs another `n^2 * d_model`. So attention FLOPs are roughly `4 * n * d_model^2 + 2 * n^2 * d_model`. **At n = 2k**, plug in d_model = 4096, d_ff = 11008. MLP work per block is about `2k * 4096 * 11008 * 2 = 360 GFLOPs`. Attention O(n) part is about `4 * 2k * 4096^2 = 137 GFLOPs`. Attention O(n^2) part is about `2 * 2k^2 * 4096 = 34 GFLOPs`. Total attention is about 171 GFLOPs vs 360 for MLP. MLP wins about 2:1. **At n = 128k**, attention O(n^2) becomes `2 * 128k^2 * 4096 = 134 TFLOPs`. MLP is `128k * 4096 * 11008 * 2 = 23 TFLOPs`. Attention dominates about 6:1. **Crossover** depends on the d_ff to d_model ratio and head config. Small d_ff (more attention-heavy) flips earlier; large d_ff (more MLP-heavy) flips later. For Llama-3-8B with SwiGLU, the crossover sits in the 8-16k range. This is exactly the regime FlashAttention targets, the O(n^2) attention work is what grows with context and what serving stacks must optimize.

TL;DR

MLP dominates at short context (about 2/3 of FLOPs at 2k). Attention's O(n^2) terms overtake at long context, crossover near 8-16k for typical 7B configs.

Memory aid

Easy to grasp

Picture two workers handling letters in a mailroom. The first worker sorts each letter by recipient, that work grows in step with how many letters there are. Ten letters means ten sorts, a hundred letters means a hundred sorts. The second worker checks every letter against every other letter to find duplicates, so a hundred letters means ten thousand checks. With only a few letters, the first worker is way busier because there is more to do per letter. Once the stack gets tall enough, the second worker takes the lead because their workload grows much faster. MLP and attention split the work the same way: MLP is the per-letter sorter, attention is the pairwise checker.

Key concepts

MLP (feed-forward) layer Attention FLOPs FlashAttention Long-context scaling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A common interview question and a common source of confusion: which sub-layer of a transformer block, attention or MLP, eats more FLOPs? The answer depends on context length, and the reason is that attention has both O(n) and O(n^2) terms while MLP is strictly O(n).

This deep dive walks through the per-block FLOP breakdown for a standard 7B decoder, computes the split at 2k and 128k context, identifies where the crossover sits, and connects the analysis to why FlashAttention exists.

Mental model: MLP is heavy per-token. Attention is light per-token but has a quadratic term in n. Short context: per-token weight wins. Long context: n^2 wins.

Per-block FLOP decomposition

What goes into one transformer block

A decoder-only transformer block has two sub-layers, attention and MLP, each preceded by LayerNorm. The FLOP breakdown:

Attention sub-layer:
- Q, K, V projections: each is (seq_len, d_model) @ (d_model, d_model), costing 2 * n * d_model^2 FLOPs per projection, three projections total.
- Score matrix: (seq_len, d_model_per_head) @ (d_model_per_head, seq_len) per head, total 2 * n^2 * d_model across heads.
- Softmax: O(n^2) in count but small constant; usually ignored in FLOP analysis.
- Attention output (weights @ V): another 2 * n^2 * d_model.
- W_O projection: 2 * n * d_model^2.
MLP sub-layer (SwiGLU): three matmuls.
- Up projection: (n, d_model) @ (d_model, d_ff), costing 2 * n * d_model * d_ff.
- Gate projection: same shape, same cost.
- Down projection: (n, d_ff) @ (d_ff, d_model), costing 2 * n * d_model * d_ff.
- Total: 6 * n * d_model * d_ff.

Compact form

MLP FLOPs per block: 6 * n * d_model * d_ff.
Attention FLOPs per block: 8 * n * d_model^2 + 4 * n^2 * d_model.

The MLP is strictly linear in n. Attention has a linear part (the four projections) and a quadratic part (the score matrix plus weighted-sum).

Concrete numbers at 2k and 128k context

How configuration tilts the crossover

Why FlashAttention exists and what it actually optimizes

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama-3-8B (d_model = 4096, d_ff = 14336, 32 layers) shows MLP about 2/3 of FLOPs at 2k context, attention dominating past about 12k.
Mistral 7B with d_ff = 14336 has a similar crossover near 12k.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the crossover shift with d_ff / d_model ratio?

Larger d_ff (e.g. 4x or higher) pushes the MLP share up, so attention takes longer to overtake. Smaller d_ff or MoE with low activation ratio pushes the crossover earlier. The relationship is roughly linear in d_ff.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming attention is always the dominant cost. At short context the MLP wins comfortably; attention only overtakes once the n^2 terms grow past the linear MLP work.

60 second bullets to scan on the way to the call.

Per-block FLOP formula for MLP (SwiGLU vs vanilla)
Per-block FLOP formula for attention (projections vs scores vs weighted-sum)

Primary sources. Browse if you want the original framing.