Same topic, related formats. Practice these next.
Prefill amortizes each weight read across many parallel tokens, so it is compute-bound; decode re-reads weights for one token per step, so it is memory bandwidth bound.
Imagine a chef who walks to a far-away pantry to fetch ingredients before cooking each dish. If twenty orders arrive at once, one trip to the pantry serves all twenty dishes, so the walk barely matters and the chef's hands are the limit. That is prefill: many tokens share one trip for the weights. Now imagine orders trickle in one at a time. The chef walks to the pantry, grabs ingredients, cooks one dish, then walks back for the next order. The walking dominates and the cooking is trivial. That is decode: one token per step, and fetching the model weights from far-away memory is what you wait on, not the math itself.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: define arithmetic intensity and the roofline ridge, then map prefill (parallel, compute-bound, TTFT) versus decode (one token, bandwidth-bound, TPOT), then the levers.
| Phase | Parallelism | Arithmetic intensity | Regime | Latency metric set |
|---|---|---|---|---|
| Prefill | Parallel over all prompt tokens | High, roughly 10 to 100 | Compute-bound | Time to first token |
| Decode (batch 1) | Sequential, one token per step | Near 1 | Memory bandwidth bound | Time per output token |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying decode is compute-bound because softmax or generation feels expensive. The cost is reading weights and the KV cache from HBM, not the arithmetic, which is trivial at one token per step.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.