Prefill is the single forward pass that processes the entire input prompt at once. Every prompt token is projected to its K and V in parallel and written into the KV cache, attention is computed across all prompt positions, and the model emits the first output token. It is a big, parallel matmul over the full prompt length, distinct from the sequential, one token at a time decode loop that follows.
When you send a prompt to an LLM, the very first thing the model does is process the entire prompt in one big pass. This phase is called **prefill**.
Concretely, every token in the prompt goes through the embedding layer, then through every transformer layer of the model, in parallel. Inside each layer, the model computes attention across all the prompt tokens at once: each token attends to every previous token to gather context. The keys (K) and values (V) computed for every position are written into a data structure called the **KV cache**, where they are stored for the entire rest of the request.
At the end of prefill, the model has done two things. First, it has built up the KV cache so that future steps do not have to recompute K and V for the prompt tokens. Second, it has emitted the first output token (the model is autoregressive, so the next token always comes out of the position immediately after the last input).
After prefill finishes, the model enters the **decode** phase: a sequential loop that produces output tokens one at a time, each consulting the KV cache to attend to everything that came before. Prefill is one big parallel matrix multiplication over the entire prompt. Decode is many small sequential matrix-vector products. They have completely different performance characteristics, and good inference engineers think about them separately.
**Prefill** is the first phase of LLM inference: a single forward pass that processes the entire input prompt at once. Every prompt token is embedded, run through all transformer layers in parallel, and contributes its K and V projections to the KV cache. Attention is computed across all prompt positions simultaneously, and the model produces logits for the first output token at the end of the pass.
The defining property of prefill is its **parallelism over the prompt length**. If the prompt is 2000 tokens, every layer's matmul has shape roughly `[2000, d_model] x [d_model, d_model]`. The 2000-token batch dimension means the weight matrix is reused 2000 times per multiply, so arithmetic intensity sits in the hundreds or thousands of FLOPs per byte. This pushes prefill onto the compute-bound side of the roofline: the GPU's tensor cores stay saturated and the workload runs near peak FLOPs.
The other defining property is that **prefill populates the KV cache**. K and V tensors for every prompt position are computed once and stored in HBM. From that point on, the decode loop only computes K and V for the single new token at each step and appends to the cache. Without this caching, decoding the second token would require re-running prefill over the full prompt plus the first output token, which would multiply total compute by a factor of `T_output`. Prefill is the upfront investment that makes decode tractable.
In production, prefill latency dominates time to first token, and TTFT is what the user perceives as 'how fast did the model start responding'. Optimizations like FlashAttention, prompt caching, and chunked prefill all target this phase.
Quick reference
TL;DR
Prefill is the one-shot parallel forward pass that processes the whole prompt at once, builds the KV cache, and produces the first output token.
Picture someone settling down to read a long book and then answer a question about it. They first read the entire book end to end, taking notes on each chapter so they can flip back without re-reading. That note-taking pass is prefill. Once the notes are ready, answering follow-up questions is fast because they consult the notes instead of re-reading the book. For an LLM, the notes are the KV cache, the book is the prompt, and the first sentence of the answer is the first thing they say after finishing the read. Prefill is one big concentrated reading session; decoding the answer afterward is many small consultations of the notes.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reasoning about per-token decode throughput Prefill happens once per request; the per-token rate during the long sequential output is governed by decode, not prefill.
Estimating bandwidth requirements for a serving fleet Prefill is compute-bound on long prompts; bandwidth modeling should focus on decode where HBM throughput is the binding constraint.
Designing for ultra-short prompts with long outputs Prefill cost is negligible at short prompts; the workload is essentially pure decode, so optimization effort should target the decode loop.
Quick scan · 2–4 min
Everything important, quickly.
3 min: parallel pass over the prompt + KV cache population + first token emission + arithmetic intensity contrast with decode + role in TTFT + chunked prefill and prompt caching as the two main optimizations.
Side-by-side
Aspect
Prefill
Decode
Compute pattern
Parallel over prompt length T
Sequential, one token per step
Arithmetic intensity
High (hundreds of FLOPs/byte)
Low (~1 FLOP/byte)
Bottleneck
Compute-bound (tensor cores)
Bandwidth-bound (HBM)
KV cache role
Writes the entire cache at once
Appends one K/V pair per step
Latency contribution
Dominates TTFT
Dominates total generation time
Real-world examples
Real products, models, and research that use this idea.
1Anthropic and OpenAI both expose prompt caching, which skips the prefill compute for repeated prefixes; this would be impossible if prefill state were not a clean, reusable object.
2FlashAttention v2 and v3 (used in vLLM, SGLang, TensorRT-LLM) optimize the attention computation that dominates long-prompt prefill.
3Sarathi-Serve and DistServe split prefill into chunks and interleave them with decode from other requests to keep the GPU busy.
4Claude Sonnet 4.6 with a 200k context can prefill the full window in roughly 5-15 seconds on H100-class hardware before the first output token appears.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is prefill compute-bound while decode is bandwidth-bound on the same hardware?
Sign in or go Pro to see the approach for this follow-up.
QHow does chunked prefill help overall throughput?
Sign in or go Pro to see the approach for this follow-up.
QWhat part of TTFT is NOT prefill?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating prefill and decode as 'the same forward pass'. They have different shapes, different bottlenecks (compute-bound vs bandwidth-bound), and different optimization techniques.