Walk through what actually happens during the prefill phase of an LLM forward pass.

Flashcard·Easy·4.0 · 0·~30s·Asked atCanva Induced Ai Midjourney·Relevant atOpenAI

Attempt it

Prefill is the single forward pass that processes the entire input prompt at once. Every prompt token is projected to its K and V in parallel and written into the KV cache, attention is computed across all prompt positions, and the model emits the first output token. It is a big, parallel matmul over the full prompt length, distinct from the sequential, one token at a time decode loop that follows.

Your answer

When you send a prompt to an LLM, the very first thing the model does is process the entire prompt in one big pass. This phase is called **prefill**. Concretely, every token in the prompt goes through the embedding layer, then through every transformer layer of the model, in parallel. Inside each layer, the model computes attention across all the prompt tokens at once: each token attends to every previous token to gather context. The keys (K) and values (V) computed for every position are written into a data structure called the **KV cache**, where they are stored for the entire rest of the request. At the end of prefill, the model has done two things. First, it has built up the KV cache so that future steps do not have to recompute K and V for the prompt tokens. Second, it has emitted the first output token (the model is autoregressive, so the next token always comes out of the position immediately after the last input). After prefill finishes, the model enters the **decode** phase: a sequential loop that produces output tokens one at a time, each consulting the KV cache to attend to everything that came before. Prefill is one big parallel matrix multiplication over the entire prompt. Decode is many small sequential matrix-vector products. They have completely different performance characteristics, and good inference engineers think about them separately.

**Prefill** is the first phase of LLM inference: a single forward pass that processes the entire input prompt at once. Every prompt token is embedded, run through all transformer layers in parallel, and contributes its K and V projections to the KV cache. Attention is computed across all prompt positions simultaneously, and the model produces logits for the first output token at the end of the pass. The defining property of prefill is its **parallelism over the prompt length**. If the prompt is 2000 tokens, every layer's matmul has shape roughly `[2000, d_model] x [d_model, d_model]`. The 2000-token batch dimension means the weight matrix is reused 2000 times per multiply, so arithmetic intensity sits in the hundreds or thousands of FLOPs per byte. This pushes prefill onto the compute-bound side of the roofline: the GPU's tensor cores stay saturated and the workload runs near peak FLOPs. The other defining property is that **prefill populates the KV cache**. K and V tensors for every prompt position are computed once and stored in HBM. From that point on, the decode loop only computes K and V for the single new token at each step and appends to the cache. Without this caching, decoding the second token would require re-running prefill over the full prompt plus the first output token, which would multiply total compute by a factor of `T_output`. Prefill is the upfront investment that makes decode tractable. In production, prefill latency dominates time to first token, and TTFT is what the user perceives as 'how fast did the model start responding'. Optimizations like FlashAttention, prompt caching, and chunked prefill all target this phase.

TL;DR

Prefill is the one-shot parallel forward pass that processes the whole prompt at once, builds the KV cache, and produces the first output token.

Memory aid

Easy to grasp

Picture someone settling down to read a long book and then answer a question about it. They first read the entire book end to end, taking notes on each chapter so they can flip back without re-reading. That note-taking pass is prefill. Once the notes are ready, answering follow-up questions is fast because they consult the notes instead of re-reading the book. For an LLM, the notes are the KV cache, the book is the prompt, and the first sentence of the answer is the first thing they say after finishing the read. Prefill is one big concentrated reading session; decoding the answer afterward is many small consultations of the notes.

Key concepts

Prefill phase Decode phase KV cache Time to first token Arithmetic intensity

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LLM inference has two phases that share almost nothing in common. Prefill is one big parallel pass over the input prompt. Decode is a sequential loop that emits output tokens one at a time. The same neural network runs in both phases, but the matrix shapes, the GPU utilization, the bottleneck, and the optimizations are completely different. Anyone designing an inference system needs to think about these two workloads separately, because techniques that help one often do nothing for the other.

This deep dive walks through what physically happens during prefill, why its arithmetic intensity puts it on the compute-bound side of the roofline plot, how it populates the KV cache so that decode does not have to recompute everything from scratch, and what production optimizations like chunked prefill and prompt caching actually do under the hood. By the end, you should be able to look at any inference benchmark and reason about how much of the time is prefill cost versus decode cost.

What happens inside prefill, layer by layer

Prefill takes the entire input prompt and runs it through the model in a single parallel forward pass. Concretely:

Tokenize and embed. The prompt is tokenized into a sequence of T token IDs and converted into an initial [T, d_model] activation tensor via the embedding layer.
Per-layer transformer block. At each layer, the activation goes through layer norm, then a self-attention sub-block, then a feed-forward sub-block. Inside attention, the activation is projected to Q, K, and V via three linear projections, each producing a [T, d_model] tensor (or [T, n_heads * d_head] after reshaping).
Attention computation. Scaled dot-product attention is computed as softmax(QK^T / sqrt(d_k)) V. The QK^T matrix has shape [T, T], which is the part that scales quadratically with prompt length and motivates FlashAttention. A causal mask is applied so position i can only attend to positions <= i.
KV cache write. The K and V tensors computed at each layer are written to the KV cache, indexed by layer, head, and position. The cache is laid out in HBM such that the decode loop can later read it sequentially.
Feed-forward and residual. After attention, the activation passes through the feed-forward block (typically two linear layers with a SwiGLU or GELU nonlinearity in between), and residual connections plus layer norm wrap each sub-block.
Output head. After the last transformer layer, a final layer norm and the output projection map the activation at position T-1 (the last prompt token) to a logits vector over the vocabulary. The first output token is sampled from these logits.

At the end of prefill, two things have happened. The KV cache is fully populated for the entire prompt across every layer, and the first output token has been emitted. The decode loop can now start from a warm cache.

Why prefill is compute-bound

How prefill defines time to first token

Chunked prefill and continuous batching

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Prefill	Decode
Compute pattern	Parallel over prompt length T	Sequential, one token per step
Arithmetic intensity	High (hundreds of FLOPs/byte)	Low (~1 FLOP/byte)
Bottleneck	Compute-bound (tensor cores)	Bandwidth-bound (HBM)
KV cache role	Writes the entire cache at once	Appends one K/V pair per step
Latency contribution	Dominates TTFT	Dominates total generation time

Real products, models, and research that use this idea.

Anthropic and OpenAI both expose prompt caching, which skips the prefill compute for repeated prefixes; this would be impossible if prefill state were not a clean, reusable object.
FlashAttention v2 and v3 (used in vLLM, SGLang, TensorRT-LLM) optimize the attention computation that dominates long-prompt prefill.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is prefill compute-bound while decode is bandwidth-bound on the same hardware?

Arithmetic intensity. Prefill's batch dimension is the prompt length, so each weight is reused thousands of times per matmul. Decode's batch dimension is 1, so each weight is read for a single multiply. The roofline plot puts prefill far right (compute) and decode far left (bandwidth).

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating prefill and decode as 'the same forward pass'. They have different shapes, different bottlenecks (compute-bound vs bandwidth-bound), and different optimization techniques.

60 second bullets to scan on the way to the call.

Definition of prefill as a single parallel forward pass over the prompt
How the KV cache is populated during prefill

Primary sources. Browse if you want the original framing.