Describe what 'autoregressive' means for LLM generation.

Flashcard·Easy·4.0 · 0·~30s·Asked atCoreweave Intel Lightning Ai·Relevant atOpenAI

Attempt it

Your answer

**Autoregressive** is a word from statistics: a process where each new value is predicted from past values of the same series. For LLMs, the series is the sequence of tokens being generated, and each new token is sampled conditioned on every token before it, both the prompt tokens and the tokens the model already emitted. The per-step loop has four parts. **Forward pass.** The model runs one forward pass through all its layers, taking the current context (prompt plus all generated tokens so far) and producing a probability distribution over the entire vocabulary. **Sample.** The decoder picks one token from that distribution; greedy decoding picks the highest-probability token, temperature sampling and top-p sampling pick stochastically from a softened distribution. **Append.** The sampled token is appended to the context. **Check.** The loop checks termination conditions and either repeats or stops. The loop terminates when one of three things happens. The model samples the end-of-sequence token (often called EOS or end-of-text), which it learned to emit at natural stopping points. The output contains a stop sequence the caller specified (like a specific newline pattern). Or the total tokens generated reaches the `max_tokens` cap. The practical consequence is that generation is inherently sequential: you cannot compute token T+1 until you have token T. This sequential dependency is the fundamental reason LLM decode is slow and bandwidth-bound, and it is the reason for every optimization that tries to break the chain: speculative decoding, parallel decoding heads, draft models, multi-token prediction.

**Autoregressive generation** means each new token is sampled conditioned on every previously generated token. The model factors the joint probability of a sequence as the product of conditionals: `P(t1, t2, ..., tN) = prod P(ti | t1...ti-1)`. Generation realizes that factorization left-to-right one token at a time. The **per-step loop** has four stages: 1. **Forward pass.** The model takes the current context (prompt plus all generated tokens so far) through all transformer layers, producing logits over the full vocabulary at the final position. A softmax converts the logits to a probability distribution. 2. **Sample.** The decoder picks one token. Greedy picks argmax, sampling picks stochastically based on temperature, top-k, top-p, or other modifiers. 3. **Append.** The sampled token is concatenated to the context and the KV cache absorbs its K and V tensors so the next step does not recompute earlier projections. 4. **Terminate or repeat.** Check stop conditions; if none fire, run another step. **Termination conditions** are three: the model samples its trained end-of-sequence token (EOS), the generated text matches a caller-supplied stop sequence, or the total tokens produced reaches `max_tokens`. Production servers track all three. The **performance consequence** of autoregression is the dominant fact of LLM serving. Token T+1 depends on token T, so generation cannot parallelize across the output dimension. Every decode step is a full forward pass through all layers for a single new token, which is why decode is matrix-vector at batch 1, why it sits at arithmetic intensity near 1, and why it is HBM-bandwidth-bound. Every modern serving optimization, from continuous batching through speculative decoding to multi-token prediction, attacks the consequences of autoregression in some way without breaking the conditional dependency the model was trained on.

TL;DR

Autoregressive generation samples one token at a time conditioned on every token before it. Each step emits a vocab distribution, samples a token, appends it to the context, and repeats until a stop condition fires.

Memory aid

Easy to grasp

Imagine writing a story one word at a time, where every word you pick depends on everything you have already written. After each word, you look back at the whole draft and ask, given all of this, what feels like the next word. You sample one option, write it down, and ask the question again with the longer draft. That loop is autoregressive generation. The model is the author, the draft is the context, the sampling step is the actual choice of word, and the loop only stops when the model itself decides to end the story (the end-of-text token), or when it runs into a stop word the user set, or when it hits a maximum length cap.

Key concepts

Autoregressive generation Decoding strategies Transformer foundations

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Autoregressive generation is the defining cost shape of LLM inference. The model factors the probability of a sequence as a product of conditionals, and serving realizes that factorization one token at a time. Token T+1 depends on token T, so the decode loop cannot parallelize across the output dimension. Every inference optimization in 2026, batching, KV cache, speculative decoding, multi-token prediction, exists because of this constraint.

This question is the foundation everything else builds on. A candidate who fumbles the per-step loop will fumble every downstream question about decode performance, throughput optimization, or speculative decoding. A candidate who explains it crisply has the mental model that makes the rest of inference engineering coherent.

The deep dive walks the mathematical framing, the per-step loop, the role of the KV cache, the termination conditions, and the optimization landscape that follows. By the end you should be able to explain why decode is the bottleneck of LLM serving and which optimizations attack which part of the autoregressive loop.

The factorization and what it encodes

An autoregressive language model factors the joint probability of a sequence into a product of conditionals, each one predicting the next token given all earlier tokens:

P(t_1, t_2, \ldots, t_N) = \prod_{i=1}^{N} P(t_i \mid t_1, \ldots, t_{i-1})

This factorization is exact for any sequence model; the modeling assumption is in the architecture (transformer with causal mask) and the training objective (maximize likelihood under this factorization). The causal mask in self-attention is what enforces the conditional dependency: when computing the representation for position i, attention scores at positions j > i are masked to negative infinity so they contribute nothing.

During training, the model learns all conditionals in parallel via teacher forcing: feed in the ground-truth sequence, predict every position at once, sum the cross-entropy loss across positions. During inference, the same conditionals are realized sequentially: sample t_1 from P(t_1 | prompt), then t_2 from P(t_2 | prompt, t_1), and so on. The training-inference asymmetry is the whole reason serving is hard.

The per-step loop in detail

Why the KV cache is the inference-time response

Termination conditions and finish_reason

The optimization landscape that follows

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Llama 4 Maverick, and DeepSeek V4 are all autoregressive transformers; the entire frontier model lineup in 2026 still uses the same left-to-right factorization.
vLLM, SGLang, and TensorRT-LLM all implement the same autoregressive decode loop under the hood, differing only in how they manage the KV cache and schedule the batches.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is prefill parallel while decode is sequential, even though both use the same model?

Prefill processes the whole prompt at once; every position's Q, K, V can be computed in parallel because the causal mask hides future positions from each query. Decode generates token by token because token T+1 does not exist yet when computing token T; the conditional dependency is a sequential one. Same architecture, different access pattern.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing the model generates the full output in one shot. Each token is its own forward pass, conditioned on every earlier token, which is what makes decode slow and bandwidth-bound.

60 second bullets to scan on the way to the call.

The mathematical factorization that autoregression encodes
The four stages of the per-step decode loop

Primary sources. Browse if you want the original framing.