Autoregressive means each new token is sampled conditioned on every previously generated token. Per step: the model produces a probability distribution over the vocabulary, the decoder samples one token from it, that token is appended to the context, and the loop repeats. Generation halts when a stop condition fires, EOS, a stop sequence, or max_tokens.
**Autoregressive** is a word from statistics: a process where each new value is predicted from past values of the same series. For LLMs, the series is the sequence of tokens being generated, and each new token is sampled conditioned on every token before it, both the prompt tokens and the tokens the model already emitted.
The per-step loop has four parts. **Forward pass.** The model runs one forward pass through all its layers, taking the current context (prompt plus all generated tokens so far) and producing a probability distribution over the entire vocabulary. **Sample.** The decoder picks one token from that distribution; greedy decoding picks the highest-probability token, temperature sampling and top-p sampling pick stochastically from a softened distribution. **Append.** The sampled token is appended to the context. **Check.** The loop checks termination conditions and either repeats or stops.
The loop terminates when one of three things happens. The model samples the end-of-sequence token (often called EOS or end-of-text), which it learned to emit at natural stopping points. The output contains a stop sequence the caller specified (like a specific newline pattern). Or the total tokens generated reaches the `max_tokens` cap.
The practical consequence is that generation is inherently sequential: you cannot compute token T+1 until you have token T. This sequential dependency is the fundamental reason LLM decode is slow and bandwidth-bound, and it is the reason for every optimization that tries to break the chain: speculative decoding, parallel decoding heads, draft models, multi-token prediction.
**Autoregressive generation** means each new token is sampled conditioned on every previously generated token. The model factors the joint probability of a sequence as the product of conditionals: `P(t1, t2, ..., tN) = prod P(ti | t1...ti-1)`. Generation realizes that factorization left-to-right one token at a time.
The **per-step loop** has four stages:
1. **Forward pass.** The model takes the current context (prompt plus all generated tokens so far) through all transformer layers, producing logits over the full vocabulary at the final position. A softmax converts the logits to a probability distribution.
2. **Sample.** The decoder picks one token. Greedy picks argmax, sampling picks stochastically based on temperature, top-k, top-p, or other modifiers.
3. **Append.** The sampled token is concatenated to the context and the KV cache absorbs its K and V tensors so the next step does not recompute earlier projections.
4. **Terminate or repeat.** Check stop conditions; if none fire, run another step.
**Termination conditions** are three: the model samples its trained end-of-sequence token (EOS), the generated text matches a caller-supplied stop sequence, or the total tokens produced reaches `max_tokens`. Production servers track all three.
The **performance consequence** of autoregression is the dominant fact of LLM serving. Token T+1 depends on token T, so generation cannot parallelize across the output dimension. Every decode step is a full forward pass through all layers for a single new token, which is why decode is matrix-vector at batch 1, why it sits at arithmetic intensity near 1, and why it is HBM-bandwidth-bound. Every modern serving optimization, from continuous batching through speculative decoding to multi-token prediction, attacks the consequences of autoregression in some way without breaking the conditional dependency the model was trained on.
Quick reference
TL;DR
Autoregressive generation samples one token at a time conditioned on every token before it. Each step emits a vocab distribution, samples a token, appends it to the context, and repeats until a stop condition fires.
Imagine writing a story one word at a time, where every word you pick depends on everything you have already written. After each word, you look back at the whole draft and ask, given all of this, what feels like the next word. You sample one option, write it down, and ask the question again with the longer draft. That loop is autoregressive generation. The model is the author, the draft is the context, the sampling step is the actual choice of word, and the loop only stops when the model itself decides to end the story (the end-of-text token), or when it runs into a stop word the user set, or when it hits a maximum length cap.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks naturally formulated as parallel structured output Some tasks (image generation, audio waveform synthesis) have non-autoregressive formulations that win on quality and latency for the right modality.
Very short outputs where prefill dominates total cost If the prompt is long and the response is one or two tokens, the autoregressive loop is a small fraction of total work and most optimization effort should go into prefill instead.
Reasoning tasks where total output token count is the binding cost Autoregression itself is fine, but the cost is dominated by hidden reasoning tokens, so the optimization target shifts from per-step latency to per-request token budget.
Quick scan · 2–4 min
Everything important, quickly.
2 min: autoregressive factorization + four-step decode loop + KV cache as the inference-time response + termination conditions + why every optimization respects the dependency.
Real-world examples
Real products, models, and research that use this idea.
1GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Llama 4 Maverick, and DeepSeek V4 are all autoregressive transformers; the entire frontier model lineup in 2026 still uses the same left-to-right factorization.
2vLLM, SGLang, and TensorRT-LLM all implement the same autoregressive decode loop under the hood, differing only in how they manage the KV cache and schedule the batches.
3DeepSeek V4 ships multi-token prediction during training to amortize the autoregressive loop at inference, generating several tokens per forward pass when speculation succeeds.
4Medusa (Stanford) and EAGLE (Microsoft) are speculative-decoding techniques that propose multiple tokens per step while preserving the autoregressive distribution exactly.
5OpenAI's o-series, Anthropic's extended-thinking Opus, and DeepSeek V4 reasoning all autoregressively generate hidden chain-of-thought tokens before the visible answer, which is why those models bill more output tokens per request.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is prefill parallel while decode is sequential, even though both use the same model?
Sign in or go Pro to see the approach for this follow-up.
QHow does speculative decoding preserve the autoregressive distribution while running multiple tokens per step?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes about the loop when extended thinking or reasoning is enabled?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing the model generates the full output in one shot. Each token is its own forward pass, conditioned on every earlier token, which is what makes decode slow and bandwidth-bound.