How does the serving stack decide when to stop generation and what bugs come from mis-set stop sequences?
Describe how an LLM serving runtime decides to terminate generation. Cover the three standard mechanisms. Then identify two production bugs that arise from stop sequence handling and explain how to avoid them.
Decoding halts on three triggers: the model samples EOS, a user stop string matches the decoded suffix, or max_tokens fires. Mis-set stops cause runaway cost or premature truncation.
Imagine dictating a letter to an assistant who writes one word at a time and never stops on their own. You need a way to tell them when to put the pen down. There are three signals. First, you teach a secret word that means 'the end': when the assistant thinks of that word, they stop. Second, you say 'stop the moment you write the phrase Dear Sir again', so a repeated heading ends the letter. Third, you set a hard limit: 'no more than two pages, full stop.' If you forget all three, the assistant keeps writing forever and you pay for every word. If you pick a stop phrase that shows up in the middle of normal text, they stop too early and hand you half a letter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Stop conditions look simple but cause real production bugs. An autoregressive model has no built-in notion of "done"; it samples from a probability distribution over the vocabulary at every step and could, in principle, continue forever. The serving runtime imposes termination through three distinct mechanisms, and each one interacts with tokenization, chat templates, cost and latency.
The cost side matters because decode is the expensive phase of inference and it is memory-bandwidth bound. Every token the model emits needs a read of the sequence's whole KV cache from HBM. A stop condition that fails to fire does more than produce messy output: it raises per-call cost and tail latency in proportion to the wasted tokens, and on a batched server it takes capacity from other requests. A stop condition that fires too early truncates valid answers mid-thought. Both failures are common in real deployments, and both can be prevented.
The three termination mechanisms
Decoding halts on whichever of three conditions fires first. Each has a different owner and a different failure mode.
- EOS token: the model samples a learned end-of-sequence token. This is the model's own signal that it considers the answer complete, learned during training. It is the only mechanism where the model itself decides to stop.
- Stop sequence: the caller supplies one or more strings, and after each step the runtime checks whether the decoded output now ends with one of them. This is application-level control, used to fence off the assistant's turn from a role header or a delimiter.
- max_tokens: a hard numeric cap on emitted tokens, independent of content. It is the safety net that bounds latency and cost and guarantees the loop terminates even if the other two mechanisms misbehave.
EOS is the model's choice, stop strings are the application's choice, and max_tokens is the limit. A robust deployment configures all three and never relies on a single one. Production APIs report which condition fired. OpenAI's Chat Completions calls it finish_reason and uses "stop" for both EOS and a stop-sequence hit and "length" for the token limit. Anthropic calls it stop_reason and separates end_turn, stop_sequence and max_tokens, with further values such as tool_use.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's Chat Completions API takes up to 4 stop sequences and a max_completion_tokens limit (max_tokens is deprecated). Its finish_reason is 'stop' for both a natural stop and a stop-sequence hit, and 'length' when the limit is reached.
- Llama 4 chat models define a distinct end-of-turn token in the tokenizer chat template, separate from the document-level EOS; vLLM reads the end-of-turn IDs from the model's generation config and registers them as stop tokens.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must a stop sequence be checked on detokenized text rather than at the token-id level?
Walk through how byte-pair encoding merges characters with their neighbors. The same visible string maps to different token IDs depending on the preceding character, so a fixed token-id pattern misses. Detokenizing the running output and doing a string suffix check is tokenization-invariant.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Checking stop sequences against raw token IDs instead of detokenized text. The same string tokenizes differently by leading character, so the match silently misses and generation runs away.
60 second bullets to scan on the way to the call.
The three termination mechanisms and which one is the model's own choice
Why EOS for instruct models is usually a chat template end of turn token
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.