How does the serving stack actually decide when to stop generation and what bugs come from mis-set stop sequences?
Describe how an LLM serving runtime decides to terminate generation. Cover the three standard mechanisms. Then identify two production bugs that arise from stop sequence handling and explain how to avoid them.
Decoding halts on three triggers: the model samples EOS, a user stop string matches the decoded suffix, or max_tokens fires. Mis-set stops cause runaway cost or premature truncation.
Imagine dictating a letter to an assistant who writes one word at a time and never stops on their own. You need a way to tell them when to put the pen down. There are three signals. First, you teach a secret word that means 'the end': when the assistant thinks of that word, they stop. Second, you say 'stop the moment you write the phrase Dear Sir again', so a repeated heading ends the letter. Third, you set a hard limit: 'no more than two pages, full stop.' If you forget all three, the assistant keeps writing forever and you pay for every word. If you pick a stop phrase that shows up in the middle of normal text, they stop too early and hand you half a letter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Stop conditions look trivial until you ship them. An autoregressive model has no inherent notion of "done"; it samples a probability distribution over the vocabulary at every step and could, in principle, continue forever. The serving runtime is what imposes termination, and it does so through three distinct mechanisms that interact with tokenization, chat templates, cost, and latency in ways that catch even experienced engineers. A question that looks like trivia ("how does it stop?") is really a probe of whether you understand the decode loop, the tokenizer, and the economics of serving at the same time.
The reason this matters economically is that decode is the expensive phase of inference and it is memory-bandwidth bound. Every token the model emits forces a read of the entire KV cache from HBM. A stop condition that fails to fire does not merely produce ugly output; it linearly inflates per-call cost and tail latency, and on a batched serving system it starves other requests of capacity. Conversely, a stop condition that fires too eagerly truncates valid answers mid-thought. Both failure modes are common in real deployments, and both are entirely preventable once you understand the three mechanisms and how they interact with the tokenizer.
This deep dive covers the three mechanisms, the tokenizer-boundary bug that makes naive stop matching unreliable, the role of chat templates and special tokens in modern instruct models, the precise cost of a runaway generation on memory-bound decode, and the production discipline (logging termination reasons, anchoring stop strings, keeping max_tokens as a backstop) that keeps a serving stack both correct and economical. By the end you should be able to explain not just the three triggers but why each one exists and what breaks when it is misconfigured.
The three termination mechanisms
Decoding halts on whichever of three conditions fires first. Each has a different owner and a different failure mode.
- EOS token: the model samples a learned end-of-sequence token. This is the model's own signal that it considers the answer complete, baked in during training. It is the only mechanism where the model itself decides to stop.
- Stop sequence: the caller supplies one or more strings, and after each step the runtime checks whether the decoded output now ends with one of them. This is application-level control, used to fence off the assistant's turn from a role header or a delimiter.
- max_tokens: a hard numeric cap on emitted tokens, independent of content. It is the safety net that bounds latency and cost and guarantees the loop terminates even if the other two mechanisms misbehave.
The mental model to carry into an interview: EOS is the model's choice, stop strings are the application's choice, and max_tokens is the clock. A robust deployment configures all three and never relies on a single one. Production APIs surface the winner of this race as a field; OpenAI calls it finish_reason and Anthropic calls it stop_reason, with values that map directly onto the three mechanisms.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's API exposes a 'stop' parameter (up to four strings) plus 'max_tokens', and the response 'finish_reason' field reports 'stop' or 'length' so callers can distinguish the termination cause.
- Llama 4 chat models define a distinct end-of-turn token in the tokenizer chat template, separate from the document-level EOS, and vLLM registers it as a stop token when the template is applied.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must a stop sequence be checked on detokenized text rather than at the token-id level?
Walk through how byte-pair encoding merges characters with their neighbors. The same visible string maps to different token IDs depending on the preceding character, so a fixed token-id pattern misses. Detokenizing the running output and doing a string suffix check is tokenization-invariant.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Checking stop sequences against raw token IDs instead of detokenized text. The same string tokenizes differently by leading character, so the match silently misses and generation runs away.
60 second bullets to scan on the way to the call.
The three termination mechanisms and which one is the model's own choice
Why EOS for instruct models is usually a chat template end of turn token
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.