How does the serving stack actually decide when to stop generation and what bugs come from mis-set stop sequences?

An LLM generates one token at a time and, left alone, would never stop. The serving runtime needs explicit rules for when to halt. There are three standard ones. First, the **EOS token**. During training the model learns a special end-of-sequence token; when sampling produces it, the runtime ends the response. This is the model's own signal that the answer is complete. Second, stop sequences. The caller supplies strings like a newline plus a role header. After each token, the runtime checks whether the decoded text now ends with one of these strings, and stops if so. This is application control. Third, max_tokens. A hard cap on how many tokens to emit, regardless of content. It is the safety net that prevents runaway generation and bounds cost. The subtle bugs come from stop sequences. The same characters can split across token boundaries differently depending on what came before, so a naive check on token IDs can miss the match. The fix is to compare against the detokenized string, and to pick stop strings anchored with delimiters that will not appear inside normal answer text.

An autoregressive model emits tokens until something tells it to stop. The serving runtime enforces three mechanisms. The first is the **EOS token**: the model samples a learned end-of-sequence token, and the runtime treats that as the answer being complete. The second is a user-supplied stop sequence. After each step the runtime checks whether the decoded suffix ends with a configured string, such as a newline plus a role header. The third is max_tokens, a hard cap that bounds latency and cost no matter what the content is. EOS is the model's signal, stop strings are application control, and max_tokens is the safety cap. Two production bugs dominate. The first is tokenizer dependence. A stop string like a role header can tokenize differently depending on the preceding character, so a check at the token-id boundary misses and generation runs away, burning tokens. Fix it by matching on detokenized text, or by encoding the stop and checking a token prefix. The second is a false-positive mid-content match. A short stop string can appear inside legitimate output, truncating the answer early. Fix it by anchoring stops with stable delimiters like a leading newline.