Spot the errors in this stop sequence configuration recommendation
Click any words you think contain an error. Click again to unmark.
Stop sequences match the detokenized output string, not token IDs; tokenization is context-sensitive, so whitespace anchors matter and a wrong stop config plus no max_tokens means runaway generation and high cost.
Imagine telling a dictation assistant to stop the moment it writes the word 'Stop'. You might assume 'Stop' is always the same secret code inside the machine. But the machine breaks words into chunks, and which chunk it uses depends on what came just before, a space, a newline, or the very start. So checking the secret code number is unreliable. Instead, the assistant should write out the actual letters and look at the visible text for the word 'Stop'. And if you never tell it a maximum length and the word never appears, it keeps writing forever, running up your bill. The safe setup matches the readable text and always caps the length.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Stop handling connects tokenizer behavior to the serving bill. The recommendation makes three wrong claims in four sentences, and each one leads to a real serving failure. Under those errors sits a larger one: a stop configuration that never fires, combined with no length cap, produces runaway generation and a large cost increase.
Three points explain the errors. First, how a runtime detects a stop: it decodes recent output to text and looks for the stop string; it does not compare token IDs. Second, why it must work that way: BPE tokenization is context-sensitive, so a substring has no single stable ID. Third, why whitespace in a stop string matters: the leading newline ties a stop to a turn boundary.
Termination is guaranteed by the trained EOS token and a hard max_tokens cap, not by the advisory stop strings.
Error one: stop matching runs on decoded text, not token IDs
The recommendation says the stack compares every generated token ID to the ID for 'Human:' and halts on a match. That is not how production runtimes work. The serving layer keeps a sliding window of the most recently generated tokens, decodes that window back into a string, and checks whether the string contains one of your stop sequences. The comparison happens in text space.
The reason is mechanical. There is no single ID for an arbitrary string, so there is nothing stable to compare against. A stop sequence is a sequence of characters the user wants to halt on, and the only reliable place those characters exist unambiguously is in the decoded output. Open engines such as vLLM work this way: vLLM accepts stop strings only when detokenization is on, searches the decoded text, and by default returns the output without the stop string.
This also explains why stops are described as best-effort. The match runs on a window, partial tokens at the boundary are handled carefully, and the decoded string is what the user finally sees. Matching on IDs depends on a fixed string-to-ID mapping that does not exist.
Streaming adds one more detail. Because the runtime decodes a window and searches it, a stop string can straddle two generated tokens, and the runtime must hold back partial output until it is sure no stop is forming. This is why naive token by token streaming sometimes leaks a fragment of the stop string before catching it. Good implementations buffer the tail until the match decision is unambiguous, which is only possible when you reason in decoded-text space rather than over raw IDs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM's SamplingParams accepts stop strings only when detokenization is on and holds back a few characters of output while it checks them, because the match runs on text, not token IDs.
- The Anthropic Messages API returns stop_reason 'stop_sequence' with the matched string in the stop_sequence field, and also reports 'end_turn', 'max_tokens' and other reasons, so the terminator is explicit per response.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does a stop string that begins in the middle of a token defeat a matcher that works on token IDs?
The model emits whole tokens. If the stop string starts inside a merged BPE token, no sequence of token IDs lines up with the string, so an ID-based matcher never fires. A matcher on the decoded text still finds the literal characters, because it compares characters, not tokens. Anchoring stops on whitespace or a known boundary also keeps the string aligned with token boundaries.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a string maps to one stable token ID and matching on IDs, then shipping with no max_tokens. A missed stop with no cap means runaway generation and a large cost increase.
60 second bullets to scan on the way to the call.
Why stop matching runs on decoded text rather than raw token IDs
How leading context changes the token ID for the same substring
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.