Spot the errors in this stop sequence configuration recommendation
Click any words you think contain an error. Click again to unmark.
Stop sequences match the detokenized output string, not token IDs; tokenization is context-sensitive, so whitespace anchors matter and a wrong stop config plus no max_tokens means runaway generation and cost blowup.
Imagine telling a dictation assistant to stop the moment it writes the word 'Stop'. You might assume 'Stop' is always the same secret code inside the machine. But the machine breaks words into chunks, and which chunk it uses depends on what came just before, a space, a newline, or the very start. So checking the secret code number is unreliable. Instead, the assistant should write out the actual letters and look at the visible text for the word 'Stop'. And if you never tell it a maximum length and the word never appears, it keeps writing forever, running up your bill. The safe setup matches the readable text and always caps the length.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Stop handling is where tokenization theory meets a real production bill. The recommendation in this question packs three confident, wrong claims into four sentences, and each one maps to an outage a serving team has actually shipped. Underneath the surface errors sits the failure the title hints at: a stop configuration that never fires, combined with no length cap, produces runaway generation and a cost blowup.
To unpack it you need three layers. First, how a runtime actually detects a stop: it decodes recent output to text and substring-matches, it does not compare token IDs. Second, why it must work that way: BPE tokenization is context-sensitive, so a substring does not own a single stable ID. Third, why whitespace in a stop string is load-bearing rather than ignored: the leading newline is the anchor that ties a stop to a turn boundary.
This deep dive walks each error, then turns to the correct mental model of termination, where the trained EOS token and a hard max_tokens cap, not the advisory stop strings, are what actually guarantee the model halts and the bill stays bounded.
The stakes are concrete. Every claim in the recommendation sounds reasonable to someone who has only read a tokenizer tutorial, which is exactly why it is a strong staff-level spot-error item. The candidate who catches all three errors and then volunteers the missing cap is the one who has actually debugged a serving stack at three in the morning.
Error one: stop matching runs on decoded text, not token IDs
The recommendation says the stack compares every generated token ID to the ID for 'Human:' and halts on a match. That is not how production runtimes work. The serving layer keeps a sliding window of the most recently generated tokens, decodes that window back into a string, and checks whether the string contains one of your stop sequences. The comparison happens in text space.
The reason is mechanical. There is no single ID for an arbitrary string, so there is nothing stable to compare against. A stop sequence is a sequence of characters the user wants to halt on, and the only reliable place those characters exist unambiguously is in the decoded output. Runtimes such as vLLM and TGI implement exactly this: detokenize the tail, search for the stop string, and if found, truncate the response before it and stop.
This also explains why stops are described as best-effort. The match runs on a window, partial tokens at the boundary are handled carefully, and the decoded string is what the user finally sees. Match on IDs and you tie yourself to an implementation detail that does not hold.
There is a streaming subtlety worth naming. Because the runtime decodes a window and searches it, a stop string can straddle two generated tokens, and the runtime must hold back partial output until it is sure no stop is forming. This is why naive token by token streaming sometimes leaks a fragment of the stop string before catching it. Good implementations buffer the tail until the match decision is unambiguous, which is only possible when you reason in decoded-text space rather than over raw IDs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and Hugging Face TGI both implement stop sequences as detokenized substring matching over a sliding window of recent output, not token-ID comparison.
- The Anthropic Messages API for Claude Opus 4.7 returns stop_reason of 'end_turn', 'stop_sequence', or 'max_tokens', making the active terminator explicit per response.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can a stop sequence that begins in the middle of a token never match cleanly?
The model emits whole tokens. If your stop string starts inside a merged BPE token, the decoded boundary never aligns with the start of your string. Anchor stops on whitespace or known token boundaries so the decoded text can actually contain the literal sequence.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a string maps to one stable token ID and matching on IDs, then shipping with no max_tokens. A missed stop with no cap means runaway generation and a cost blowup.
60 second bullets to scan on the way to the call.
Why stop matching runs on decoded text rather than raw token IDs
How leading context changes the token ID for the same substring
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.