Define a stop sequence in an LLM API call

Attempt it

A stop sequence is a string (or list of strings) the runtime checks against the streamed output during decode. As soon as a generated suffix matches one of them, generation halts and the matching tokens are stripped from what reaches the client. It is a content-conditional stopping mechanism used to terminate at a section marker, tool-call boundary, role tag (e.g. \n\nUser:), or any custom delimiter. Distinct from max_tokens, which is an unconditional hard cap, and from the model's own EOS token, which is a learned natural stop.

Your answer

**A stop sequence** is a string you pass into an LLM API call to tell the server: stop generating the moment the output ends with this. You can usually pass several stop sequences at once, and the server stops as soon as any of them matches. Mechanically, the runtime watches the tokens as they are produced. After each token, it checks whether the latest tail of the decoded text matches one of the stop strings. If yes, it halts generation immediately, strips the matching characters from the response, and closes the stream. If no, it continues to the next decode step. This is different from two other ways generation can end. `max_tokens` is an unconditional cap on output length, applied regardless of what was generated. The model's special EOS (end of sequence) token is something the model itself emits when it has finished its thought, learned during training. A stop sequence is content-conditional, defined by the caller, and triggered by what the model actually wrote. Typical use cases: stop at a section heading like `\n## ` so the model does not start a new section, stop at `\n\nUser:` so the model does not roleplay the user's next turn, or stop at a custom delimiter that ends a structured response.

**A stop sequence** is a content-conditional termination string (or list of strings) supplied by the caller. The decoder checks the current output suffix against the stop list at every step; the first match halts generation and the matching tokens are removed from the returned text. The check happens just outside the sampling loop, after each new token is appended to the output buffer. Most engines implement it as a suffix-match against the detokenized string, not against token IDs, because tokenizer boundaries do not line up with arbitrary stop strings (a stop like `</tool>` may straddle two BPE tokens). vLLM, TGI, and the OpenAI runtime all stream tokens out only up to the point where a match becomes possible, sometimes buffering one or two tokens to handle the case where the match completes on the next token. Three termination mechanisms coexist in one request. `max_tokens` is an unconditional integer cap; the runtime halts after that many output tokens regardless of content, and the response carries `finish_reason: length`. The model's learned EOS token is a special vocabulary entry the model emits when it considers the response complete; this produces `finish_reason: stop` with an empty stop string. A user stop sequence is the third option, producing `finish_reason: stop` with the matched string surfaced for debugging. Typical uses: terminate at a role tag (`\nUser:`, `\nHuman:`) to prevent the model from continuing a roleplayed dialogue, terminate at a structural delimiter (`</tool>`, `}` for a JSON object, `---` for a section break), or terminate at a custom marker in few-shot prompts that separate examples. OpenAI accepts up to four stop strings per request; Anthropic accepts a list with no documented cap. The cost is small and the lever is the cleanest way to bound generation by content.

Quick reference

TL;DR

A stop sequence is a string the server matches against decoded output during generation; when it matches, decoding halts and the matching tokens are stripped from the response.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a child reading a book aloud. You tell them: stop the moment you read the word `THE END`. They keep reading, watching for that exact phrase, and the instant they say it, they snap the book shut. A stop sequence is the same instruction handed to an LLM. The runtime watches every token as it streams out and compares the recent suffix against the list of stop strings you provided. The moment one matches, generation halts. The matching string itself gets snipped off before the response is returned, so you get a clean cut. People use this to end a tool call cleanly, to stop at a section heading, to prevent the model from pretending the user said something next, or to honor any custom delimiter their format uses.

Key concepts

Decoding max_tokens EOS Token LLM Serving

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define a stop sequence as a caller-supplied termination string, walk the decode-loop check against the output suffix, contrast with `max_tokens` and EOS, name the streaming lookahead subtlety, and give one production use case like role-tag or tool-call termination.

Real-world examples

Real products, models, and research that use this idea.

OpenAI Chat Completions accepts up to four `stop` strings; if any appears in the output, generation halts and `finish_reason` is `stop`.
Anthropic Messages exposes `stop_sequences` and reports the matched string in `stop_sequence` on the response for debugging.
Tool-use prompts often use `</tool>` or `\n\nUser:` as a stop to prevent the model from continuing past the structured boundary.
vLLM and Hugging Face TGI both implement stop-sequence matching against the detokenized output and buffer one to two tokens during streaming to disambiguate matches.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does a stop sequence work when the stop string spans two BPE tokens?

Sign in or go Pro to see the approach for this follow-up.

QWhy might a stop sequence cause noticeable pauses in a streamed response?

Sign in or go Pro to see the approach for this follow-up.

QHow do tool-use APIs use stop sequences internally?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating stop sequences with `max_tokens`. `max_tokens` is an unconditional length cap; a stop sequence fires only when specific content is generated. Both can apply to one request.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.