When you call an LLM API, you usually want two kinds of termination control. The first is a safety cap: stop after some maximum number of generated tokens no matter what. The second is a semantic cap: stop when the model finishes saying a particular thing.
**`max_tokens`** is the safety cap. You pass an integer like `max_tokens=500`, and the server commits to generating at most 500 completion tokens before halting. This is an absolute ceiling. It does not matter what the model is saying when it hits the limit. Generation stops, the partial output is returned, and the response is flagged with `finish_reason: length` so your code knows the answer is truncated.
**`stop`** is the semantic cap. You pass a list of strings, like `stop=['\n\nHuman:', '###']`. The server watches the output as it streams and the moment it produces any of those strings, decoding halts. The matched string itself is NOT included in the returned output. If the model never emits any of the stop strings, this parameter never fires, and decoding continues until `max_tokens` or the context limit.
In practice you use both. `max_tokens` bounds your worst-case spend. `stop` gives the model a clean off-ramp so it can end at the right narrative or structural boundary instead of getting cut off mid-word.
**`max_tokens`** is an unconditional integer cap on completion length. The server keeps a running token count and halts the decode loop the instant that count is reached. The response carries `finish_reason: length` so the client knows it was truncated rather than completed.
**`stop`** is a conditional list of string sentinels. As tokens stream out of the decoder, the server scans the rolling buffer for any of the supplied strings. The moment one matches, generation stops. Critically, the matched sentinel is stripped from the returned output, so the response ends just before the stop string. If no sentinel ever appears, `stop` has no effect and decoding runs until some other limit binds.
Production code uses both, for different reasons. `max_tokens` is a financial guardrail: every API call has a known maximum cost, which is essential for billing predictability and abuse mitigation. `stop` is a structural guardrail: it lets the model end naturally at a meaningful boundary, like the end of a JSON object, a turn marker in a chat template, or a delimiter in a structured prompt.
The failure mode to avoid is relying on `stop` alone. If the model never emits your sentinel string, generation continues until `max_tokens` or the model's context limit, which can be tens of thousands of tokens and dollars of cost per call. Always set `max_tokens` even when you have a tight `stop` list.
Quick reference
TL;DR
`max_tokens` is a hard numeric ceiling that always fires; `stop` is a list of strings that only halts generation when the model happens to emit one.
Picture two ways to tell a storyteller when to stop. The first way is a kitchen timer. You set it to 60 seconds, and no matter what the storyteller is saying when it dings, they have to put down the microphone. That is `max_tokens`. The second way is a code word. You say 'when you reach the end of the story, say THE END, and we will know to clap.' If the storyteller never says the code word, you keep listening forever. That is `stop`. Production systems use both. The timer protects you from an endless tale that runs up the bill. The code word lets the storyteller end naturally at the right narrative beat instead of getting cut off mid-sentence by the timer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Setting `max_tokens` extremely low for a complex reasoning task The model will truncate mid-thought and return a useless partial answer; size the cap to the realistic worst case for the task.
Using `stop` with a sentinel the model has never been trained or prompted to emit The condition will never fire, leaving `max_tokens` as the sole termination, which defeats the structural-boundary intent.
Relying on `stop` to filter unsafe content mid-generation Stop sequences are termination signals, not content filters; safety filtering needs a dedicated moderation layer.
Quick scan · 2–4 min
Everything important, quickly.
3 min: numeric cap vs string sentinel + unconditional vs conditional firing + cost-ceiling vs structural-boundary roles + the always-set-max_tokens rule + streaming tail-buffering subtlety.
Side-by-side
Aspect
max_tokens
stop
Type
Integer
List of strings
Trigger condition
Unconditional, count reached
Conditional, string matched
Cost guarantee
Yes, hard ceiling
No, depends on model output
Output includes trigger
N/A, counts tokens
No, sentinel is stripped
Primary use
Bound spend and latency
Shape structured boundaries
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's `/v1/chat/completions` endpoint accepts both `max_tokens` and `stop` (up to 4 strings); responses include `finish_reason: 'length'` or `'stop'` to indicate which fired.
2Anthropic's Messages API uses `max_tokens` (required) and `stop_sequences`; responses include `stop_reason: 'max_tokens'` or `'stop_sequence'`.
3LangChain agents historically used `stop=['Observation:']` so the LLM would halt after emitting the next thought-action and let the executor run the tool.
4vLLM and TGI both implement server-side stop-string detection with tail-buffering so streamed chunks never leak partial sentinels to the client.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does `stop` work with multi-token strings in a streaming response?
Sign in or go Pro to see the approach for this follow-up.
QIf both `max_tokens` and `stop` are configured, which one wins?
Sign in or go Pro to see the approach for this follow-up.
QWhy might you set `max_tokens` higher than you actually expect to use?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Relying on `stop` alone to bound cost. If the model never emits the sentinel string, decoding runs until the model's context limit, which can cost dollars per request.