**max_tokens** caps the number of output tokens the model will produce in its reply. It does not limit the input length at all. The input is bounded by the model's context window, which is a separate (much larger) ceiling.
The mechanism is simple. During generation, the model produces one output token at a time. After each token, the inference loop checks: have we produced max_tokens tokens yet? If yes, halt. The output stops wherever the count is hit, even if that is mid-sentence or mid-JSON.
The response payload tells you whether the model stopped because of max_tokens (finish_reason=length on OpenAI, stop_reason=max_tokens on Anthropic) or for some other reason (a stop sequence matched, or the model produced its natural end of turn token). Production code logs this so you can spot prompts that are routinely getting truncated.
Why you set it. max_tokens is the safety net that bounds both cost and latency. Without a cap, a misbehaving prompt can generate until the context window runs out, costing dollars per call. Setting max_tokens to a sensible ceiling (200 for short answers, 1,000 to 2,000 for long-form, 4,000+ for extraction tasks) protects against runaway generation.
**max_tokens** is an output-length cap. It bounds the number of tokens the model is allowed to generate in its reply. The model produces tokens one at a time, and the moment the count hits max_tokens, decoding halts.
The critical distinction is from the context window. The context window caps the total tokens (input + output) the model can process in one call. max_tokens caps only the output slice. A 4K max_tokens setting on a 1M-token context window means the input can use up to 1M minus 4K tokens, and the model will produce at most 4K tokens of reply.
Three practical consequences follow. First, hitting max_tokens often means mid-sentence truncation. The model does not gracefully wrap up when it sees the limit approaching; it just stops. The response payload reports finish_reason=length (or stop_reason=max_tokens, depending on provider), and production code should treat that as a quality signal indicating the prompt either asked for too much or the format is unstable.
Second, max_tokens is a cost cap. Output tokens are typically 3 to 5 times more expensive than input tokens, so an uncapped reply can dominate the cost of a call. Setting max_tokens to the smallest value that fits the expected reply size is one of the easiest cost levers in production.
Third, max_tokens and stop sequences are complementary. Stop sequences express where to halt by content; max_tokens expresses where to halt by length. Production prompts set both, with stop sequences as the intent and max_tokens as the safety net.
Quick reference
TL;DR
max_tokens caps the number of OUTPUT tokens the model will generate; it does not limit input length (the context window does that).
Imagine you ask a friend a question and add 'reply in at most 100 words'. That instruction does not say anything about how long your question was. You can ask a 5,000-word question with the same 100-word reply limit. max_tokens is that reply limit, except it counts tokens instead of words. Your input can be as long as the model's context window allows. The cap only applies to what the model writes back. If the model is mid-sentence when it hits the cap, it just stops, even mid-word. The reply will look truncated. The way to detect this is to check the finish_reason field in the response, which says 'length' when max_tokens fired.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Open-ended creative writing where you want the model to find its own natural ending Hard-capping max_tokens too low produces mid-sentence truncation; set it generously and rely on the model's natural end of turn token plus a stop sequence.
Extraction or structured-output tasks where you know the exact JSON schema max_tokens alone does not guarantee a clean format; pair it with a stop sequence on the closing tag or use JSON mode for schema enforcement.
Streaming chat UX where you want the reply to flow until the model is done An overly tight max_tokens cap interrupts the natural pacing of a streamed reply; size it to the maximum useful reply length for the use case.
Quick scan · 2–4 min
Everything important, quickly.
6 min: define max_tokens as output-only cap, contrast with context window, walk through mid-sentence truncation, name finish_reason reporting, connect to cost and latency control.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's chat completions API uses max_tokens (or max_completion_tokens on newer models) to cap reply length and reports finish_reason='length' when it fires.
2Anthropic's Claude messages API uses max_tokens as a required parameter (no default) and reports stop_reason='max_tokens' on the response when the cap is hit.
3Cursor and Cody log stop_reason for every code-completion call; a spike in length terminations triggers a prompt-template review.
4Production summarization pipelines set max_tokens proportional to expected summary length (often 200 to 500) rather than the model's maximum, both for cost and for forcing brevity.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the difference between max_tokens and max_completion_tokens in OpenAI's newer API, and why did they introduce the latter?
Sign in or go Pro to see the approach for this follow-up.
QHow does max_tokens interact with prompt caching for cost optimization?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the inference engine halt exactly at max_tokens rather than wrapping up the current sentence first?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing max_tokens with the context-window limit; max_tokens caps the reply length only, the context window caps total input plus output.