Soft directives like 'be brief' are surprisingly weak (~30-50% non-compliance); max_tokens is hard but truncates mid-sentence. Combine both and eval the output-length distribution.
Picture asking a chatty friend to keep their answer short. They mean well, but they get carried away and ramble for a full minute anyway. Now picture setting a kitchen timer for 30 seconds; the timer cuts them off, but often mid-word. The answer is to do both: ask for brevity AND set the timer, with the timer set generously enough that they finish their sentence on normal questions but get cut off on truly runaway ones. Then check a recording of ten real conversations to see if the strategy is actually working.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Output-length control is one of those prompt-engineering surfaces that looks simple from a tutorial and bites in production. The two obvious tools, a soft directive in the prompt and the max_tokens parameter on the API call, each have failure modes the other does not. Neither is sufficient alone, and teams who pick just one tend to discover the gap when a UX regression lands in front of users.
The deep dive below walks through why soft directives are weaker than they look, why max_tokens does not gracefully wrap up, the layered pattern that production teams converge on, and the eval discipline that catches drift before it ships.
Why soft directives leak
Soft length directives like 'be brief', 'answer in under 100 words', or 'one paragraph maximum' are inference-time nudges on a model whose post-training rewards thoroughness. Instruction-tuned models go through RLHF and similar processes where human raters consistently prefer complete, well-explained, well-cited responses over terse ones. The model learns a length prior that biases long.
At inference, a brevity directive fights that learned prior. Sometimes the model complies; often it does not, especially on questions where it thinks more explanation actively helps the user. Production telemetry from teams instrumenting length compliance typically shows 30-50% non-compliance on simple brevity directives, with worse rates on complex reasoning prompts.
This is not a bug; it is the model doing what it was trained to do. The implication for prompt engineering is that soft directives shift the central tendency of the response-length distribution but leave a heavy right tail. If your UX cannot tolerate that tail, you need a second tool.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Voice assistants like Alexa and Google Assistant cap responses tightly because text to speech latency scales with length; they layer prompt directives with hard caps and tight evaluation.
- GitHub Copilot Chat uses soft brevity directives in its system prompt and a generous max_tokens, then eval-checks response length against UX guardrails.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do instruction-tuned models systematically over-produce length even when told to be brief?
RLHF and helpfulness training reward thoroughness, citations, and explanations. The base policy biases long; brevity directives at inference fight that prior and only partially succeed.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Relying on 'be brief' alone and being surprised when production outputs run 3x longer than asked; the directive shapes intent, not behavior, and 30-50% of responses ignore it.
60 second bullets to scan on the way to the call.
Why soft directives have ~30-50% non-compliance rates
What goes wrong when max_tokens is set too tight
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.