Output tokens dominate cost and latency, so cut them with a tight max_tokens cap, structured outputs, and stop sequences — not by changing temperature or language.
Imagine paying a writer by the word, where each word also takes time to write down. To spend less, you don't change their mood or ask them to use fancier vocabulary — that doesn't make the letter shorter. Instead you tell them: stop at one page, fill in this form instead of writing an essay, and put down your pen the moment the answer is done. Cutting output tokens works the same way — cap the length, ask for a compact form, and stop as soon as the answer is complete.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Ask any team running an LLM product where the bill comes from and the honest answer is usually 'the output side'. Output tokens are priced higher than input on every major API, and because of how decoding works, each one also sits on the latency critical path. That makes 'how many tokens does the model emit' one of the most controllable cost variables in the whole system.
The trap this question is built around is confusing two different kinds of knobs. Some settings change how the answer sounds — its randomness, its phrasing, its language. Others change how much the model emits. Only the second category cuts cost, and the distractors are specifically chosen to look like cost levers while belonging to the first category.
This deep dive explains why output tokens dominate, walks through each of the three real levers and exactly how it shrinks emitted tokens, then dissects why temperature and language choice are the wrong tools — so you can sort any proposed 'cost optimization' into the right bucket.
Why output tokens are the expensive, slow side of a call
An LLM call has two phases with very different cost profiles. Prefill ingests your prompt: it processes every input token in parallel, so it's compute-bound and fast per token. Decode generates the answer one token at a time, each step depending on the last, so it's sequential and memory bandwidth bound.
That asymmetry is why output tokens hurt more. On the dollar side, providers price output tokens 2-5x higher than input because generation is the expensive phase. On the latency side, every output token is a separate forward pass on the critical path — 200 output tokens means 200 sequential steps, while 200 input tokens were processed together.
So reducing emitted tokens is a two for one: you pay less per call and the call returns faster. Reducing input tokens helps the bill too, but the output side is where the leverage concentrates, and it's the side fully under your control via API parameters and prompt design.
This is the mental model the whole question rests on. A 'cost lever' that doesn't reduce the number of tokens the model emits isn't reducing the expensive thing. Hold that test in mind and the right answers fall out.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Lever | Reduces emitted tokens? | Mechanism |
|---|---|---|
| Tight max_tokens cap | Yes | Hard ceiling on generation length |
| Structured output | Yes | Compact fields/IDs replace prose |
| Stop sequences | Yes | Halt decode when answer is complete |
| Raise temperature | No | Changes randomness, not length |
| Wordier language | No | Increases tokens for same meaning |
Real products, models, and research that use this idea.
- OpenAI and Anthropic price output tokens 2-5x higher than input, making output reduction the highest-leverage cost lever.
- OpenAI structured outputs / JSON mode and Anthropic tool-use schemas force compact field output instead of prose.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does cutting output tokens help latency more than cutting input tokens of the same count?
Prefill processes input in parallel and is compute-bound; decode emits output one token at a time and is memory bandwidth bound, so each output token sits on the sequential critical path.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for temperature to control output length — temperature changes randomness, not how many tokens the model emits, so it doesn't cut cost.
60 second bullets to scan on the way to the call.
Why output tokens dominate cost and latency over input tokens
How a realistic max_tokens cap differs from the API default
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.