Is trimming the prompt usually the biggest cost lever on a chat endpoint?

It's a natural assumption that a long prompt is the expensive part of an LLM call, since the prompt is often the biggest chunk of text. But on most APIs that's backwards. Providers charge two rates: a lower one for input tokens (your prompt) and a higher one for output tokens (what the model writes). Output usually costs **three to five times more per token**. So even when the prompt has more tokens, the output side often drives the bill. There's a second reason output matters more. The model writes output one token at a time, in sequence, so long answers also make the user wait longer. The prompt, by contrast, gets read in one fast pass. Put together, the bigger lever is usually the output: cap it with a sensible `max_tokens`, ask for structured or short responses, request a code or label instead of a paragraph. Trimming the prompt still helps and is worth doing, but per token it's the cheaper side, so it's rarely where the biggest savings are.

Trimming the prompt is rarely the biggest cost lever on a chat endpoint. The reason is a **price asymmetry**: output tokens are billed several times higher than input tokens on virtually every 2026 rate card, commonly 3-5×. Write the per-call cost as two terms: $$\text{cost} = t_{in}\,p_{in} + t_{out}\,p_{out}, \quad p_{out} \approx 3\text{–}5 \times p_{in}$$ Because `p_out` is so much larger, the output term often dominates. That holds even when there are fewer output tokens than input tokens. Optimizing the smaller-priced side first is the wrong order. Output also dominates **latency**. It's generated autoregressively — one sequential forward pass per token — while the prompt is processed in a single parallel prefill. So long outputs cost more money and more wall-clock time simultaneously, which makes output reduction a two for one win. The practical levers all target output. Use a tight `max_tokens`, plus structured outputs or function calling so the model emits compact JSON instead of prose. Ask for a code or classification rather than an explanation, and stop early. Prompt trimming and prompt caching still matter, but the highest-leverage first move on a chatty endpoint is almost always shrinking what the model writes back.