Model routing per task complexity beats prompt trimming because it changes the per token price tier rather than nibbling at token count.
Imagine you have a team of engineers and you send every task to the most senior one. Of course your wage bill is high. The fix is not to ask the senior to type fewer words. The fix is to send the simple tickets to a junior who costs a fraction as much and only escalate the hard ones. LLM apps work the same way. Routing simple tasks like classification or extraction to a cheap, fast model and reserving the flagship for tasks that genuinely need it cuts the bill by half or more, long before you start arguing over which word in the system prompt to delete.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cost on a production LLM app is one of those problems where the first instinct is usually wrong. The instinct is to look at the prompt, count tokens, and start trimming. That work is real, but it is the last fifteen percent. The first sixty percent lives in which model the request hits and how much of the prompt is reused across requests.
This question is framed for an app spending around ten thousand dollars a month, but the same logic scales up and down. The structure of LLM pricing is multiplicative: cost is the product of token count and per token price, summed across traffic. Token count moves by maybe a third under aggressive trimming. Per-token price moves by a factor of five to fifteen across model tiers. The math forces a clear order of operations.
This deep dive walks through that order: routing first, caching second, trimming third. It names the production patterns and the failure modes that show up when teams skip the first two steps.
The math behind the order of operations
Total cost can be written as sum over traffic of (tokens_in * price_in + tokens_out * price_out). Each of the four factors is a lever. Most teams instinctively pull tokens_in first by editing the system prompt, because that is the lever they own most directly.
The problem is that the range of tokens_in is bounded. A well-written system prompt is already a few hundred tokens. You might cut it in half with effort, which is a thirty-percent reduction. The range of price_in is much larger: flagships price at roughly five to fifteen times their smaller siblings, sometimes more for the deepest tiers. Moving a workload one tier down is a five to fifteen times reduction on those tokens.
This math is why the order of operations is price, repetition, count. Pull the lever with the largest range first.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic exposes prompt caching for Claude Opus 4.7 and Haiku, cutting input cost roughly tenfold on cache hits and changing the economics of agent loops.
- OpenAI ships Cached prefix discounts on GPT-5.5 and Structured Outputs, which together let teams route classification to GPT-5-mini and reserve GPT-5.5 for synthesis.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a request-complexity router?
Start with cheap heuristics (length, code presence, format); add a small classifier or LLM judge on hard slices; instrument escalation rates and quality deltas.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Starting cost work by trimming the system prompt token by token, when the dominant multiplier is the per token price of the flagship model you are still sending every request to.
60 second bullets to scan on the way to the call.
Why model price is multiplicative vs token count
How to design a request-complexity classifier
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.