What's the highest-leverage cost optimization for a production LLM app spending $10k+/month on tokens, BEFORE prompt-level token trimming? · Zenaique

Attempt it

Quick reference

TL;DR

Model routing per task complexity beats prompt trimming because it changes the per token price tier rather than nibbling at token count.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you have a team of engineers and you send every task to the most senior one. Of course your wage bill is high. The fix is not to ask the senior to type fewer words. The fix is to send the simple tickets to a junior who costs a fraction as much and only escalate the hard ones. LLM apps work the same way. Routing simple tasks like classification or extraction to a cheap, fast model and reserving the flagship for tasks that genuinely need it cuts the bill by half or more, long before you start arguing over which word in the system prompt to delete.

Key concepts

Prompt Engineering Inference Optimization LLM System Design

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: price as the dominant multiplier + routing as the first lever + prompt caching second + trimming third + how to keep quality stable across tiers.

Real-world examples

Real products, models, and research that use this idea.

Anthropic exposes prompt caching for Claude Opus 4.7 and Haiku, cutting input cost roughly tenfold on cache hits and changing the economics of agent loops.
OpenAI ships Cached prefix discounts on GPT-5.5 and Structured Outputs, which together let teams route classification to GPT-5-mini and reserve GPT-5.5 for synthesis.
Perplexity routes simple lookups to smaller open-weight models hosted on its own infrastructure and reserves frontier models for complex multi-source synthesis.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a request-complexity router?

Sign in or go Pro to see the approach for this follow-up.

QWhen does prompt caching beat model routing?

Sign in or go Pro to see the approach for this follow-up.

QHow do you keep quality stable while routing?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Starting cost work by trimming the system prompt token by token, when the dominant multiplier is the per token price of the flagship model you are still sending every request to.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.