Design a routing tier that sends only the hard queries to a frontier model

The idea behind **model routing** is simple: most queries are easy, so don't pay frontier-model prices for all of them. Run a cheap, fast default model — like GPT-4o-mini — for the bulk of traffic, and send only the hard ones up to a frontier model. The key design question is what decides escalation. Common signals are a small classifier that scores query difficulty, the cheap model's own confidence in its answer, simple heuristics like length or task type, or a verifier that checks the cheap answer and retries on a bigger model if it looks wrong. This captures most of the cost savings because the expensive model only runs on the small slice that needs it. The main risk is the router getting it wrong — sending a hard query to the weak model, or over-escalating and losing the savings. The router also adds its own latency and cost, so it has to be cheap and tuned against real labeled examples.

The design is a two-tier system: a cheap default model handles the majority of requests, and a router escalates the hard or low-confidence ones to a frontier model. The escalation signal is the heart of it. There are four common options. The first is a lightweight **classifier** trained to predict query difficulty. The second is the small model's own confidence or uncertainty, where low confidence triggers escalation. The third is explicit heuristics like prompt length or detected task type. The fourth is verify and retry, where you run the cheap model first, check its answer with a verifier, and re-run on the frontier model only on failure. These differ in cost and accuracy. A classifier is cheapest per query. Verify and retry is most accurate but pays for two calls on escalated queries. The value is capturing most of the cost-quality frontier: you pocket the savings on the easy majority while protecting quality on the hard tail. The main risks are router error and the cost of the routing step itself. Router error means misrouting hard queries to the weak model, or over-escalating and erasing the savings. So the router must be cheap relative to the call it gates, and tuned against a labeled set so you can pick an operating point that balances cost against quality.