How model routing turns into a structural lever for LLM cost control

Flashcard·Easy·4.0 · 0·~30s·Asked atMongodb Neptune Ai Reliance Jio

Attempt it

Model routing classifies each request and sends it to the cheapest model that can satisfy it. Simple intent triage and reformatting go to a cheap small model (gpt-4o-mini, Haiku, Gemini Flash, an open-weights 8B); complex reasoning and code generation go to a frontier model; only the residual goes to the most expensive option.

The lever is structural because the price gap between tiers is large (often 10 to 30 times per token), so routing 70 percent of traffic to the cheap tier cuts the bill far more than any negotiable discount. The router itself is usually a small classifier or a rules engine on intent plus input length, sometimes a learned router scored against a held-out set. The risk is regressions when the cheap model takes a request it cannot handle, which is why the router needs its own eval gate.

Your answer

An **LLM bill** is the per-token cost of every model call. The price differs sharply between providers and between tiers within a provider. A frontier model can cost 10 to 30 times more per token than a small fast model from the same provider. **Model routing** is the practice of looking at each incoming request and choosing which model to send it to, based on what the request actually needs. Simple requests (rephrase this sentence, classify this intent, extract this date) go to a cheap small model. Hard requests (write working code, reason across long context, follow complex instructions) go to a frontier model. The cheapest model that can satisfy the request wins. The reason this is more powerful than negotiating discounts is the size of the tier gap. A 30 percent volume discount cuts your bill by 30 percent. Routing 70 percent of traffic from the frontier to a tier that costs 20 times less can cut your bill by 60 percent or more. The structural lever wins. The router itself is usually a small classifier, a rules engine, or even an LLM-based router. It has its own accuracy bar and its own eval gate, because mis-triage to the cheap tier produces a bad answer instead of a regression.

**Model routing** classifies each request and dispatches it to the cheapest model that can satisfy the quality bar. The lever exploits a structural price gap: in 2026 a request through gpt-4o-mini, Haiku, Gemini Flash, or a self-hosted 8B open-weights model costs 10 to 30 times less per token than the same call through gpt-4o, Claude Opus 4.5, or Gemini 2.5 Pro. **Why it dominates discounts.** A volume contract might shave 20 percent off list. Routing 70 percent of traffic to a tier that is 20 times cheaper cuts the bill by 60 to 70 percent. The two levers compose, but the structural one is the only one that scales with traffic instead of negotiation leverage. **Router shapes.** The router is most often a rules engine on intent plus input length (cheap, fast, easy to audit), sometimes a small classifier fine-tuned on labeled requests, sometimes an LLM-based router using a tiny model to call the routing decision. The router itself runs on every request and adds latency, so it has to be cheap. **The eval gate.** Mis-triage to the cheap tier produces a bad answer rather than an error, which is a subtle failure mode. The router needs its own held-out evaluation set with the same regression-gate discipline as the prompts: per-tier acceptance rates, mis-triage rate by intent, escalation rate from cheap to expensive when the cheap answer is rejected by a downstream check.

TL;DR

Model routing classifies each request and sends easy ones to a cheap small model and hard ones to the frontier model, exploiting a 10 to 30 times price gap that no discount could match.

Memory aid

Easy to grasp

Imagine you run a help desk and every question comes in through one queue. A senior expert costs you a lot per minute; a junior costs ten times less. Most of the questions are simple lookups, so handing everything to the senior is wasteful. You install a triage clerk at the front who reads the question, sends the simple ones to the junior, the medium ones to the regular agent, and only the hard ones to the senior. The clerk is fast and cheap. Your bill drops sharply because most of the work was overkill before. The catch is that when the clerk mis-triages a hard question to the junior, the user gets a bad answer. So you keep watching the clerk's accuracy on a test set and retrain it when it slips.

Key concepts

LLMOps Inference optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cost in LLMOps is unlike cost in classic web ops. A single API call can cost more than thousands of typical web requests. The cost line on the dashboard is dominated by which model answered, not by how many requests were served. That makes model routing the single highest-leverage cost lever in the stack, and the one a cost-aware team implements first.

This explanation works through the intuition behind why routing dominates discounts, the three common router architectures, the escalation cascade that bounds the worst case, and the eval discipline that keeps the router from silently regressing.

Why the structural lever beats the discount lever

Provider price lists are not smooth. Within a single provider, the small tier and the frontier tier sit at roughly an order of magnitude apart. In 2026 numbers, Claude Haiku is around $0.25 per 1M input tokens; Claude Opus 4.5 is around $15. gpt-4o-mini is $0.15; gpt-4o is around $2.50. Gemini Flash is similarly an order of magnitude below Gemini 2.5 Pro.

A volume discount or committed-use contract might shave 20 to 30 percent off list price. Moving the request itself from a $15 tier to a $0.25 tier is a 60x reduction. The two levers compose multiplicatively if you can negotiate on the cheap tier too, but the structural one is the only one that scales with traffic instead of with negotiation leverage.

The corollary is that routing 70 percent of requests to the cheap tier roughly halves the bill, even if the request mix means the expensive tier still answers some of the calls. That kind of impact is impossible to extract from a discount conversation.

Three router architectures

The escalation cascade

Eval discipline for the router itself

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LiteLLM Router with a fallback chain (Haiku then Sonnet then Opus) is the canonical implementation in 2026.
Vercel AI Gateway exposes routing rules by token-budget and latency-budget per request.

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how you would build the escalation cascade so a mis-triage gets caught and retried on the next tier up.

Cheap tier answers first; a fast self-check (regex on output shape, a small judge model scoring confidence or faithfulness, or a structured-output schema validator) decides accept or escalate. On escalate, the request re-runs on the next tier with the original input. Track escalation rate per intent; rising escalation is a router-drift signal that triggers retraining.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the router as build once forget. It is its own model with its own failure modes (mis-triage to the cheap tier on a hard request), and it needs an eval gate just like the prompts it routes.

60 second bullets to scan on the way to the call.

The order of magnitude price gap between small and frontier tiers in 2026
Three router architectures: rules, classifier, LLM as router

Primary sources. Browse if you want the original framing.