Pick every lever that cuts reasoning spend without hurting hard query accuracy

Multi-select·Hard·4.0 · 0·~1 min·Asked atFractal Analytics Gnani Swiggy

Attempt it

Four of the six options are real cost levers. Two are traps. **Tier reasoning effort by request class.** Setting a global high effort spends thinking tokens on easy traffic that gains nothing. Tiering effort, low for FAQs, medium for triage, high for genuinely hard work, cuts spend without touching the hard-query path. **Escalation router with fast model first.** Most production traffic does not need a reasoning model at all. Sending traffic to a fast model first and only escalating low-confidence answers to the reasoning tier saves the majority of the reasoning-tier bill while preserving the deep path for queries that need it. **Prefix caching.** Long system prompts and stable document context cost the same input tokens on every call. Most providers offer a discounted rate for cached prefixes (often 50-90% off). This is a pure win, no accuracy impact. **Distillation for the easy slice.** If a high-volume slice of your reasoning traffic is structured and repetitive, fine-tuning a smaller model on outputs from the reasoning tier can move that slice off the expensive path entirely. The smaller model never serves hard queries; those still go to the reasoning tier.

**Four genuine levers, two traps.** **Effort tiering.** A global high default sprays thinking tokens across all traffic. Setting effort per request class (FAQs minimal, triage low or medium, hard tickets high) preserves accuracy where it matters and cuts the spend on the rest. **Escalation routing.** Fast model first, reasoning tier only on low-confidence or flagged-hard queries. Shifts the cost distribution sharply because the fast tier handles the majority of traffic at a fraction of the cost, and the reasoning tier still owns the genuinely hard slice. **Prefix caching.** Stable system prompts and shared document context bill at a discounted rate when cached. No accuracy impact, often the largest single saving on a long-context deployment. **Distillation.** Fine-tuning a smaller model on the reasoning tier's outputs handles the high-volume easy slice without ever touching the deep path. The reasoning tier still serves hard queries. **Why the two traps are dangerous.** Cutting `max_tokens` below the thinking budget does not 'just save cost'; thinking and visible tokens share one completion window, and a low cap truncates the trace mid-reasoning. Hard queries are exactly where reasoning runs longest, so this option targets the failure mode the reasoning tier was meant to handle. Universal 16-sample majority voting is the opposite of cost control. It multiplies output-billed tokens by 16x on every request, including the FAQs that did not need any thinking. Self-consistency has value on closed-form hard problems with a verifier; deploying it as a default on all traffic burns the budget.

TL;DR

Effort tiering, escalation routing, prefix caching, and distillation cut spend without breaking hard queries; truncating max_tokens and universal majority voting are traps.

Memory aid

Easy to grasp

Picture cost control as choosing which dials in a power plant to turn. Some dials lower the bill without dimming the lights: pick smaller bulbs for empty rooms, only switch the big floodlight on when you actually need it, reuse the same warm power feed across appliances, and let a cheap LED handle the corridors you walk through most. Other dials look like savings but cause blackouts: unplugging the breaker midway through running the operating room is not a saving, it kills the procedure. Running every appliance at full power because more must be better is the opposite of efficiency. The trick is telling the safe dials from the dangerous ones.

Key concepts

Reasoning Models

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Reasoning-tier cost control is a distribution-shaping exercise, not a single-knob optimization. The bill is the integral of per-call cost over the traffic distribution, and good cost work changes the distribution: it shifts easy traffic off the expensive path, discounts the shared portions of context, and reserves the heavy machinery for traffic that actually needs it. The traps in cost work are knobs that look like savings on paper but either break the hard-query path or move spending in the wrong direction on the wrong slice.

The stem offers six knobs and asks which ones move spend down without hurting accuracy on hard queries. Four are genuine levers. Two are traps that, deployed at scale, will either crater accuracy on the high-value slice or actively increase spend.

The four levers, each targeting a different sub-distribution

Effort tiering. Production reasoning models expose an effort parameter (OpenAI's reasoning_effort, Anthropic's thinking budget, Gemini's thought budget, vendor-specific equivalents on DeepSeek-R1 deployments). A global high default sprays thinking tokens on every request. Tiering by request class, minimal for FAQs, low for triage, medium for moderately hard, high for genuinely hard, shifts the spend distribution: the bulk of traffic now consumes far less thinking, while the hard slice still gets the deep deliberation it needs.

Escalation routing. A cheap classifier or a fast-model first pass handles the majority of requests. Only requests that fail a confidence threshold (or are explicitly flagged as hard by the classifier) escalate to the reasoning tier. This is the single highest-leverage lever in most deployments because it removes whole categories of traffic from the reasoning tier entirely, not just trims their per-call cost. A typical production split is 60-80% fast, 20-40% reasoning, with the exact mix depending on product surface.

Prefix caching. Modern providers offer steep discounts (often 50-90%) on input tokens that match a cached prefix. Long system prompts, shared retrieved-document context, and stable tool-use schemas all qualify. On long-context agentic deployments this is frequently the largest single cost saving, sometimes cutting the input-side bill by an order of magnitude with zero accuracy impact. The engineering cost is minor: structure prompts so the stable prefix comes first, set the cache control field correctly.

Distillation for the easy slice. Fine-tuning a smaller model on outputs from the reasoning tier produces a cheap serving model for the high-volume structured-repetitive slice. The pattern, exemplified by DeepSeek-R1-Distill-Qwen-7B and many similar releases through 2026, moves a substantial fraction of traffic off the reasoning tier entirely. Distillation has a fixed cost (training, eval, ops) so it earns out only when the easy slice is large enough to justify it; on consumer-scale deployments it usually is.

Trap one: max_tokens below the thinking budget

Trap two: universal 16-sample majority voting

Ranking the four levers by typical impact

Guardrails that turn cost work into safe cost work

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI o-series exposes reasoning_effort settings; production teams tier them per request class to control spend.
Anthropic Claude Opus 4.7 prompt-caching gives a 90% discount on cached input tokens, often the largest single lever on long system prompt deployments.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you rank the four levers by expected dollar impact on a typical deployment?

Usually escalation routing first (shifts traffic class entirely), then prefix caching on long-context deployments, then effort tiering, then distillation. The exact order depends on traffic mix and context shape.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating any cost-reducing knob as safe. Some cuts break the hard path you were trying to preserve; cost work has to be paired with accuracy guarantees on the slice that matters.

60 second bullets to scan on the way to the call.

List four cost levers that preserve hard-query accuracy
Identify two anti-pattern cost knobs and explain why each fails

Primary sources. Browse if you want the original framing.