Rank these chat API cost levers from highest to lowest typical ROI

Order steps·Medium·4.0 · 0·~1 min·Asked atComet Ml Persistent Snorkel Ai·Relevant atAnthropic OpenAI

Attempt it

1Switch to a smaller model that still meets quality bar (e.g. 70B -> 8B with a quality eval)
2Turn on speculative decoding with a small draft model
3Quantize weights to INT4 (AWQ / GPTQ) on a self-hosted deployment
4Enable prompt caching on a repeated system prompt that fires on most requests
5Quantize the KV cache to FP8 on a self-hosted deployment

Cost levers fall into two families. The first family changes **what work the system does**, a smaller model, or skipping work via caching. The second family changes **how efficiently the same work runs**, quantization, KV-cache compression, faster decoding tricks. The first family almost always wins, because doing less work beats doing the same work faster. Switching from a 70B to an 8B model that passes your quality eval cuts compute and memory by roughly an order of magnitude. Nothing on the infra side comes close to a 10x reduction. So if your quality bar permits it, this is always the first lever to pull. Prompt caching is next on real production traffic. Most chat applications send the same system prompt, RAG context chunk, or few-shot prefix on nearly every request. Anthropic and OpenAI both let you cache that prefix at roughly one-tenth the input rate (Anthropic) or half rate (OpenAI). If 70% of your input is a stable prefix, you cut nearly that fraction of your input bill. INT4 weights and FP8 KV are real wins on self-hosted serving, but smaller. Speculative decoding mostly buys latency, not cost; the draft model adds compute even when it accelerates user-facing time.

**Cost-reduction ROI** is dominated by two structural moves and then a long tail of infrastructure tuning. Model substitution is the king lever because it changes the unit cost of every token by 5-20x. A serious quality eval often finds that an 8B or a frontier-mini tier serves 60-80% of production traffic with no measurable user impact, and the remaining hard cases can be routed to a larger model. Nothing in the infrastructure stack approaches a 10x cut. Prompt caching is second because it targets the input side, which on RAG and agentic workloads is often the dominant cost term. Anthropic charges 1.25x on the write and 0.1x on each read, with break-even at roughly the first read. OpenAI auto-caches prefixes of 1024+ tokens at ~0.5x. If your system prompt + tool schemas total 4K tokens and fire on every request, caching slashes that fraction of input nearly to zero. INT4 weight quantization on a self-hosted deployment is third, it cuts per-step HBM bytes ~4x and is essentially free in quality on most chat workloads if you use AWQ or GPTQ. FP8 KV cache is fourth and overlaps in benefit because both target decode-phase bandwidth; you stack them but the second adds less than the first did. Speculative decoding is last because its primary product is latency reduction. On batched serving it can even *raise* cost (you pay for both draft and verifier compute) and it only helps cost when the draft acceptance rate is very high and the verifier is underutilized.

TL;DR

Model substitution dominates because it cuts the workload by 5-20x. Prompt caching wins on whatever fraction of input repeats.

Memory aid

Easy to grasp

Picture a kitchen that is too expensive to run. The biggest single fix is hiring a cook who is one-tenth the salary but can still make every dish you actually serve. That dwarfs every other change. The next fix is realizing the chef keeps re-reading the same opening recipe ten times a day, so you laminate it and the chef glances at it for almost free. After that, you can buy thinner ingredients (lighter weight stock), use smaller storage containers (smaller KV cache), or pre-prep two dishes in parallel and throw out the bad one (speculative decoding). All three help, but they share the same constraint, the kitchen's serving counter, so their gains overlap and shrink each other. And the speculative trick mostly makes plates come out faster, not cheaper.

Key concepts

Prompt caching Quantization KV cache Speculative decoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cost-reduction ranking is the question every team faces after their first big invoice. The instinct is to reach for the most technically interesting lever, quantize, distill, write a custom CUDA kernel, and skip past the boring product-shaped levers like 'switch to a smaller model' and 'cache your system prompt'. The cost report keeps punishing teams for that instinct because the boring levers reduce the workload itself, while the interesting levers just make the workload run more efficiently. Doing less beats doing the same thing faster, and it does so by an order of magnitude.

The ranking, model substitution, prompt caching, INT4 weights, FP8 KV cache, speculative decoding, is not a fixed law; it is what the cost math says on a typical production chat workload with repeated system prompts, mixed difficulty traffic, and decode-heavy generation. On atypical workloads (latency-critical single-user demos, frontier-only reasoning tasks, embedding-heavy pipelines) the order shifts. The point of the ranking is the reasoning behind each step: which bottleneck the lever attacks, how much absolute win it produces, and whether it composes with the other levers or competes with them.

This deep dive walks each lever in order, names the bottleneck it attacks, gives the napkin math for the typical ROI, and ends with the composition rules, which levers stack cleanly, which fight each other, and what the residual cost looks like after all five have landed.

Lever 1: model substitution, the order of magnitude lever

Active parameters set both the FLOPs per forward pass and the HBM bytes streamed per decode step. Dropping from a 70B to an 8B model takes both down ~8x; dropping from a 405B to a 70B is another ~5x. On hosted-API pricing this is visible directly in the per-token rate: gpt-5 mini-class tiers run roughly 10-20x cheaper than frontier tiers, Anthropic Claude Haiku is roughly 10x cheaper than Claude Sonnet which is itself cheaper than Claude Opus, and the open-weight ecosystem mirrors the shape.

The lever is gated by quality. A serious substitution eval looks like this: take a stratified sample of real production traffic (a few hundred to a few thousand prompts spanning the hard cases), run completions on both the current and candidate models, score with a task-level rubric and an LLM-judge, and explicitly inspect the regressions. Aggregate accuracy hides distribution shifts; the eval has to oversample edge cases (long context, multi-turn tool use, multi-step reasoning) because that is where the smaller model breaks first.

When the eval passes for 60-80% of traffic, the right move is usually a router: small model on easy traffic, large model on hard. This captures most of the savings without sacrificing quality on the cases that need the big model. Teams that skip the eval either degrade quality silently or, more often, refuse to commit and leave the savings on the table for another quarter.

Lever 2: prompt caching, the input-side multiplier

Levers 3-4: weight and KV quantization, the bandwidth bottleneck

Lever 5: speculative decoding, a latency tool sometimes mistaken for a cost tool

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's GPT-4o-mini and Anthropic's Claude Haiku tiers exist precisely to capture the model-substitution savings without forcing teams to self-host.
Cursor, Perplexity, and many RAG-heavy products report 60-80% input-cost reductions from Anthropic prompt caching on stable system prompts and retrieval contexts.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build a model-substitution eval that doesn't leave 30% of regressions in the dark?

Run paired completions on a stratified sample of real production traffic, score with both task-level rubrics and an LLM-judge with a strong frontier model, and explicitly oversample edge cases (long context, tool calls, multi-step reasoning) since aggregate accuracy hides distribution shifts.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Putting quantization or speculative decoding first because they sound technical. Model substitution beats them by an order of magnitude when the quality bar allows, and prompt caching often beats them on real production traffic.

60 second bullets to scan on the way to the call.

Why model substitution dominates other cost levers on a typical bill
How prompt caching pricing works for Anthropic vs OpenAI

Primary sources. Browse if you want the original framing.