Rank these chat-API cost levers from highest to lowest typical ROI

Attempt it

1Switch to a smaller model that still meets quality bar (e.g. 70B -> 8B with a quality eval)
2Turn on speculative decoding with a small draft model
3Quantize weights to INT4 (AWQ / GPTQ) on a self-hosted deployment
4Enable prompt caching on a repeated system prompt that fires on most requests
5Quantize the KV cache to FP8 on a self-hosted deployment

Cost levers fall into two families. The first family changes **what work the system does**, a smaller model, or skipping work via caching. The second family changes **how efficiently the same work runs**, quantization, KV-cache compression, faster decoding tricks. The first family almost always wins, because doing less work beats doing the same work faster. Switching from a 70B to an 8B model that passes your quality eval cuts compute and memory by roughly an order of magnitude. Nothing on the infra side comes close to a 10x reduction. So if your quality bar permits it, this is always the first lever to pull. Prompt caching is next on real production traffic. Most chat applications send the same system prompt, RAG context chunk, or few-shot prefix on nearly every request. Anthropic and OpenAI both let you cache that prefix at roughly one-tenth the input rate (Anthropic) or half rate (OpenAI). If 70% of your input is a stable prefix, you cut nearly that fraction of your input bill. INT4 weights and FP8 KV are real wins on self-hosted serving, but smaller. Speculative decoding mostly buys latency, not cost; the draft model adds compute even when it accelerates user-facing time.

**Cost-reduction ROI** is dominated by two structural moves and then a long tail of infrastructure tuning. Model substitution is the king lever because it changes the unit cost of every token by 5-20x. A serious quality eval often finds that an 8B or a frontier-mini tier serves 60-80% of production traffic with no measurable user impact, and the remaining hard cases can be routed to a larger model. Nothing in the infrastructure stack approaches a 10x cut. Prompt caching is second because it targets the input side, which on RAG and agentic workloads is often the dominant cost term. Anthropic charges 1.25x on the write and 0.1x on each read, with break-even at roughly the first read. OpenAI auto-caches prefixes of 1024+ tokens at ~0.5x. If your system prompt + tool schemas total 4K tokens and fire on every request, caching slashes that fraction of input nearly to zero. INT4 weight quantization on a self-hosted deployment is third, it cuts per-step HBM bytes ~4x and is essentially free in quality on most chat workloads if you use AWQ or GPTQ. FP8 KV cache is fourth and overlaps in benefit because both target decode-phase bandwidth; you stack them but the second adds less than the first did. Speculative decoding is last because its primary product is latency reduction. On batched serving it can even *raise* cost (you pay for both draft and verifier compute) and it only helps cost when the draft acceptance rate is very high and the verifier is underutilized.

Quick reference

TL;DR

Model substitution dominates because it cuts the workload by 5-20x. Prompt caching wins on whatever fraction of input repeats.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a kitchen that is too expensive to run. The biggest single fix is hiring a cook who is one-tenth the salary but can still make every dish you actually serve. That dwarfs every other change. The next fix is realizing the chef keeps re-reading the same opening recipe ten times a day, so you laminate it and the chef glances at it for almost free. After that, you can buy thinner ingredients (lighter weight stock), use smaller storage containers (smaller KV cache), or pre-prep two dishes in parallel and throw out the bad one (speculative decoding). All three help, but they share the same constraint, the kitchen's serving counter, so their gains overlap and shrink each other. And the speculative trick mostly makes plates come out faster, not cheaper.

Key concepts

Prompt caching Quantization KV cache Speculative decoding

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: workload-changing vs efficiency-changing distinction + the model-substitution win + prompt-cache pricing math + why the three infra levers share the bandwidth bottleneck + speculative decoding as a latency lever.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's GPT-4o-mini and Anthropic's Claude Haiku tiers exist precisely to capture the model-substitution savings without forcing teams to self-host.
Cursor, Perplexity, and many RAG-heavy products report 60-80% input-cost reductions from Anthropic prompt caching on stable system prompts and retrieval contexts.
vLLM in 2026 defaults to W4A16 + FP8 KV on H100 deployments because the stack matches the decode-bandwidth bottleneck.
NVIDIA TensorRT-LLM's Medusa and EAGLE speculative-decoding modes ship primarily as latency features, with serving guides noting the verifier-compute overhead.
Anthropic's Claude Sonnet 4.5 and OpenAI's gpt-5-mini are the 2026 'right-sized' tiers that downstream teams route most non-hard traffic to.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build a model-substitution eval that doesn't leave 30% of regressions in the dark?

Sign in or go Pro to see the approach for this follow-up.

QWhy does INT4 weights + FP8 KV cache stack less than additively?

Sign in or go Pro to see the approach for this follow-up.

QWhen does speculative decoding actually save money rather than just latency?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Putting quantization or speculative decoding first because they sound technical. Model substitution beats them by an order of magnitude when the quality bar allows, and prompt caching often beats them on real production traffic.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.