Rank these chat-API cost levers from highest to lowest typical ROI

Cost levers fall into two families. The first family changes **what work the system does**, a smaller model, or skipping work via caching. The second family changes **how efficiently the same work runs**, quantization, KV-cache compression, faster decoding tricks. The first family almost always wins, because doing less work beats doing the same work faster. Switching from a 70B to an 8B model that passes your quality eval cuts compute and memory by roughly an order of magnitude. Nothing on the infra side comes close to a 10x reduction. So if your quality bar permits it, this is always the first lever to pull. Prompt caching is next on real production traffic. Most chat applications send the same system prompt, RAG context chunk, or few-shot prefix on nearly every request. Anthropic and OpenAI both let you cache that prefix at roughly one-tenth the input rate (Anthropic) or half rate (OpenAI). If 70% of your input is a stable prefix, you cut nearly that fraction of your input bill. INT4 weights and FP8 KV are real wins on self-hosted serving, but smaller. Speculative decoding mostly buys latency, not cost; the draft model adds compute even when it accelerates user-facing time.

**Cost-reduction ROI** is dominated by two structural moves and then a long tail of infrastructure tuning. Model substitution is the king lever because it changes the unit cost of every token by 5-20x. A serious quality eval often finds that an 8B or a frontier-mini tier serves 60-80% of production traffic with no measurable user impact, and the remaining hard cases can be routed to a larger model. Nothing in the infrastructure stack approaches a 10x cut. Prompt caching is second because it targets the input side, which on RAG and agentic workloads is often the dominant cost term. Anthropic charges 1.25x on the write and 0.1x on each read, with break-even at roughly the first read. OpenAI auto-caches prefixes of 1024+ tokens at ~0.5x. If your system prompt + tool schemas total 4K tokens and fire on every request, caching slashes that fraction of input nearly to zero. INT4 weight quantization on a self-hosted deployment is third, it cuts per-step HBM bytes ~4x and is essentially free in quality on most chat workloads if you use AWQ or GPTQ. FP8 KV cache is fourth and overlaps in benefit because both target decode-phase bandwidth; you stack them but the second adds less than the first did. Speculative decoding is last because its primary product is latency reduction. On batched serving it can even *raise* cost (you pay for both draft and verifier compute) and it only helps cost when the draft acceptance rate is very high and the verifier is underutilized.