Why are output tokens typically 3-5× more expensive than input tokens?

Attempt it

Explain the physical reason output tokens are priced 3-5× higher than input tokens. Tie the pricing back to the underlying hardware utilization.

Free · 2 AI evals / day

Quick reference

TL;DR

Input is one parallel prefill pass that saturates compute, so it is cheap per token. Output is N sequential decode steps, each gated by HBM bandwidth, so providers price it 3-5x higher.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a bakery oven. Reading your order (the input) is like baking a full tray at once: you heat the oven once and dozens of cookies come out together, so each cookie is cheap. Writing the reply (the output) is like baking one cookie, taking it out, then reheating the whole oven for the next single cookie. The expensive part is not the cookie, it is firing up that huge oven again and again for one item at a time. The model weights are the oven, and loading them from memory is the reheating. Input bakes a whole tray in one pass. Output reheats per cookie. That is why the company charges you more for each word it writes than for each word it reads.

Key concepts

Prefill vs decode Memory-bandwidth bound Arithmetic intensity Batching

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: prefill compute-bound versus decode bandwidth-bound, the roofline and arithmetic intensity argument, the per-instance decode throughput ceiling, how batching narrows the gap, then map the throughput ratio onto the 3-5x price band.

Real-world examples

Real products, models, and research that use this idea.

OpenAI prices GPT-5.5 output tokens several times higher than input, with cached input far cheaper still, reflecting the prefill versus decode split.
Anthropic's Claude Opus 4.7 lists output at roughly 5x input, and prompt caching cuts repeated input cost to a fraction of full prefill.
vLLM and SGLang use continuous batching to raise realized decode throughput, directly narrowing the input versus output cost gap on self-hosted Llama 4.
NVIDIA's TensorRT-LLM disaggregates prefill and decode onto separate GPU pools so each phase runs at its own optimal batch and the decode cost floor drops.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does decode stay bandwidth-bound even when the GPU has spare compute?

Sign in or go Pro to see the approach for this follow-up.

QHow does batching change the input versus output cost ratio, and what bounds it?

Sign in or go Pro to see the approach for this follow-up.

QWhy does prompt caching let providers charge a fraction of normal input price for cached tokens?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming output costs more because generation does more math. It is the opposite: decode does the same matmuls as prefill but spreads them over one token per weight read, so it is bandwidth starved, not compute heavy.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.