Why do hosted LLM APIs charge separate per-million rates for input and output tokens?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Input tokens (the prompt) are processed in a single parallel prefill pass, where weights are read from HBM once and reused across the whole sequence: compute-bound, cheap per token. Output tokens are produced one at a time in decode, where each step rereads the full weights and KV cache from HBM: bandwidth-bound, expensive per token. Providers pass this asymmetry through, so output tokens are typically billed at 3-5x the input rate.

Your answer

Hosted LLM APIs split pricing into two columns because input tokens and output tokens cost the provider very different amounts of GPU time. **Input tokens** are processed in a single phase called **prefill**. The whole prompt goes through one parallel forward pass: the GPU reads the model weights from HBM once and uses them to compute attention and feedforward for every token in the prompt simultaneously. Because the weights are reused across hundreds or thousands of tokens, the math per byte read ratio is high. The GPU is fully utilized and the per-token cost is small. **Output tokens** are produced one at a time in the **decode** phase. Each new token requires its own forward pass, which means rereading the full weight matrices and the accumulated KV cache from HBM. The GPU spends most of its time waiting for memory, with very little math per byte read. Arithmetic intensity is low and the per-token cost is much higher. Providers pass this asymmetry through. GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro, and every other hosted API charge output tokens at roughly 3-5x the input rate. Cached input tokens drop even lower: Anthropic charges 0.1x for cache reads, OpenAI charges 0.5x on prefixes of 1024+ tokens.

Input and output tokens hit the GPU in structurally different ways, which is why the bill splits. **Input tokens** flow through prefill in one parallel forward pass over the entire prompt. The weight matrices are read from HBM once and reused across every token in the sequence; arithmetic intensity is high (many FLOPs per byte moved), so the kernel is compute-bound and the GPU's tensor cores stay busy. The marginal cost of one more input token is small. **Output tokens** flow through decode, one autoregressive step per token. Each step reads the full weights and the entire KV cache from HBM but operates on a single new token's activations. Arithmetic intensity is low; the GPU spends most of its time waiting on memory bandwidth, not computing. The marginal cost per output token is much higher than per input token. The pricing split passes this through. Typical 2026 ratios: Claude Sonnet 4.6 charges $3/M input and $15/M output (5x ratio). GPT-5.5 sits at similar ratios. Cached input tokens are even cheaper because prefill collapses to a KV-cache copy: Anthropic charges 0.1x on cache reads (and 1.25x on cache writes); OpenAI charges 0.5x on prefixes of 1024+ tokens. The practical takeaway: if your application leans on long shared prompts (system messages, few-shot examples, retrieved context), prefix caching can cut bills by 70-90 percent. If you write extra-long completions, output cost dominates total spend even when input volume is large.

Quick reference

TL;DR

Input tokens run in one parallel compute-bound prefill pass; output tokens run one-at-a-time in bandwidth-bound decode steps. Output is 3-5x more expensive, so providers split the rate.

Memory aid

Easy to grasp

Imagine the kitchen has to prepare a dish in two stages. The first stage is dumping all the ingredients on the counter at once and chopping them in parallel: ten cooks each take a knife and finish the prep together in a single minute. The second stage is plating, where only one cook can place one bite at a time onto the plate while everyone else stands around. The first stage is fast and cheap per ingredient; the second stage is slow and expensive per bite. That is exactly why your bill from a hosted LLM splits input tokens (cheap prep) from output tokens (expensive plating).

Key concepts

Input/output pricing Prefill Decode Arithmetic intensity KV cache

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: prefill parallel and compute-bound + decode sequential and bandwidth-bound + 3-5x pricing ratio + prefix caching discounts (0.1x Anthropic, 0.5x OpenAI) + MoE skews the ratio.

Real-world examples

Real products, models, and research that use this idea.

Claude Sonnet 4.6 publishes $3/M input and $15/M output (5x ratio), with cache reads at $0.30/M (0.1x of input).
GPT-5.5 charges roughly $5/M input and $15/M output, with cached input at 0.5x on prefixes of 1024+ tokens.
Gemini 3.1 Pro on Google AI Studio splits input/output rates similarly, with separate per-million prices visible in the pricing console.
Self-hosted vLLM and SGLang on Llama 4 Maverick show the same asymmetry directly: GPU utilization is near 100 percent during prefill and 30-50 percent during decode.
DeepSeek V4's hosted API charges a notably smaller input/output ratio (closer to 2x) because their MoE routing reduces decode cost per active expert.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf output is so much more expensive, why do providers not just charge a single blended rate?

QHow does the input/output cost ratio change for a small model versus a large MoE model?

QWhy is output cost dominated by HBM bandwidth rather than compute, even on a $40K H100?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming input and output cost the same because they are both tokens. The asymmetry is structural: one phase is compute-bound, the other is bandwidth-bound, and the provider passes that through.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.