Why do hosted LLM APIs charge separate per million rates for input and output tokens?

Flashcard·Easy·4.0 · 0·~30s·Asked atBrowserbase Ltimindtree Synthesia·Relevant atOpenAI

Attempt it

Input tokens (the prompt) are processed in a single parallel prefill pass, where weights are read from HBM once and reused across the whole sequence: compute-bound, cheap per token. Output tokens are produced one at a time in decode, where each step rereads the full weights and KV cache from HBM: bandwidth-bound, expensive per token. Providers pass this asymmetry through, so output tokens are typically billed at 3-5x the input rate.

Your answer

Hosted LLM APIs split pricing into two columns because input tokens and output tokens cost the provider very different amounts of GPU time. **Input tokens** are processed in a single phase called **prefill**. The whole prompt goes through one parallel forward pass: the GPU reads the model weights from HBM once and uses them to compute attention and feedforward for every token in the prompt simultaneously. Because the weights are reused across hundreds or thousands of tokens, the math per byte read ratio is high. The GPU is fully utilized and the per-token cost is small. **Output tokens** are produced one at a time in the **decode** phase. Each new token requires its own forward pass, which means rereading the full weight matrices and the accumulated KV cache from HBM. The GPU spends most of its time waiting for memory, with very little math per byte read. Arithmetic intensity is low and the per-token cost is much higher. Providers pass this asymmetry through. GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro, and every other hosted API charge output tokens at roughly 3-5x the input rate. Cached input tokens drop even lower: Anthropic charges 0.1x for cache reads, OpenAI charges 0.5x on prefixes of 1024+ tokens.

Input and output tokens hit the GPU in structurally different ways, which is why the bill splits. **Input tokens** flow through prefill in one parallel forward pass over the entire prompt. The weight matrices are read from HBM once and reused across every token in the sequence; arithmetic intensity is high (many FLOPs per byte moved), so the kernel is compute-bound and the GPU's tensor cores stay busy. The marginal cost of one more input token is small. **Output tokens** flow through decode, one autoregressive step per token. Each step reads the full weights and the entire KV cache from HBM but operates on a single new token's activations. Arithmetic intensity is low; the GPU spends most of its time waiting on memory bandwidth, not computing. The marginal cost per output token is much higher than per input token. The pricing split passes this through. Typical 2026 ratios: Claude Sonnet 4.6 charges $3/M input and $15/M output (5x ratio). GPT-5.5 sits at similar ratios. Cached input tokens are even cheaper because prefill collapses to a KV-cache copy: Anthropic charges 0.1x on cache reads (and 1.25x on cache writes); OpenAI charges 0.5x on prefixes of 1024+ tokens. The practical takeaway: if your application leans on long shared prompts (system messages, few-shot examples, retrieved context), prefix caching can cut bills by 70-90 percent. If you write extra-long completions, output cost dominates total spend even when input volume is large.

TL;DR

Input tokens run in one parallel compute-bound prefill pass; output tokens run one-at-a-time in bandwidth-bound decode steps. Output is 3-5x more expensive, so providers split the rate.

Memory aid

Easy to grasp

Imagine the kitchen has to prepare a dish in two stages. The first stage is dumping all the ingredients on the counter at once and chopping them in parallel: ten cooks each take a knife and finish the prep together in a single minute. The second stage is plating, where only one cook can place one bite at a time onto the plate while everyone else stands around. The first stage is fast and cheap per ingredient; the second stage is slow and expensive per bite. That is exactly why your bill from a hosted LLM splits input tokens (cheap prep) from output tokens (expensive plating).

Key concepts

Input/output pricing Prefill Decode Arithmetic intensity KV cache

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Every hosted LLM API in 2026 publishes a two-column pricing page: one rate for input tokens, another (3-5x higher) for output tokens. New engineers reading those pages often assume the split is an arbitrary commercial choice, like restaurants charging more for dinner than lunch. It is not. The asymmetry is a direct passthrough of how transformer inference works on modern GPUs.

This deep dive walks through the prefill versus decode compute model, the arithmetic-intensity argument that makes input tokens cheap and output tokens expensive, the 2026 pricing landscape across major providers, and the prefix-caching innovation that has reshaped cost optimization for production workloads.

Prefill: the parallel, compute-bound phase

When a request arrives at an LLM server, the entire input prompt is processed in a single phase called prefill. The server runs one forward pass that handles all input tokens simultaneously: the attention matmul operates on a T x d_model tensor where T is the prompt length, and the feedforward block similarly processes all T tokens in parallel.

The critical property is that the model's weight matrices, which sit in HBM, are read once and reused across every token in the prompt. For a 70B model that is roughly 140GB of weights at FP16. Reading 140GB through HBM at 3 TB/s takes about 47 milliseconds, but the FLOPs the GPU performs in that time scale linearly with T. A 2000-token prompt produces 10x more FLOPs than a 200-token prompt while reading the same weight bytes.

Arithmetic intensity

This FLOPs-per-byte ratio is called arithmetic intensity. Prefill on long prompts puts the kernel high on the roofline plot, where tensor cores saturate and the GPU runs at peak compute. The per-token cost of an input token is correspondingly small. Providers reflect this with low input rates.

Decode: the sequential, bandwidth-bound phase

The 2026 pricing landscape and the cache tier

Implications for cost optimization

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Claude Sonnet 4.6 publishes $3/M input and $15/M output (5x ratio), with cache reads at $0.30/M (0.1x of input).
GPT-5.5 charges roughly $5/M input and $15/M output, with cached input at 0.5x on prefixes of 1024+ tokens.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf output is so much more expensive, why do providers not just charge a single blended rate?

Two reasons. First, customers tune workloads: cheaper input means RAG and few-shot prompting stay economical, while output-heavy chatbots pay their fair share. Second, transparency: splitting input and output rates makes prefix caching's value legible. A blended rate would hide the prefix-cache discount entirely.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming input and output cost the same because they are both tokens. The asymmetry is structural: one phase is compute-bound, the other is bandwidth-bound, and the provider passes that through.

60 second bullets to scan on the way to the call.

Definition of prefill and how it processes input tokens in parallel
Definition of decode and how it produces output tokens one at a time

Primary sources. Browse if you want the original framing.