Contrast online interactive inference and offline batch inference along the axes that actually drive deployment decisions: optimization target, scheduling, batch sizing, and pricing. Why are batch APIs ~50% cheaper at the same hardware?
Online inference optimizes per-request latency under an SLO and caps batch size; offline inference optimizes throughput per dollar and runs at max batch, which is why batch APIs cost about half.
Picture two coffee shops with identical machines. The first is a sit-down cafe: you want your drink in two minutes, so the barista never lets the queue grow past a few orders, keeping each one fast. The second is a catering kitchen: nobody waits at the counter, so the barista crams the machine to its absolute limit, brewing huge trays at once. Each individual cup takes longer, but the kitchen makes far more cups per hour from the same machine. The cafe sells speed; the kitchen sells volume. Online inference is the cafe, offline batch inference is the kitchen, and the kitchen charges less per cup because it squeezes more out of the same equipment with no rush.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Online versus offline inference is one of the cleanest systems questions in LLM serving, because the entire deployment design flows from a single choice: is a human waiting on this response or not? That answer fixes the optimization target, and the optimization target fixes the scheduling, the batch sizing, the provisioning, and ultimately the price a customer pays per token.
The subtle part is that both regimes run the same model on the same kind of GPU with the same serving engine. Nothing about the math changes, and no special offline-only model is involved. What changes is where on the latency-throughput curve you choose to operate, and that choice is worth roughly a factor of two in cost per token. Interviewers love this question precisely because a candidate who only ever called a synchronous API often has no mental model for the offline regime, and the gap shows immediately.
This deep dive walks through the two regimes, the SLO metrics that define online serving, why offline serving runs saturated, the saturation curve that makes batching nearly free up to a point, and the precise mechanics behind the batch-API discount. It also covers the failure modes of mixing the two and the provisioning consequences that follow from each target. By the end you should be able to explain not just that batch is cheaper, but exactly which three levers produce the discount and why none of them is available to online traffic.
Two regimes, one engine, opposite targets
Online inference serves anything with a human in the loop: interactive chat, agent loops, RAG-backed assistants, code completion. The defining constraint is that someone is staring at a screen waiting for output. The optimization target is therefore per-request latency under a service-level objective, and every scheduling decision exists to protect that bound.
Offline inference serves work that has no real-time observer: bulk document classification, large evaluation runs, synthetic data generation, nightly summarization pipelines, and the hosted batch APIs that wrap them. The defining freedom is that a job can finish in ten minutes or ten hours and nobody cares. The optimization target is throughput per dollar, often expressed concretely as tokens per second per GPU divided by the hourly cost of that GPU.
These targets are not just different, they are in direct tension. Latency wants small batches and spare capacity, because both keep the per-token time low and the tail bounded. Throughput wants large batches and full utilization, because both amortize fixed cost over more output. You cannot maximize both at once on one stream of traffic, which is why providers run physically or logically separate pools for the two and route requests by the SLO the caller signed up for.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI and Anthropic both ship Batch APIs priced about 50 percent below their synchronous endpoints, with a 24-hour completion window in exchange.
- vLLM and SGLang expose batch-size and scheduling knobs that let one engine serve low-latency online traffic and high-throughput offline jobs differently.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does running a larger batch barely cost throughput per request until a certain point?
Decode is memory bandwidth bound: each step streams weights and the KV cache from HBM regardless of batch size. Extra requests ride along on the same memory reads until you saturate compute or exhaust KV memory, which is the knee in the curve.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying the batch discount is just a margin or marketing choice. It reflects a real cost difference: max batching roughly doubles tokens per second per GPU on the same hardware.
60 second bullets to scan on the way to the call.
The one metric each regime lives or dies by (latency SLO versus throughput per dollar)
Typical online SLO bars for time to first token and time per output token
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.