Contrast online interactive inference and offline batch inference along the axes that actually drive deployment decisions: optimization target, scheduling, batch sizing, and pricing. Why are batch APIs ~50% cheaper at the same hardware?
Free · 2 AI evals / day
LLM serving comes in two flavors with opposite goals. **Online inference** powers interactive things like chat, agents, and search. A human is waiting, so the metric that matters is latency: how fast the first token appears and how smoothly the rest stream out.
Offline inference runs jobs where nobody is waiting in real time: nightly document processing, large eval runs, bulk summarization. Here the metric is throughput per dollar. You want to push as many tokens through each GPU per hour as possible, and a job finishing in ten minutes versus ten seconds does not matter.
The goals force different scheduling. Online servers deliberately keep the batch small so each request stays fast. Pack too many requests onto a GPU and every user's tokens slow down. Offline servers do the opposite: they cram the batch as large as the hardware allows, accepting slow per-request speed to maximize total tokens per second.
This is why batch APIs from OpenAI and Anthropic cost roughly half. Same GPUs, but max batching doubles output per machine, jobs can run during quiet hours, and no spare capacity is held back for traffic spikes. Looser timing, lower cost.
**Online inference** serves interactive workloads: chat, agent loops, RAG. A user waits, so the optimization target is per-request latency under a service-level objective. Typical bars are time to first token under 500ms and time per output token under 30ms.
The scheduler caps batch size below GPU saturation to keep that per-token latency predictable. Throughput is simply whatever fits under the SLO. You leave hardware on the table on purpose.
Offline inference serves batch APIs, document pipelines, and eval runs. There is no latency SLO; jobs may wait minutes or hours. The optimization target flips to throughput per dollar. The scheduler runs at the maximum sustainable batch, near the compute roof, to extract every token per second from each GPU.
This is why batch APIs price about 50 percent below online. Three mechanisms compound. Running at max batch roughly doubles tokens per second per GPU. Deferring jobs to off-peak windows fills idle fleet capacity. Removing the latency tail buffer means no spare GPUs sit reserved for spikes. Same hardware, looser scheduling, half the cost.
Quick reference
TL;DR
Online inference optimizes per-request latency under an SLO and caps batch size; offline inference optimizes throughput per dollar and runs at max batch, which is why batch APIs cost about half.
Picture two coffee shops with identical machines. The first is a sit-down cafe: you want your drink in two minutes, so the barista never lets the queue grow past a few orders, keeping each one fast. The second is a catering kitchen: nobody waits at the counter, so the barista crams the machine to its absolute limit, brewing huge trays at once. Each individual cup takes longer, but the kitchen makes far more cups per hour from the same machine. The cafe sells speed; the kitchen sells volume. Online inference is the cafe, offline batch inference is the kitchen, and the kitchen charges less per cup because it squeezes more out of the same equipment with no rush.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A user-facing chat or agent feature where a human waits on every response Offline-style max batching destroys the per-token latency that interactive use demands.
A one-off tiny job of a few requests on a single GPU Neither regime's machinery pays off; scheduling overhead exceeds any throughput gain you could win.
Hard real-time deadlines measured in low milliseconds, such as live voice Even online batching adds queueing latency; you may need dedicated single-stream serving instead.
Workloads that must complete within seconds but arrive in large bursts A deferred batch window of minutes or hours violates the deadline despite the cost savings.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name each regime's target (latency SLO versus throughput per dollar) + the batch-size posture each forces + the three mechanisms behind the 50 percent batch discount + why decode being memory-bound makes the knee exist.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI and Anthropic both ship Batch APIs priced about 50 percent below their synchronous endpoints, with a 24-hour completion window in exchange.
2vLLM and SGLang expose batch-size and scheduling knobs that let one engine serve low-latency online traffic and high-throughput offline jobs differently.
3Anthropic's Claude Opus 4.7 is offered through both an interactive API tuned for time to first token and a Message Batches API tuned for throughput per dollar.
4Document pipelines and eval suites running Llama 4 or DeepSeek V4 on offline clusters routinely push batch sizes far past what any interactive chat deployment would tolerate.
5Together AI and Modal market dedicated offline batch tiers that fill spare GPU capacity during low-demand windows at reduced per-token rates.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does running a larger batch barely cost throughput per request until a certain point?
Sign in or go Pro to see the approach for this follow-up.
QHow does deferring batch jobs to off-peak windows actually lower the provider's cost?
Sign in or go Pro to see the approach for this follow-up.
QWhat stops you from simply running every workload at max batch to cut cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying the batch discount is just a margin or marketing choice. It reflects a real cost difference: max batching roughly doubles tokens per second per GPU on the same hardware.