Contrast online interactive inference and offline batch inference along the axes that actually drive deployment decisions: optimization target, scheduling, batch sizing, and pricing. Why are batch APIs ~50% cheaper at the same hardware?
Online inference optimizes per-request latency under an SLO and caps batch size; offline inference optimizes throughput per dollar and runs at max batch, which is why batch APIs cost about half.
Picture two coffee shops with identical machines. The first is a sit-down cafe: you want your drink in two minutes, so the barista never lets the queue grow past a few orders, keeping each one fast. The second is a catering kitchen: nobody waits at the counter, so the barista crams the machine to its absolute limit, brewing huge trays at once. Each individual cup takes longer, but the kitchen makes far more cups per hour from the same machine. The cafe sells speed; the kitchen sells volume. Online inference is the cafe, offline batch inference is the kitchen, and the kitchen charges less per cup because it squeezes more out of the same equipment with no rush.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: name each regime's target (latency SLO versus throughput per dollar) + the batch-size posture each forces + the three mechanisms behind the 50 percent batch discount + why decode being memory-bound makes the knee exist.
Real products, models, and research that use this idea.
- OpenAI and Anthropic both ship Batch APIs priced about 50 percent below their synchronous endpoints, with a 24-hour completion window in exchange.
- vLLM and SGLang expose batch-size and scheduling knobs that let one engine serve low-latency online traffic and high-throughput offline jobs differently.
- Anthropic's Claude Opus 4.7 is offered through both an interactive API tuned for time to first token and a Message Batches API tuned for throughput per dollar.
- Document pipelines and eval suites running Llama 4 or DeepSeek V4 on offline clusters routinely push batch sizes far past what any interactive chat deployment would tolerate.
- Together AI and Modal market dedicated offline batch tiers that fill spare GPU capacity during low-demand windows at reduced per-token rates.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does running a larger batch barely cost throughput per request until a certain point?
QHow does deferring batch jobs to off-peak windows actually lower the provider's cost?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying the batch discount is just a margin or marketing choice. It reflects a real cost difference: max batching roughly doubles tokens per second per GPU on the same hardware.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.