When an async batch API beats the real time endpoint
Use a batch API for non-interactive jobs that can wait minutes to hours; you trade immediacy for a large discount (often ~50%) and isolation from live serving capacity.
Imagine you can either mail a single letter overnight for full price, or drop a big stack into a bulk service that delivers within a day for half the cost. If you need an answer right now, you pay for speed. If nobody's waiting, you save by letting them handle it whenever they have spare capacity. An LLM batch API is the bulk service. For work where no person is sitting there waiting — processing yesterday's data, scoring a big pile of records overnight — you hand the whole stack over, accept that it comes back later, and pay much less. You only use the instant, full-price option when someone is actually waiting on the reply.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every LLM provider ships two front doors to the same model: a synchronous real-time endpoint that answers in seconds, and an asynchronous batch API that takes a whole job and returns results later, typically within a day. They run the identical model, so the choice between them is never about quality. It's about a latency contract and what that contract costs.
The reason this matters in practice is that the default instinct — always call the real-time endpoint — quietly wastes money and, worse, lets bulk background work contend with the capacity your live users depend on. A nightly job that re-embeds a million records doesn't need a two-second response, yet running it synchronously pays the premium for speed nobody wanted and eats into the same rate limits serving your chat traffic.
This walkthrough frames the choice as a single question about who's waiting, explains the two concrete benefits batch buys you, maps the workloads where it's the obvious call, and then covers the engineering reality — the async retrieval flow and the boundary cases — that a complete answer has to acknowledge.
The deciding question: who is waiting?
Strip the decision down to one question: at the moment this request fires, is anything blocked until the answer comes back? A human watching a spinner, an API call your client is awaiting, a synchronous pipeline step that can't proceed — those are blocked, and they need the real-time endpoint. A row in a table you'll process overnight, an evaluation set you'll review tomorrow, a backfill that just has to finish by morning — nothing is blocked, and those belong on batch.
This framing beats trying to memorize a list of "batch workloads," because it generalizes. Any task, however unusual, sorts itself the moment you ask whether its result is on a synchronous critical path. If removing instant responses would break a user experience or stall a dependent system, it's real-time. If you could collect the results in an hour with no one noticing the wait, it's batch.
The instinct to build is asymmetric. Real-time is the option you should have to justify — you reach for it because something genuinely waits. Batch is the sensible default for everything else. Most teams have the bias backwards: they reach for real-time reflexively because it's the endpoint they already wired up for the product, then run their offline jobs through it too. Flipping that default is most of the value of understanding the distinction.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | Real-time endpoint | Batch API |
|---|---|---|
| Latency | Seconds | Minutes to hours (often up to ~24h) |
| Cost | Full per-token rate | Discounted, often ~50% off |
| Capacity | Shares interactive serving pool | Isolated from live traffic |
| Use when | A human or system is waiting | Nothing is waiting on the result |
Real products, models, and research that use this idea.
- OpenAI's Batch API processes jobs within 24 hours at roughly 50% off the synchronous price for offline workloads.
- Anthropic's Message Batches API offers a discount for asynchronous bulk processing of Claude requests.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can providers offer batch at roughly half the price?
Deferred scheduling lets them fill spare capacity instead of reserving interactive headroom, smoothing utilization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Running large offline jobs on the real-time endpoint, paying full price and starving live user traffic when batch would be cheaper and isolated.
60 second bullets to scan on the way to the call.
The deciding question about who or what is waiting on the result
What batch trades away in immediacy and synchronous SLA terms
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.