When the OpenAI or Anthropic batch API turns into the right tool
Move anything off the user request path to batch for the 50 percent discount; keep latency-sensitive paths on realtime.
Imagine sending a package. If someone is standing at the door waiting, you pay for same-day courier. If the box is going into a warehouse and nobody opens it until next morning, you ship overnight ground for half the price. The batch API is the overnight ground: cheaper, slower, perfect for things nobody is watching in real time. Backfilling search indexes, scoring last week's logs, summarizing a million old support tickets, all overnight-ground work. The mistake is using the same-day courier for the warehouse delivery and watching the shipping bill double.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The batch API is the cheapest piece of LLMOps savings most teams leave on the table. Both OpenAI and Anthropic publish a batch endpoint priced at roughly 50 percent of the realtime per-token rate, in exchange for a turnaround commitment measured in hours rather than seconds. The deal is straightforward, the dollar impact at any meaningful scale is large, and the operational surface is small.
The reason teams miss it is almost never a technical objection. It is that the realtime SDK call site already exists, the engineer building the nightly job copies the pattern, and nobody questions whether the job needed realtime in the first place.
The one decision question
Every LLM workload boils down to a single routing question: is a human user actively waiting on this response?
If yes, the workload belongs on realtime. Chat replies, autocomplete suggestions, search ranking, summarization invoked from a UI button, anything where seconds of latency degrade the user experience. The realtime tier exists for this and there is no debate.
If no, the workload is a batch candidate by default. The provider gives you 24 hours to complete the job, but in practice most batches finish in well under an hour. Embedding a new document corpus, regenerating summaries after a prompt change, scoring a golden eval set, periodic classification of historical records, weekly digest jobs, dataset enrichment for analytics dashboards, none of these has a user staring at a spinner. They are jobs feeding other jobs.
The default should be inverted from what most teams do. New non-interactive workload starts on batch and only moves to realtime if a latency requirement forces it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI Batch API takes a JSONL file of chat or embedding requests at half price with a 24-hour completion window.
- Anthropic Message Batches accepts up to 100,000 requests per batch at 50 percent off, with results retrievable for 29 days.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide whether to run an eval suite on batch or realtime?
Frequency drives it: pre-merge gates that need fast signal stay realtime; nightly regression sweeps over the full golden set go batch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Running a nightly enrichment job through the realtime API and paying the full price on workload no user is waiting for.
60 second bullets to scan on the way to the call.
What is the one routing question you ask before placing any LLM workload?
Which workload shapes belong on the batch tier by default?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.