Pick the workload where a provider's Batch API beats the realtime endpoint

MCQ·Medium·4.0 · 0·~1 min·Asked atAnthropic OpenAI

Attempt it

**Correct answer: B (the nightly evaluation run).** Provider **Batch APIs** from OpenAI, Anthropic, and Google all share the same shape. You upload a file of requests, the provider processes them whenever capacity allows within a 24-hour SLA, and you download the results. The discount is roughly 50 percent off both input and output tokens compared with the realtime endpoint. Model, quality, and prompt format are identical; the only difference is that the provider gets to choose when to run your requests, batched against unused capacity. The decision rule is simple: can the workload tolerate a 24-hour turnaround? If yes, use Batch. If no, use realtime. Let us walk the options: - **A (chat assistant, 2-second p95):** the SLA is 24 hours. Two seconds versus 24 hours is four orders of magnitude. The cost savings are real but irrelevant when you cannot meet latency. - **B (nightly evaluation, same-day acceptable):** perfect fit. A 200k-row rubric eval does not care when each row finishes, and the 50 percent saving on 200k completions is material. - **C (IDE autocomplete):** the user is typing now. Sub-second latency budget. Batch is disqualified. - **D (voice agent tool call):** the conversation is in progress. Sub-second budget. Batch is disqualified. Option A is the classic trap. The cost number tempts you, but the SLA breaks before cost even matters.

**Option B (the nightly evaluation run) is correct.** Provider Batch APIs make a single tradeoff: they let the provider schedule your requests against spare capacity within a 24-hour SLA, in exchange for roughly 50 percent off both input and output token rates. Quality, model, prompt format, and even rate limits are otherwise identical to the realtime endpoint. So the selection question is purely about latency tolerance. If the workload can wait 24 hours, Batch is essentially free money. If it cannot, Batch is disqualified regardless of cost. Walk the options: - **A** is a customer-facing chat assistant with a 2-second p95 target. The Batch SLA is 24 hours. Cost savings cannot rescue you from missing latency by 43,200x. - **B** is a nightly 200,000-row eval. Same-day completion is acceptable, the dataset is fixed at job start, and there is no human waiting per row. A 50 percent saving on 200k completions is thousands of dollars per run, often the dominant line item in an ML team's monthly inference bill. - **C** is an IDE autocomplete plugin. A human is mid-keystroke; the latency budget is hundreds of milliseconds. Batch is disqualified. - **D** is a voice agent in active conversation calling a tool. The latency budget is sub-second. Batch is disqualified. The canonical Batch API workloads are: offline evaluation runs, dataset labeling, batch summarization of long documents, nightly enrichment pipelines, agentic experiments that fan out many candidates per task, and any analytical job where the result is consumed by another batch process rather than a person. Option B fits this pattern cleanly. A is the trap that tests whether you can recognize cost optimization is irrelevant when latency is the binding constraint.

Quick reference

TL;DR

Batch APIs trade ~50% cost discount for a 24-hour SLA, so only latency-tolerant offline workloads qualify; nightly evaluation runs are the canonical fit, latency-bound chat or tool workflows are disqualified outright.

Memory aid

Easy to grasp

Picture a printing service with two counters. The express counter prints your document in minutes but costs double. The economy counter prints it for half the price but does not promise it until tomorrow. You pick based on whether you need it today or whether you just need it eventually. Anything a user is actively waiting on must go through the express counter. Anything you queue up overnight should go through the economy counter to save money. Provider Batch APIs are that economy counter for LLM calls. The model and the quality are identical; the only difference is when you get your answer back.

Key concepts

Batch API Inference cost Latency Service-level objective

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: same model and quality + 50 percent discount + 24-hour SLA + latency tolerance is the only discriminator + option A is the trap + B fits because no human waits per row + canonical workload list for Batch.

Side-by-side

Endpoint	Latency	Cost	Best for
Realtime	Sub-second to a few seconds	Full input and output rates	Chat, autocomplete, voice agents, tool calling, any human or tool waiting
Batch	Up to 24-hour SLA	Roughly 50 percent off input and output rates	Eval runs, dataset labeling, batch summarization, nightly enrichment, synthetic data

Real-world examples

Real products, models, and research that use this idea.

OpenAI's /v1/batches endpoint serves GPT-5.5 with a 50 percent discount for nightly evaluation runs and dataset labeling pipelines.
Anthropic's Message Batches API serves Claude Opus 4.7 and Sonnet 4.6 with similar 50 percent off pricing for offline workloads.
Google Gemini 3.1 Pro offers batch processing for large-scale enrichment jobs at discounted rates.
Machine-learning teams routinely move LLM-as-judge eval runs to Batch APIs to cut quarterly compute spend without affecting model quality.
Synthetic data generation pipelines for fine-tuning Llama 4 Scout often use Batch APIs since latency per generation is irrelevant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you operationally split a workload into realtime and Batch tiers?

QWhy do providers offer a 50 percent discount on Batch rather than a smaller one?

QAre there workloads where Batch is the wrong choice even though the SLA fits?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking the Batch API for a workload that has a human or tool waiting on the response. The cost savings are real but irrelevant when the latency budget is sub-second.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.