**Correct answer: B (the nightly evaluation run).**
Provider **Batch APIs** from OpenAI, Anthropic, and Google all share the same shape. You upload a file of requests, the provider processes them whenever capacity allows within a 24-hour SLA, and you download the results. The discount is roughly 50 percent off both input and output tokens compared with the realtime endpoint. Model, quality, and prompt format are identical; the only difference is that the provider gets to choose when to run your requests, batched against unused capacity.
The decision rule is simple: can the workload tolerate a 24-hour turnaround? If yes, use Batch. If no, use realtime.
Let us walk the options:
- **A (chat assistant, 2-second p95):** the SLA is 24 hours. Two seconds versus 24 hours is four orders of magnitude. The cost savings are real but irrelevant when you cannot meet latency.
- **B (nightly evaluation, same-day acceptable):** perfect fit. A 200k-row rubric eval does not care when each row finishes, and the 50 percent saving on 200k completions is material.
- **C (IDE autocomplete):** the user is typing now. Sub-second latency budget. Batch is disqualified.
- **D (voice agent tool call):** the conversation is in progress. Sub-second budget. Batch is disqualified.
Option A is the classic trap. The cost number tempts you, but the SLA breaks before cost even matters.
**Option B (the nightly evaluation run) is correct.**
Provider Batch APIs make a single tradeoff: they let the provider schedule your requests against spare capacity within a 24-hour SLA, in exchange for roughly 50 percent off both input and output token rates. Quality, model, prompt format, and even rate limits are otherwise identical to the realtime endpoint.
So the selection question is purely about latency tolerance. If the workload can wait 24 hours, Batch is essentially free money. If it cannot, Batch is disqualified regardless of cost.
Walk the options:
- **A** is a customer-facing chat assistant with a 2-second p95 target. The Batch SLA is 24 hours. Cost savings cannot rescue you from missing latency by 43,200x.
- **B** is a nightly 200,000-row eval. Same-day completion is acceptable, the dataset is fixed at job start, and there is no human waiting per row. A 50 percent saving on 200k completions is thousands of dollars per run, often the dominant line item in an ML team's monthly inference bill.
- **C** is an IDE autocomplete plugin. A human is mid-keystroke; the latency budget is hundreds of milliseconds. Batch is disqualified.
- **D** is a voice agent in active conversation calling a tool. The latency budget is sub-second. Batch is disqualified.
The canonical Batch API workloads are: offline evaluation runs, dataset labeling, batch summarization of long documents, nightly enrichment pipelines, agentic experiments that fan out many candidates per task, and any analytical job where the result is consumed by another batch process rather than a person. Option B fits this pattern cleanly. A is the trap that tests whether you can recognize cost optimization is irrelevant when latency is the binding constraint.
Quick reference
TL;DR
Batch APIs trade ~50% cost discount for a 24-hour SLA, so only latency-tolerant offline workloads qualify; nightly evaluation runs are the canonical fit, latency-bound chat or tool workflows are disqualified outright.
Picture a printing service with two counters. The express counter prints your document in minutes but costs double. The economy counter prints it for half the price but does not promise it until tomorrow. You pick based on whether you need it today or whether you just need it eventually. Anything a user is actively waiting on must go through the express counter. Anything you queue up overnight should go through the economy counter to save money. Provider Batch APIs are that economy counter for LLM calls. The model and the quality are identical; the only difference is when you get your answer back.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads with a human or tool waiting on the response The 24-hour Batch SLA is incompatible with sub-second latency budgets, so cost savings are irrelevant when latency cannot be met.
Streaming output requirements Batch APIs return complete responses after job completion; they do not support token by token streaming.
Workloads with strict data-residency or privacy SLAs forbidding extended provider storage Batch jobs hold input and output data in the provider's storage for the SLA window, which may violate compliance requirements.
Quick scan · 2–4 min
Everything important, quickly.
3 min: same model and quality + 50 percent discount + 24-hour SLA + latency tolerance is the only discriminator + option A is the trap + B fits because no human waits per row + canonical workload list for Batch.
Side-by-side
Endpoint
Latency
Cost
Best for
Realtime
Sub-second to a few seconds
Full input and output rates
Chat, autocomplete, voice agents, tool calling, any human or tool waiting
Batch
Up to 24-hour SLA
Roughly 50 percent off input and output rates
Eval runs, dataset labeling, batch summarization, nightly enrichment, synthetic data
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's /v1/batches endpoint serves GPT-5.5 with a 50 percent discount for nightly evaluation runs and dataset labeling pipelines.
2Anthropic's Message Batches API serves Claude Opus 4.7 and Sonnet 4.6 with similar 50 percent off pricing for offline workloads.
3Google Gemini 3.1 Pro offers batch processing for large-scale enrichment jobs at discounted rates.
4Machine-learning teams routinely move LLM-as-judge eval runs to Batch APIs to cut quarterly compute spend without affecting model quality.
5Synthetic data generation pipelines for fine-tuning Llama 4 Scout often use Batch APIs since latency per generation is irrelevant.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you operationally split a workload into realtime and Batch tiers?
Sign in or go Pro to see the approach for this follow-up.
QWhy do providers offer a 50 percent discount on Batch rather than a smaller one?
Sign in or go Pro to see the approach for this follow-up.
QAre there workloads where Batch is the wrong choice even though the SLA fits?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking the Batch API for a workload that has a human or tool waiting on the response. The cost savings are real but irrelevant when the latency budget is sub-second.