When an async batch API beats the real-time endpoint

Attempt it

Use a batch API for non-interactive jobs that tolerate minutes to hours of latency: offline data enrichment, evaluation runs, embeddings backfills, bulk classification. You trade immediacy for a large per-token discount (often ~50%) and isolation from your interactive serving capacity, so heavy background work doesn't compete with live user traffic. Use the real-time endpoint only when a human or synchronous system is waiting on the response.

Your answer

Quick reference

TL;DR

Use a batch API for non-interactive jobs that can wait minutes to hours; you trade immediacy for a large discount (often ~50%) and isolation from live serving capacity.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you can either mail a single letter overnight for full price, or drop a big stack into a bulk service that delivers within a day for half the cost. If you need an answer right now, you pay for speed. If nobody's waiting, you save by letting them handle it whenever they have spare capacity. An LLM batch API is the bulk service. For work where no person is sitting there waiting — processing yesterday's data, scoring a big pile of records overnight — you hand the whole stack over, accept that it comes back later, and pay much less. You only use the instant, full-price option when someone is actually waiting on the reply.

Key concepts

LLM System Design Inference Optimization

Quick scan · 2–4 min

Everything important, quickly.

Spend 5-7 minutes framing it as an SLA decision, then cover the discount, capacity isolation, the canonical workloads, and the async retrieval flow.

Side-by-side

Dimension	Real-time endpoint	Batch API
Latency	Seconds	Minutes to hours (often up to ~24h)
Cost	Full per-token rate	Discounted, often ~50% off
Capacity	Shares interactive serving pool	Isolated from live traffic
Use when	A human or system is waiting	Nothing is waiting on the result

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Batch API processes jobs within 24 hours at roughly 50% off the synchronous price for offline workloads.
Anthropic's Message Batches API offers a discount for asynchronous bulk processing of Claude requests.
Teams run nightly embeddings backfills and eval suites through batch endpoints to avoid contending with live chat traffic.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can providers offer batch at roughly half the price?

Sign in or go Pro to see the approach for this follow-up.

QHow does batch protect live user latency beyond just cost?

Sign in or go Pro to see the approach for this follow-up.

QWhat sits between real-time and batch for latency-tolerant but not overnight work?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.