When an async batch API beats the real-time endpoint

Most LLM providers offer two ways to call the model: a **real-time endpoint** that answers in seconds, and a batch API that takes a whole job, processes it whenever it has spare capacity, and returns the results later — often within a few hours. The deciding question is simple: is anything waiting on the answer right now? If a person is staring at a screen, or another system is blocked until the response comes back, you need real-time. If the answer can come back in an hour or overnight, batch is the better fit. What you give up with batch is immediacy — you don't get an instant reply. What you get in return is a big price discount (often around half) and isolation, meaning your large background job runs separately and doesn't compete with the capacity serving your live users. Good batch jobs are non-interactive: enriching a database of records, running an evaluation set, backfilling embeddings, classifying a big pile of documents. None of those have a human waiting, so paying extra for instant responses would be wasteful.

Choose between the two endpoints by asking what's waiting on the result. If a human or a synchronous system is blocked on the response, use the **real-time endpoint**. If nothing is waiting and the work can tolerate minutes to hours of latency, use the batch API. The batch API trades immediacy for two concrete benefits: - **Cost.** A large per-token discount, often around 50% off the synchronous rate. The provider earns it by scheduling the work flexibly to fill spare capacity. - **Capacity isolation.** Batch jobs run off your interactive serving path. A big background workload doesn't consume the throughput or rate-limit headroom that live user traffic needs. The sweet-spot workloads are non-interactive and high-volume: offline data enrichment, evaluation and benchmark runs, embeddings backfills, bulk classification or summarization of a large corpus, synthetic data generation. None of these have a user waiting, so the latency cost is free and the discount is pure savings. The boundary is latency tolerance and SLA. Anything user-facing and synchronous — chat, autocomplete, an API a client blocks on — must stay on the real-time endpoint, where you pay the premium precisely because someone is waiting. A common production pattern is to route the same model both ways: real-time for live requests, batch for the nightly bulk processing.