OpenAI's batch embedding API and similar offerings from other vendors discount embedding cost by ~50% in exchange for a relaxed (e.g. 24 hour) latency SLA. Explain the underlying economic reason and identify the workloads where the discount matters vs where it's irrelevant.
Batch embedding APIs discount ~50% because relaxed-latency jobs let providers fill idle GPU capacity, and the savings only matter for backfill-style workloads where 24-hour turnaround is acceptable.
Imagine a pizza shop with two prices. If you want a pizza in twenty minutes during the Friday night rush, you pay full price because the kitchen has to drop everything for you. If you say I just need five hundred pizzas sometime tomorrow, do them whenever, the shop slots your order into the slow Tuesday afternoon shift and charges you half. The big cooking ovens behind the counter work the same way for any bulk service. The overnight lane tells the kitchen do these orders whenever you would otherwise be standing around, and they pass the savings back to you. You only benefit if your work can actually wait. Converting an entire backlog of old documents into searchable fingerprints fits perfectly. A search box that needs an answer in two hundred milliseconds does not.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every embedding vendor in 2026 ships two endpoints. The synchronous one returns vectors in tens of milliseconds and charges list price. The batch one returns vectors within roughly 24 hours and charges about half. The question 'why is batch half the price' looks like a pricing trivia question, but it actually probes something deeper: do you understand the unit economics of GPU inference, and can you design a pipeline that exploits them.
This deep dive walks through the underlying economics, the architectural pattern that production RAG systems use to capture the savings, the failure modes that come from misusing the discount, and the point at which the discount stops helping and you should be considering self-hosted open-weights embedders instead.
Why GPU inference is cheaper when you can wait
Inference fleets are sized for the synchronous peak. Providers look at the busiest minute of the busiest day of the week and provision enough GPU capacity to serve it without queueing. Outside that peak (overnight, weekends, holidays) a sizeable fraction of the fleet is idle. The GPUs still cost money to power, depreciate, and host in a data center, but no revenue flows through them.
The batch endpoint is the scheduler's tool for absorbing that slack. A batch job arrives with a soft deadline (usually 24 hours), and the scheduler is free to slot it into any idle window during that interval. The provider trades latency flexibility for fleet utilization, and the savings are real: not a discount on margin, but a reduction in marginal cost. Pricing the batch endpoint at roughly half of synchronous is approximately the spread between fully-utilized and partially-utilized fleet economics.
Why this is structural, not promotional
The discount is sometimes mistaken for a promotional rate that might disappear. It is not. The same pattern shows up across cloud spot instances, off-peak electricity tariffs, and overnight shipping rates. Anywhere infrastructure capacity is provisioned for the peak and demand is bursty, there is an economic case for a deferred-execution lane at a lower price. Embedding APIs are just the latest expression of it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | Synchronous API | Batch API |
|---|---|---|
| Latency SLA | milliseconds to seconds | up to 24 hours |
| Price | list price | ~50% of list |
| Best for | live query, realtime ingest | corpus backfill, migration, eval |
| Failure handling | retry inline | job-level retry, async callback |
| Concurrency cap | rate limits per minute | job-size cap, concurrent jobs cap |
Real products, models, and research that use this idea.
- OpenAI Batch API offers 50% off on text-embedding-3-small and text-embedding-3-large with a 24-hour SLA, the canonical reference for this pricing model.
- Voyage AI offers a similar batch discount on voyage-3 and voyage-code-3 for bulk re-indexing workloads.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design the cache key for an embedding pipeline that uses both batch and sync APIs?
Same key shape for both paths: content hash plus model id plus model version. The path that produced the vector should not change the lookup.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the discount applies to every embedding call and discovering at scale that latency-sensitive query traffic is blocking on a 24-hour SLA.
60 second bullets to scan on the way to the call.
Economic reason the batch discount is roughly 50%
Workloads that are appropriate for the batch endpoint
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.