Explain why batch embedding APIs cost ~50% less, and when that discount actually matters

An embedding API turns text into a vector of numbers so you can search by meaning. Vendors like OpenAI, Voyage, and Cohere offer two flavors: a regular synchronous API that returns vectors in milliseconds. A batch API that returns them within roughly 24 hours for about half the cost. The 50% discount is not a marketing gimmick. GPU servers cost the same to run whether they sit idle or not, so providers want to keep them busy. Synchronous traffic spikes during business hours and falls off overnight, leaving gaps. Batch jobs can be slotted into those gaps without competing with realtime customers, which is genuinely cheaper to serve. The discount is useful when you have a lot of text to embed and no immediate deadline. Indexing a one million document corpus before launch, re-embedding everything after switching from text-embedding-3-small to a new model, or running an evaluation sweep across thousands of queries are all good batch jobs. The discount does not help when a user is waiting for a search result.

The 50% discount is real-cost passthrough. Embedding providers run large GPU fleets sized for synchronous peak traffic, which leaves substantial idle capacity overnight and on weekends. The batch API gives the scheduler permission to use those idle windows by accepting a flexible deadline (usually 24 hours). They share the savings back as a roughly half-price rate. Where it matters in practice: initial corpus indexing before launch, re-embedding the entire corpus after a model migration, nightly incremental re-indexing of a content firehose, evaluation runs over labeled retrieval sets. These are the workloads that dominate token volume but have no user-facing latency requirement. Where it does not matter: live query embedding (a search box needs sub-second turnaround), realtime ingest where new documents must be searchable within seconds, anything user-facing. Trying to route query traffic through batch defeats the entire product. The pragmatic pattern is dual-use: synchronous API for the user-facing path, batch API for backfills and migrations. In most RAG systems the token volume splits roughly 95% backfill versus 5% live, so the batch discount applies to almost all the spend.