At what point does self-hosting on vLLM beat a managed LLM API?
You're choosing between a managed API (OpenAI/Anthropic) and self-hosting an open model on vLLM/TGI. What factors decide the crossover, and why isn't per token sticker price the deciding one?
Self-hosting beats a managed API once sustained token volume amortizes fixed GPU cost — or when privacy and custom-model needs force it, not because of sticker price.
Imagine you need a car. Renting (the managed API) costs a few dollars per trip — perfect if you drive rarely. Buying a car (self-hosting GPUs) means a big upfront cost plus insurance, parking, and oil changes, but it gets cheap per trip once you drive every day. The trick is not comparing the per-trip price. It's asking how much you'll drive, whether you need a special vehicle nobody rents, and whether you want to handle the maintenance yourself.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every team building an LLM product hits this fork: keep calling a managed API, or stand up open models on your own GPUs. The seductive but wrong way to decide is to put OpenAI's per-token price next to a GPU's per-token cost and pick the smaller number. That comparison hides almost everything that actually matters.
The real question is about cost shape, not a single price. A managed API bills per token, so its total cost is a straight line through the origin — usage doubles, bill doubles. Self-hosting is a big fixed cost (the GPUs are rented or owned whether or not they're busy) plus an operational tax. Its effective per-token cost depends entirely on how hard you keep those GPUs working.
This section walks through where the two curves cross, why utilization is the lever that moves the crossover, the operational burden the sticker price hides, and the hard constraints — privacy, custom models, SLAs — that can settle the decision before any arithmetic happens.
Two cost curves with very different shapes
Picture cost on the y-axis and monthly token volume on the x-axis. The managed API is a straight line from the origin: every token costs the same, so the line's slope is the API's per-token rate. There's no floor and no ceiling — pay for exactly what you use.
Self-hosting looks completely different. You commit to a number of GPUs up front, and that fixed cost exists at zero tokens and at a billion tokens. On top of it sits an operational cost that also doesn't shrink to zero. So the self-host curve starts high (the fixed cost) and rises slowly.
The two lines cross at one volume. Left of the crossover, the API's linear line is below the self-host curve — paying per call beats paying for mostly idle silicon. Right of the crossover, the self-host curve's gentle slope wins because you've spread that big fixed cost across enormous volume.
The whole game is finding that crossover for your workload and asking which side of it you live on — today and a year from now.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Factor | Managed API | Self-hosted (vLLM/TGI) |
|---|---|---|
| Cost shape | Linear, pay per token | Fixed GPU cost + ops, falls with utilization |
| Best volume | Low or spiky | High and sustained |
| Operational burden | Hidden in price | Autoscaling, batching, upgrades, on-call |
| Data residency / custom models | Vendor-limited | Full control |
| Tail-latency control | Shared pool | Direct, tunable |
Real products, models, and research that use this idea.
- vLLM and TGI (Hugging Face) are the standard self-hosting serving stacks teams benchmark against OpenAI and Anthropic APIs.
- OpenAI and Anthropic managed APIs win for spiky or low-volume products where idle GPU cost would dominate.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the break-even token volume for a specific open model on A100s versus the OpenAI API?
Compute GPU hourly cost divided by sustained tokens/sec to get a per-token cost, then compare to the API rate at realistic 50-70% utilization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing the per-token API price against the per-token GPU price and ignoring the operational cost of running your own serving stack.
60 second bullets to scan on the way to the call.
Why per-token sticker price is the wrong unit of comparison
Shape of the API cost curve versus the self-host cost curve
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.