Pick which workloads move onto your own GPUs in a hybrid hosted plus self-host setup
Self-host the steady, high-volume workloads a small model handles well plus residency-bound traffic. Keep spiky frontier-quality calls on hosted APIs where elasticity is free.
Think of GPUs like a private bus you lease for a year. The lease only pays off if the bus is full most of the day, every day. Steady commuter routes fit perfectly. But a sports stadium that fills once a week is wasteful: you pay all week for one packed trip, when an Uber surge would have been cheaper. Hosted APIs are the Uber. Self-hosted GPUs are the leased bus. Pick each workload by asking which shape it has. The bus also makes sense when the route itself is private (sensitive data that legally cannot ride a public taxi), no matter the volume. The wrong move is to lease one bus for everything, or to keep using Uber for the dense daily commute when the bus would already pay for itself.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The question "hosted or self-hosted?" sounds like an architecture choice, but it is really a portfolio decision. Every workload your product runs has a different traffic shape, quality requirement, and data sensitivity, and the right home for one workload is wrong for another. The job is to assign each workload to the tier whose cost curve and constraints actually fit it, then connect both tiers behind a routing gateway so the split can be retuned without rewriting application code.
This walkthrough breaks down the two axes that decide each assignment, the utilization math that makes self-hosting pay or not pay, the operational realities that come with each tier, and the routing patterns that let a hybrid stack scale gracefully.
Mental model: hosted is rent, self-hosted is mortgage. Both make sense, for different things, and almost every serious LLM product ends up holding both.
The two axes that decide each workload
Axis one: traffic shape
Owned GPUs are a fixed cost the moment you provision them. Whether you run them at 5 percent utilization or 85 percent, the monthly bill is roughly the same. So the only way self-hosting beats hosted on tokens per dollar is to keep the GPUs busy.
Workloads with steady, predictable load (batch processing, high-QPS classification, embedding refresh, internal chatbots used in business hours) naturally drive high utilization, especially when you shape low-priority work to fill troughs. Workloads with spiky load (occasional product launches, irregular research queries, end of quarter report generation) leave GPUs idle most of the time. Hosted APIs absorb spikes for free because someone else is amortizing the capacity across thousands of tenants.
Axis two: data sensitivity
If the data is restricted by contract, regulation, or residency, the cost axis becomes secondary. A workload that processes EU personal data may be barred from leaving an EU region; a healthcare workload may be barred from leaving your perimeter at all. Both push toward self-hosted, or at minimum toward a sovereign hosted endpoint that meets the residency terms, regardless of how cheap a general hosted call would be.
Putting the axes together
Steady plus non-sensitive: cleanest win for self-hosting if a smaller model is acceptable. Spiky plus non-sensitive: hosted is obviously right. Spiky plus sensitive: hosted in a sovereign region or a small self-hosted reserve sized for the average. Steady plus sensitive: the slam-dunk self-host case, both axes agree.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic and OpenAI publish hosted APIs as the elastic tier; most enterprise customers route frontier reasoning through Claude Opus 4.7 or GPT-5.5 and keep self-hosted Llama 4 Maverick on vLLM or SGLang for the high-volume classification path.
- AWS Bedrock and Azure OpenAI offer regional and sovereign endpoints precisely so residency-bound workloads can stay hosted without violating contracts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which model to fine-tune for the self-hosted tier?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing hosted per-token price to self-hosted per-token price at 100 percent utilization. The real comparison is against your actual average utilization, often 25 to 40 percent.
60 second bullets to scan on the way to the call.
What decides whether a workload should sit on owned GPUs or hosted APIs
Why average utilization (not peak) sets the real self-host cost
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.