Which workloads move onto your own GPUs in a hybrid hosted plus self-host setup?
Self-host the steady, high-volume workloads a small model handles well plus residency-bound traffic. Keep spiky frontier-quality calls on hosted APIs where elasticity is free.
Think of GPUs like a private bus you lease for a year. The lease only pays off if the bus is full most of the day, every day. Steady commuter routes fit perfectly. But a sports stadium that fills once a week is wasteful: you pay all week for one packed trip, when an Uber surge would have been cheaper. Hosted APIs are the Uber. Self-hosted GPUs are the leased bus. Pick each workload by asking which shape it has. The bus also makes sense when the route itself is private (sensitive data that legally cannot ride a public taxi), no matter the volume. The wrong move is to lease one bus for everything. It is also wrong to keep using Uber for the dense daily commute when the bus would already pay for itself.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The question "hosted or self-hosted?" sounds like an architecture choice, but it is a portfolio decision. Every workload your product runs has a different traffic shape, quality requirement, and data sensitivity. The right home for one workload is wrong for another. The job is to assign each workload to the tier whose cost curve and constraints fit it. Then connect both tiers behind a routing gateway, so you can retune the split without rewriting application code.
Mental model: hosted is rent, self-hosted is mortgage. Both make sense, for different things, and almost every serious LLM product ends up holding both.
The two axes that decide each workload
Axis one: traffic shape
Owned GPUs are a fixed cost the moment you provision them. Whether you run them at 5 percent utilization or 85 percent, the monthly bill is roughly the same. So the only way self-hosting beats hosted on tokens per dollar is to keep the GPUs busy.
Workloads with steady, predictable load (batch processing, high-QPS classification, embedding refresh, internal chatbots used in business hours) naturally drive high utilization. This is especially true when you shape low-priority work to fill troughs. Workloads with spiky load (occasional product launches, irregular research queries, end of quarter report generation) leave GPUs idle most of the time. Hosted APIs absorb spikes for free because someone else is amortizing the capacity across thousands of tenants.
Axis two: data sensitivity
If the data is restricted by contract, regulation, or residency, the cost axis becomes secondary. A workload that processes EU personal data may be barred from leaving an EU region. A healthcare workload may be barred from leaving your perimeter at all. Both push toward self-hosted, or at minimum toward a sovereign hosted endpoint that meets the residency terms. This holds regardless of how cheap a general hosted call would be.
Putting the axes together
Steady plus non-sensitive: cleanest win for self-hosting if a smaller model is acceptable. Spiky plus non-sensitive: hosted is obviously right. Spiky plus sensitive: hosted in a sovereign region or a small self-hosted reserve sized for the average. Steady plus sensitive: the clearest self-host case, because both axes agree.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic and OpenAI publish hosted APIs as the elastic tier. Most enterprise customers route frontier reasoning through Claude Opus 4.7 or GPT-5.5. They keep a self-hosted small open-weight model such as Qwen3.5 or Gemma 4 on vLLM or SGLang for the high-volume classification path.
- AWS Bedrock and Azure OpenAI offer regional and sovereign endpoints so that residency-bound workloads can stay hosted without violating contracts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which model to fine-tune for the self-hosted tier?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing hosted per-token price to self-hosted per-token price at 100 percent utilization. The real comparison is against your actual average utilization, often 25 to 40 percent.
60 second bullets to scan on the way to the call.
What decides whether a workload should sit on owned GPUs or hosted APIs
Why average utilization (not peak) sets the real self-host cost
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.