Self-hosted LLM
Also known as: On-prem LLM, Open-weights deployment
An open-weights model on your own GPUs via vLLM or TGI. Data control, unrestricted fine-tuning, ops burden.
An open-weights model (Llama-4, Qwen-3, Mistral, DeepSeek-V3) served on your own GPUs, whether bare-metal, cloud, or a serving framework like vLLM or TGI. You pay per GPU-hour regardless of request rate, keep data in your VPC, and fine-tune without provider mediation. Predictable cost past a break-even volume of roughly 10 to 50 million tokens per day depending on the model.
In practice
Every applied-AI role weighs this against the API path. Interviewers probe break-even reasoning, ops burden, and where open weights have closed the frontier gap.
How it compares
Self-hosting caps at the strongest open weights and demands ops; APIs give you the frontier model but push data through the provider.
Comparisons that include Self-hosted LLM
Related topics
Questions that mention this term
- Qwen 3.5: when does running your own GPU beat paying the hosted API per token?
- Estimate the GPU nodes needed to self-host a 70B model at 24,000 tokens per second
- Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1 point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?
- Why a self-hosted LLM deployment needs a warm up step before traffic and how it differs from cache priming
- At what point does self-hosting on vLLM beat a managed LLM API?
- Sketch a Llama 3.1 70B self-host versus hosted API break even on the back of an envelope
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.