Self-hosted LLM
Also known as: On-prem LLM, Open-weights deployment
An open-weights model on your own GPUs via vLLM or TGI. Data control, unrestricted fine-tuning, ops burden.
An open-weights model (Llama-4, Qwen-3, Mistral, DeepSeek-V3) served on your own GPUs, whether bare-metal, cloud, or a serving framework like vLLM or TGI. You pay per GPU-hour regardless of request rate, keep data in your VPC, and fine-tune without provider mediation. Predictable cost past a break-even volume of roughly 10 to 50 million tokens per day depending on the model.
In practice
Every applied-AI role weighs this against the API path. Interviewers probe break-even reasoning, ops burden, and where open weights have closed the frontier gap.
How it compares
Self-hosting caps at the strongest open weights and demands ops; APIs give you the frontier model but push data through the provider.
Comparisons that include Self-hosted LLM
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.