Glossary · Production

API LLM

Also known as: Hosted LLM, Provider API

An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.

A large language model accessed through a provider API (OpenAI, Anthropic, Google, Bedrock). The provider owns weights, inference infrastructure, and scaling; you pay per input and output token. Ops burden is minimal and the model ceiling is the frontier, at the cost of data leaving your premises unless you use a private-cloud tier.

In practice

The API-vs-self-hosted decision is a canonical applied-AI interview question. Break-even volume, data residency, and fine-tuning control all trace back to this choice.

How it compares

vs. Self-hosted LLM →Deep comparison →

APIs give you the frontier model with zero ops; self-hosting gives you data control, unrestricted fine-tuning, and predictable cost past break-even.

Comparisons that include API LLM

API vs Self-hosted LLMsAPI providers give you the best models with zero ops. Self-hosting gives you data control, custom fine-tunes, and predictable cost past a break-even volume, usually 10 to 50 million tokens per day.

Related terms

KV Cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

Quantization

Run the model at lower numerical precision to save memory and accelerate inference.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

Grouped-Query Attention (GQA)

Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.