GGUF
Also known as: GGML successor
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
A binary file format for quantized LLM weights popularized by llama.cpp. Bundles weights, vocab, hyperparams, and metadata in one file optimized for CPU and Apple Silicon inference.
In practice
If you've used Ollama you've used GGUF. Production interviews probe its trade-offs vs safetensors and provider APIs.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.