What is GGUF and how do llama.cpp / mlc-llm enable on device inference where vLLM cannot run?
Describe the GGUF format and explain why edge runtimes like llama.cpp and mlc-llm exist when production grade serving stacks (vLLM, TensorRT-LLM) target server GPUs. What hardware do they actually run on?
GGUF is a single-file, mmap-able quantized model format; llama.cpp and mlc-llm run it on CPUs, Apple Silicon, phones, and browsers, where the CUDA-bound vLLM stack cannot execute.
A datacenter serving stack is like an industrial kitchen: huge ovens, three-phase power, a loading dock for ingredients. It cooks for thousands of diners at once but cannot fit in your apartment. Edge inference is a microwave meal: one portion, on the counter you already own, no delivery driver, no bill per plate. GGUF is the shrink-wrapped meal packaged so it drops straight into that microwave with nothing else needed. llama.cpp and mlc-llm are two microwave brands. One ships hand-built heating coils tuned for each counter. The other reads your specific microwave and compiles a custom heating program. Neither matches the industrial kitchen on volume or finish, but the food is private, instant, and free to reheat.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Edge inference is the discipline of running an LLM on hardware the user already owns: a laptop, a phone, a tablet, or even a browser tab, with no datacenter GPU in the loop. It is a genuinely different engineering problem from production serving, and interviewers use it to separate candidates who understand serving as a system from those who have only memorised the vLLM playbook.
The whole field exists because the production stack makes assumptions that the edge violates. vLLM and TensorRT-LLM assume an NVIDIA datacenter GPU, CUDA, abundant fast memory, and many concurrent requests to batch. Strip those assumptions away and almost nothing transfers: no CUDA, often no discrete GPU at all, a few gigabytes of memory shared with the operating system, a battery that punishes sustained compute, and exactly one user. Each missing assumption forces a concrete design change, and the GGUF format plus its runtimes are the accumulated set of those changes.
This deep dive covers what GGUF actually is, why the CUDA-locked stack cannot follow you onto a device, how llama.cpp and mlc-llm solve the same problem through opposite kernel strategies, the hardware they reach, and the honest tradeoff between autonomy and capability that decides when edge inference is the right call. The throughline is simple. On a server you are buying throughput per dollar; on a device you are buying autonomy, and the entire toolchain is shaped by that goal.
What GGUF actually is
GGUF, the GGML Universal Format, is a container file, not a model architecture and not a quantization algorithm. It bundles everything a runtime needs into one self-contained file: the quantized weight tensors, the quantization metadata that says how to dequantize each block, the tokenizer, the chat template, and model metadata like layer count, head count, and context length. The name to remember is container, because conflating it with a quant method is the single most common slip on this question.
Two design choices matter. First, it is a single file, so deploying a model is copying one artifact with no separate config, no sharded checkpoints, and no external tokenizer file to keep in sync. Second, the layout is designed to be memory-mapped. The runtime maps the file into its address space and the operating system pages weights in on demand, which gives near-instant cold starts and lets multiple processes share one read-only copy of the weights in RAM. On a device where memory is shared with everything else, that sharing is not a nicety, it is what makes running the model alongside a browser feasible.
GGUF replaced the older GGML and GGJT formats precisely to add this metadata-rich, forward-compatible structure, so a newer runtime can read an older file and a file can carry fields a runtime does not yet understand. It standardises a family of low-bit quantization types, the K-quants from roughly 2-bit to 8-bit, which is what lets a model that would need fourteen gigabytes in fp16 ship in roughly four. The quant type is recorded per tensor, so the runtime knows exactly how to unpack each block at load time.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Ollama wraps llama.cpp and GGUF to run Llama 4 and Qwen 3 locally on a MacBook with a single pull command in 2026.
- LM Studio ships a desktop app over llama.cpp, loading GGUF quants of DeepSeek and Mistral models on consumer hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does GGUF emphasize being mmap-able rather than just compact on disk?
Memory mapping lets the OS page weights in lazily and share read-only pages across processes. Cold start drops because you skip a full load into a parsing pipeline, and several apps can reference one resident copy of the weights.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating edge runtimes as a scaled-down vLLM. They are a different architecture: no PagedAttention, no CUDA assumption, single-stream rather than high-throughput batching, and a quality ceiling set by tiny quantized models.
60 second bullets to scan on the way to the call.
What GGUF bundles into one file and why mmap matters
Why vLLM and TensorRT-LLM cannot run on a laptop or phone
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.