Why is GGUF the dominant on device LLM format rather than a PyTorch state dict file?
GGUF wins on edge because it bundles weights, tokenizer, and quantization tables into one mmap-able file that a single C++ binary runs with no Python or PyTorch runtime.
Think about shipping a board game to a friend. A PyTorch checkpoint is like mailing the pieces in one box, the rules in another, the dice in a third, and a note saying you also need a special table from a furniture store to play. A GGUF file is the whole game in one sealed box: pieces, rules, dice, all inside, and it works on any table you already own. You hand it over, your friend opens it, and they play immediately. No hunting for extra parts, no buying special equipment. That self-contained, runs-anywhere packaging is exactly what a phone or laptop needs, because those devices cannot install the heavy workshop a server uses.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Ask why GGUF beats a PyTorch state-dict on a phone and you are really asking what makes a format deployable on the edge. The answer is not accuracy and it is not raw speed. It is the runtime contract: what the file contains, what software is needed to run it, and how the device's memory behaves while it runs. Get that framing right and the MCQ answers itself.
A PyTorch checkpoint is a server artifact. It assumes a Python interpreter, the torch library, a separate tokenizer file, and a config describing the architecture. On a data-center GPU that is fine, because you build the container image and pin every version. On a consumer laptop, an Android phone, or an embedded Linux board, that stack is heavy, fragile, or outright unavailable. The user never installed Python and never will.
The edge ecosystem has converged on a small set of formats, each with a distinct contract: GGUF for the CPU and unified-memory world, CoreML for Apple devices, TFLite now called LiteRT for Android, and ONNX as a cross-platform interchange layer. Every one of them assumes the same hard truth: memory is the binding constraint, so weights must be aggressively quantized. This deep dive walks through why packaging matters, what each format targets, why low-bit quantization is non-negotiable, and how to choose between them under interview pressure.
The packaging problem GGUF solves
A PyTorch deployment is a bag of loosely coupled parts. You ship the checkpoint, a tokenizer, a config file, and you assume the target machine already has Python and torch installed at compatible versions. Version skew between any of those parts breaks the load. On a server you control the image and this is a non-issue. On a user's device you control almost nothing.
GGUF collapses the whole bag into one self-contained file. The quantized weights, the tokenizer, the quantization tables, and the architecture metadata all live inside it. There are no sidecar files to lose and no version handshake to negotiate. You hand over one artifact and it carries its own description of how to be loaded.
The file is also designed to be memory-mapped. Instead of reading the whole model into RAM, the runtime maps the file into virtual memory and lets the operating system page weights in on demand. Cold start becomes nearly instant, the resident footprint can stay below the full model size, and multiple processes can share a single mapped copy. On a 16 GB laptop running a quantized 8B model, that difference is the difference between usable and unusable.
The layout also makes the file forward-compatible. Metadata is stored as typed key-value pairs, so a newer model can add fields describing a new architecture without breaking older readers that ignore what they do not understand. That is why a single llama.cpp build can load models released long after it was compiled, and why the ecosystem moves fast without constant format churn.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Format | Primary runtime | Best target | Accelerator reach | Typical quantization |
|---|---|---|---|---|
| GGUF | llama.cpp | CPU, Apple Silicon, WASM | CPU and GPU, not the Neural Engine | 2 to 8 bit (k-quant, i-quant) |
| CoreML | Apple CoreML | iOS and macOS | Apple Neural Engine, GPU, CPU | Palettization, 4 to 8 bit |
| TFLite / LiteRT | LiteRT runtime | Android | NNAPI, GPU delegate | int8, 4 bit |
| ONNX | ONNX Runtime | Cross-platform interchange | Vendor execution providers | int8, 4 bit |
Real products, models, and research that use this idea.
- Ollama and LM Studio distribute thousands of GGUF builds of Llama 4 and Qwen 3 that run locally on consumer laptops with no Python install.
- Apple ships on-device foundation models for Apple Intelligence through CoreML, dispatching to the Neural Engine on iPhone and Mac silicon.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does memory mapping a GGUF file matter so much on a constrained device?
Mmap lets the runtime treat the file as virtual memory rather than copying it into RAM. The OS pages in weights on demand, so cold start is fast and the resident set can stay below the full model size. Multiple processes can also share one mapped copy.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming GGUF is more accurate than other formats. Accuracy comes from the quantization method, not the container. GGUF's edge is packaging and a dependency free runtime.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.