What is GGUF and how do llama.cpp / mlc-llm enable on-device inference where vLLM cannot run?

Attempt it

Describe the GGUF format and explain why edge runtimes like llama.cpp and mlc-llm exist when production-grade serving stacks (vLLM, TensorRT-LLM) target server GPUs. What hardware do they actually run on?

Free · 2 AI evals / day

Quick reference

TL;DR

GGUF is a single-file, mmap-able quantized model format; llama.cpp and mlc-llm run it on CPUs, Apple Silicon, phones, and browsers, where the CUDA-bound vLLM stack cannot execute.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

A datacenter serving stack is like an industrial kitchen: huge ovens, three-phase power, a loading dock for ingredients. It cooks for thousands of diners at once but cannot fit in your apartment. Edge inference is a microwave meal: one portion, on the counter you already own, no delivery driver, no bill per plate. GGUF is the shrink-wrapped meal packaged so it drops straight into that microwave with nothing else needed. llama.cpp and mlc-llm are two microwave brands. One ships hand-built heating coils tuned for each counter. The other reads your specific microwave and compiles a custom heating program. Neither matches the industrial kitchen on volume or finish, but the food is private, instant, and free to reheat.

Key concepts

Edge inference GGUF format Quantization On-device LLM

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: GGUF as a single mmap-able container, why vLLM is CUDA-locked, llama.cpp hand-tuned kernels versus mlc-llm TVM compilation, target hardware, and the privacy, latency, and cost wins against the capability ceiling.

Real-world examples

Real products, models, and research that use this idea.

Ollama wraps llama.cpp and GGUF to run Llama 4 and Qwen 3 locally on a MacBook with a single pull command in 2026.
LM Studio ships a desktop app over llama.cpp, loading GGUF quants of DeepSeek and Mistral models on consumer hardware.
Apple ships on-device foundation models for Apple Intelligence, running quantized models directly on M-series and A-series Neural Engines.
mlc-llm's WebLLM demo runs Llama-class models entirely in the browser via WebGPU, with zero server inference.
Google's Gemini Nano runs on-device on Pixel phones through the Android AICore runtime for private summarization and reply suggestions.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does GGUF emphasize being mmap-able rather than just compact on disk?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you choose mlc-llm over llama.cpp despite the extra compile step?

Sign in or go Pro to see the approach for this follow-up.

QHow does 4-bit quantization change the memory math for a 7B model on a laptop?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating edge runtimes as a scaled-down vLLM. They are a different architecture: no PagedAttention, no CUDA assumption, single-stream rather than high-throughput batching, and a quality ceiling set by tiny quantized models.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.