Every expert has to be resident because routing is per-token, so the only lever that shrinks the weight footprint below 24 GB is quantization.
Picture a library with eight specialist librarians. When you walk in, a quick clerk decides which two librarians will answer your question. You only consult two, but all eight must still be at their desks because the next visitor might need a different pair. If your building only has space for two desks, sending fewer visitors to fewer librarians does not help. You either need a bigger building or you need to ask each librarian to fold up into a much smaller desk. Quantization is the folding trick. It compresses every librarian into a quarter of the space so all eight fit, even though the building never grew.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A common interview trap with Mixture of Experts models is to conflate the active-parameter count with the memory footprint. Mixtral 8x7B advertises about 13B active parameters per token, which sounds like it should comfortably fit on a 24 GB consumer GPU. It does not, and the reason exposes a basic truth about how sparse models work.
This deep dive walks through the memory math, explains why routing does not let you skip loading any expert, contrasts the candidate levers (quantization, top-k, context length, tensor parallelism) and shows when each actually helps.
Active vs. total parameters: what each number describes
Mixtral has 8 experts per MoE layer. Each token's router picks 2 of them (top-k=2) and runs only those two FFNs. The compute per token is therefore proportional to the size of those 2 experts plus the shared attention block, giving roughly 13B effective parameters per forward pass.
Memory follows a different rule. The router is data dependent, so the next token could pick any pair. To serve any request you have to keep all 8 experts plus attention and routing weights in VRAM at once. The total parameter count is what sets that footprint, not the active count.
In numbers: 47B parameters at fp16 is about 94 GB. At fp8 it is about 47 GB. At int4 it is about 24 GB. The 24 GB card only has room at int4, and even then only just.
For a wider perspective, consider how this scales across the 2026 frontier MoE lineup:
- Mixtral 8x7B: 47B total, 13B active. At int4, weights are 24 GB; fits one consumer GPU.
- Mixtral 8x22B: 141B total, 39B active. At fp8, weights are 130 GB; needs 2 H100 80GB. At int4, 65 GB; fits a single H100.
- DBRX: 132B total, 36B active. Similar profile to Mixtral 8x22B at int4.
- Qwen3-MoE 235B: 235B total, 22B active. At fp8, 220 GB; needs 3 H100 80GB or 2 H200 141GB.
- DeepSeek-V3: 671B total, 37B active. At fp8, 670 GB; needs 5 H200 141GB nodes for weights alone.
- Llama 4 Maverick: 400B total, 17B active. At fp8, 380 GB; needs 5 H100 80GB or 3 H200 141GB.
Notice the pattern: active parameters cluster around 15 to 40B across the lineup, but total parameters span an order of magnitude. The fleet hardware bill is set by the total column, the per-token compute by the active column. Picking the right model for a deployment requires reading both columns together.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral 8x7B running on a 24 GB RTX 4090 via 4-bit NF4 with bitsandbytes or ExLlamaV2
- DBRX 132B fitting on dual H100s only after 4-bit quantization, despite having only 36B active parameters
What an interviewer would ask next. Try answering before peeking at the approach.
QSuppose top-k=1 made routing deterministic. Would that change the residency answer?
Routing is data dependent. Even with top-1 the chosen expert can be any of the eight, so all must be loaded unless you accept paging cost.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming sparse activation means sparse residency, so lowering top-k or dropping context length must free enough VRAM to fit the model.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.