Spot the error: MoE inference and GPU memory
Click any words you think contain an error. Click again to unmark.
The error conflates conditional compute with conditional loading, all N expert weights typically stay in GPU memory; sparsity saves FLOPs, not RAM.
The claim is like saying a restaurant only keeps two chefs in the building because each order goes to two cooks. In reality, all eight chefs must be present because different orders need different people. You save work per order, not floor space.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Spot-the-error questions on MoE inference memory test a single conceptual distinction: conditional compute versus conditional storage. The erroneous text embeds a plausible-sounding claim, only top-k weights load, inactive experts stay off-device, that collapses two separate resource dimensions into one.
Candidates who miss this error often understand top-k routing mechanically but have not internalized how production serving engines lay out expert weights. The correction is short but precise; the deep dive explains why the error is tempting and why it is wrong at scale.
Walk through this like a code review: identify the false clause, state the correction, then explain the systems reasoning behind it.
Easy spot-error still requires precise language. The false clause claims demand-loaded experts, only top-k weights load per token, inactive experts stay off-device. Your correction names the conflation of conditional compute with conditional storage and anchors with a familiar number (Mixtral ~47B stored, ~13B active).
Easy spot-error still needs a numeric anchor, Mixtral ~47B stored versus ~13B active is the fastest credible example under time pressure.
Spot-error grading rewards the correction phrase "all N experts resident", use it every time.
Parsing the erroneous claim
The text makes two linked assertions inside one sentence. First: "only the top-k expert weights are loaded per token." Second: "inactive experts stay off-device until routed." Both describe demand-loaded expert weights, as if the serving runtime fetches expert matrices on the fly based on each token's routing decision.
The error range covers exactly this clause. The correction replaces it with the production reality: all N expert weights are typically resident in GPU memory, and sparsity saves FLOPs per token, not full-model RAM.
In spot-error format, your job is to name what is wrong and why. Do not just negate, explain the mechanism that makes the original claim fail.
Parse technique: underline "only top-k weights loaded" and "off-device until routed", both assert dynamic loading. Production stacks load the full bank at init; routing only selects which loaded weights compute.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral serving requires multi-GPU sharding for the full ~47B expert bank.
- vLLM MoE kernels assume expert weights are pre-loaded per device shard.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would break if you truly loaded only k experts per token?
Routing latency spikes, PCIe bandwidth saturation, unpredictable expert access patterns across concurrent requests.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing inactive experts stay off-device, standard serving loads the full expert bank.
60 second bullets to scan on the way to the call.
Identify the false clause in memory-myth claims
Sparsity saves FLOPs not RAM
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.