Finance flags your serving budget: 'The MoE only computes 37B of its 671B parameters per token, why are we paying for a GPU fleet sized like a 671B model?' Defend the fleet size by explaining what actually drives MoE serving cost.
Free · 2 AI evals / day
**MoE serving cost** has two axes that finance often confuses. Total parameters set how much memory you need. Active parameters set how fast each token is computed.
When DeepSeek-V3 says 37B of 671B parameters activate per token, that 37B is compute, not storage. Routing decides per token, per layer which experts run. Any token can pick any expert, so every expert must already be loaded in GPU memory. You cannot stream them in at request time without destroying latency.
That memory floor is what fixes the GPU count. You need enough VRAM across the fleet to hold all 671B parameters plus KV cache plus activation buffers. Once the fleet exists, the 37B active-FLOPs figure tells you how many tokens per second each GPU can produce. That is where MoE earns its keep: dense-671B quality at the throughput of a much smaller model.
The honest pitch to finance: this looks like a big model on the hosting bill and a small model on the per-token cost. Quality per dollar, not hardware count alone, is the right metric.
**Fleet size for an MoE is a memory question. Per-token cost is a FLOPs question. Conflating them is the finance bug here.**
The MoE router picks experts per token, per layer, dynamically. There is no static subset you can host: any future token might route to any expert. That means every parameter sits in VRAM all the time. For DeepSeek-V3 at 671B parameters in fp8, the weights alone are roughly 670 GB, plus KV cache that scales with concurrent users and context length, plus activation and routing buffers. That sets the GPU count floor before a single request lands.
Once the fleet exists, the 37B active figure does pay off. Per-token compute is a sparse 37B forward, not a dense 671B forward, so each GPU can serve far more tokens per second than it could if you were hosting a true 671B dense model on the same hardware. Throughput per dollar is where the win shows up.
There are also secondary costs: expert-parallel all to all dispatch between GPUs, capacity headroom to absorb routing skew without dropping tokens, and KV cache memory that grows with concurrency. The right finance metric is quality per dollar or cost per token at target quality, not GPU count compared to active-parameter count.
Quick reference
TL;DR
MoE fleet size is set by resident memory for all experts, not by active FLOPs. Active params only set throughput per GPU once the fleet exists.
Picture a library where every reader only checks out two books per visit. You might say the library only needs two shelves. But the library never knows which two books any reader will want, so it must keep every book on the shelf, ready. The number of shelves is fixed by the whole collection, not by what one reader borrows. The MoE serving fleet is that library. Total parameters set how many shelves (GPUs) you build. Active parameters set how fast each visit goes once the reader is inside. A small visit per reader does not shrink the building. What it does is let many more readers pass through per hour, because each visit is quick.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-tenant low-QPS workload where you can swap experts off-GPU between requests Offload changes the memory equation; latency tolerance lets you trade VRAM for slower per-request loading.
Comparing two MoE models with similar total parameters The memory floor is similar, so per-token throughput and quality matter more than the fleet-size discussion.
Edge or on-device deployment You cannot fit the full expert set; distillation to a dense student is the right move, not a fleet-cost argument.
Quick scan · 2–4 min
Everything important, quickly.
6 min: the two cost axes + memory floor mechanics + per-token throughput math + secondary costs + the right finance metric.
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek-V3 (671B total, 37B active) public guidance states multi-node H100/H200 deployments are required for serving; you cannot fit it on 37B-class hardware.
2Mixtral 8x22B (141B total, 39B active) requires roughly 8 x H100s in fp8 for production serving, not the smaller fleet a dense 39B would need.
3DBRX (132B total, 36B active) Databricks serving guidance specifies the memory footprint is set by total parameters and concurrent KV cache, not active count.
4Qwen3-MoE 235B-A22B documentation explicitly distinguishes the storage footprint from active compute when sizing inference clusters.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you could offload cold experts to host RAM and stream them in, how would the cost story change?
Sign in or go Pro to see the approach for this follow-up.
QHow does KV cache memory interact with total-parameter memory at long context?
Sign in or go Pro to see the approach for this follow-up.
QHow would you set capacity factor in production, and how does that propagate into the GPU count?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Equating MoE active-parameter count with a smaller hosting bill. Active params drive per-token compute; total params drive the VRAM floor that fixes GPU count.