How should a team disclose MoE model size for honest latency and cost comparisons?
How should a team disclose MoE model size for honest latency and cost comparisons?
Disclose both total params (memory) and active params per token (compute), plus top-k and expert count, a single headline number misleads buyers and infra teams alike.
A MoE model is like a hospital with eight specialist departments but only two treat each patient. Quoting 'eight departments' tells you building size (memory); quoting 'two doctors per patient' tells you wait time (compute). Honest reporting gives both numbers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MoE model size is inherently a two-number (or four-field) disclosure problem. Sparse activation decouples total capacity from per-token compute, and different stakeholders care about different sides of that decoupling. Interviewers ask this question to test product sense and systems honesty, can you write a spec that does not mislead buyers or infra teams?
The failure modes are symmetric: total-only misleads latency comparisons; active-only misleads memory provisioning. The fix is a structured model card with both numbers plus routing config.
This deep dive covers the disclosure template, stakeholder mapping, and common dishonest patterns in vendor materials.
Model size disclosure is a stakeholder problem disguised as arithmetic. Total params answer infra; active params answer latency and API cost; k and N let third parties verify your claims. Single-number marketing is either accidentally or deliberately misleading, interviewers want the four-field honest spec.
Product and infra candidates both get this question, frame answers as stakeholder service, not arithmetic exercise.
Treat every MoE headline number as guilty until proven paired with active, k, and N, the same skepticism you apply to benchmark claims without methodology footnotes.
The four-field honest spec
Minimum honest MoE disclosure:
- Total parameters, all expert weights + shared layers + router + embeddings
- Active parameters per token, shared layers + k expert FFNs that execute
- Top-k, how many experts activate per token per layer (or average if variable)
- Expert count N, routed experts in the bank
Optional but valuable: shared-expert count, quantization format, per-layer k if non-uniform. These four fields let any reader derive memory needs (total), compute needs (active), and verify arithmetic (k, N).
Example honest entry: "DeepSeek-V3 class: 671B total, 37B active per token, top-k routing, 256 routed experts." A reader can immediately map total → HBM, active → latency baseline.
Four-field template: total, active, k, N. Optional: shared expert count, quantization, per-layer k if non-uniform. Derive memory from total, latency baseline from active.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 model card reports total and active parameters separately.
- Mistral publishes Mixtral as ~47B total with ~13B active inference behavior.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should API pricing reflect MoE dual-number reporting?
Price on active FLOPs per token; disclose total memory requirements for self-hosted deployments separately.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Publishing only total params for marketing or only active params to hide memory requirements.
60 second bullets to scan on the way to the call.
Total params for memory provisioning
Active params for latency and cost
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.