Decode Mixtral 8x7B: what do the numbers mean?
Mixtral 8×7B means 8 routed experts of ~7B-class FFN size with top-2 active per token, the name encodes bank layout, not total or active params.
Think of 8 specialist doctors on staff, each as capable as a '7B-class' expert, but each patient sees only the top 2 doctors chosen for their case. The name tells you how many specialists exist and how big each is, not how many treat every patient.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mixtral 8×7B is the most cited MoE naming example in LLM interviews. The fill-blank format tests whether you can decode N (expert count), B (per-expert FFN class), and k (active per token) from a compact label. Getting any one wrong cascades into wrong memory estimates, wrong latency comparisons, and wrong cost models.
The name is an architecture shorthand, not a parameter count. It tells you how the expert bank is structured. You must supply the routing configuration from domain knowledge, Mixtral uses top-2, to complete the picture.
This deep dive decodes the convention, walks through the arithmetic, and shows how to generalize to other MoE releases.
Fill-blank on Mixtral 8×7B tests whether N, B, and k live in your head as separate variables. The name gives N=8 and B≈7B-class; you supply k=2 from domain knowledge. Blanks alone are insufficient in follow-ups, interviewers immediately ask active vs total arithmetic.
Treat fill-blank as the first 10 seconds of a sizing conversation, blanks unlock arithmetic, not replace it.
Write 8 and 2 on paper, then immediately write ~47B total and ~13B active before the interviewer asks, preempt the follow-up.
Decoding N×B: what each number means
8 = the number of routed expert FFNs in the MoE layer bank. Each token's router scores all 8 and selects the top-k.
7B = the FFN capacity class per expert, roughly the parameter count of each expert's feed-forward block if it were a standalone dense FFN of that width. It is not the total model size and not the active size.
The fill-blank template asks for both N (8) and k (2). Mixtral's published architecture uses top-2 routing: two of eight experts activate per token, with outputs weighted by router softmax scores.
N is expert bank size, not active count. B is per-expert FFN class, not total model size. k is routing config, often omitted from marketing names, treat k as mandatory mental fill-in.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def decode_mixtral_name(name: str, top_k: int = 2) -> dict:
"""Decode NxB MoE names like Mixtral 8x7B."""
left, right = name.lower().replace('b', '').split('x')
num_experts = int(left.strip())
expert_class_b = float(right.strip())
return {
'num_routed_experts': num_experts, # N in the bank
'expert_ffn_class_b': expert_class_b, # B-class FFN per expert
'active_experts_per_token': top_k, # k (top-2 for Mixtral)
'note': 'Name encodes bank layout, not total or active params',
}
print(decode_mixtral_name('8x7'))
# {'num_routed_experts': 8, 'expert_ffn_class_b': 7.0, 'active_experts_per_token': 2, ...}Real products, models, and research that use this idea.
- Mistral AI's Mixtral 8×7B uses exactly 8 experts with top-2 routing per token.
- Mixtral 8×22B extends the pattern: 8 experts of ~22B-class FFN, still top-2.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does Mixtral 8×22B change the arithmetic?
Still 8 experts, top-2, but each expert FFN is ~22B-class; active and total params scale with B, not N.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Multiplying 8 × 7B to get 56B active parameters, only top-2 experts run per token.
60 second bullets to scan on the way to the call.
N×B naming convention
N = expert count in bank
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.