Decode Mixtral 8x7B: what do the numbers mean?

Attempt it

Mixtral 8x7B means routed experts, each with a ~7B-class FFN, with experts active per token.

Quick reference

TL;DR

Mixtral 8×7B means 8 routed experts of ~7B-class FFN size with top-2 active per token, the name encodes bank layout, not total or active params.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of 8 specialist doctors on staff, each as capable as a '7B-class' expert, but each patient sees only the top 2 doctors chosen for their case. The name tells you how many specialists exist and how big each is, not how many treat every patient.

Key concepts

Mixture of Experts Transformer Architecture

Quick scan · 2–4 min

Everything important, quickly.

5 min: N×B decode + top-k completion + active vs total arithmetic + interview quoting.

Code example

python

def decode_mixtral_name(name: str, top_k: int = 2) -> dict:
    """Decode NxB MoE names like Mixtral 8x7B."""
    left, right = name.lower().replace('b', '').split('x')
    num_experts = int(left.strip())
    expert_class_b = float(right.strip())
    return {
        'num_routed_experts': num_experts,      # N in the bank
        'expert_ffn_class_b': expert_class_b,   # B-class FFN per expert
        'active_experts_per_token': top_k,      # k (top-2 for Mixtral)
        'note': 'Name encodes bank layout, not total or active params',
    }

print(decode_mixtral_name('8x7'))
# {'num_routed_experts': 8, 'expert_ffn_class_b': 7.0, 'active_experts_per_token': 2, ...}

Real-world examples

Real products, models, and research that use this idea.

Mistral AI's Mixtral 8×7B uses exactly 8 experts with top-2 routing per token.
Mixtral 8×22B extends the pattern: 8 experts of ~22B-class FFN, still top-2.
DeepSeek and Qwen MoE models use different N, B, and k, always read the spec sheet.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does Mixtral 8×22B change the arithmetic?

Sign in or go Pro to see the approach for this follow-up.

QWhy do vendors omit k from the model name?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Jiang et al., Mixtral of Experts