Decode Mixtral 8x7B: what do the numbers mean?

Fill in blank·Easy·4.0 · 0·~1 min·Asked atMistral AI

Attempt it

Mixtral 8x7B means routed experts, each with a ~7B-class FFN, with experts active per token.

Quick reference

TL;DR

Mixtral 8×7B means 8 routed experts of ~7B-class FFN size with top-2 active per token, the name encodes bank layout, not total or active params.

Memory aid

Easy to grasp

Think of 8 specialist doctors on staff, each as capable as a '7B-class' expert, but each patient sees only the top 2 doctors chosen for their case. The name tells you how many specialists exist and how big each is, not how many treat every patient.

Key concepts

Mixture of Experts Transformer Architecture

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: N×B decode + top-k completion + active vs total arithmetic + interview quoting.

Code example

python

def decode_mixtral_name(name: str, top_k: int = 2) -> dict:
    """Decode NxB MoE names like Mixtral 8x7B."""
    left, right = name.lower().replace('b', '').split('x')
    num_experts = int(left.strip())
    expert_class_b = float(right.strip())
    return {
        'num_routed_experts': num_experts,      # N in the bank
        'expert_ffn_class_b': expert_class_b,   # B-class FFN per expert
        'active_experts_per_token': top_k,      # k (top-2 for Mixtral)
        'note': 'Name encodes bank layout, not total or active params',
    }

print(decode_mixtral_name('8x7'))
# {'num_routed_experts': 8, 'expert_ffn_class_b': 7.0, 'active_experts_per_token': 2, ...}

Real-world examples

Real products, models, and research that use this idea.

Mistral AI's Mixtral 8×7B uses exactly 8 experts with top-2 routing per token.
Mixtral 8×22B extends the pattern: 8 experts of ~22B-class FFN, still top-2.
DeepSeek and Qwen MoE models use different N, B, and k, always read the spec sheet.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does Mixtral 8×22B change the arithmetic?

QWhy do vendors omit k from the model name?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Multiplying 8 × 7B to get 56B active parameters, only top-2 experts run per token.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.

Jiang et al., Mixtral of Experts