Given 4 experts of 2B params each, top-1 routing, and 5B always-on shared layers, how many active parameters does one token use?

Attempt it

A MoE transformer has shared attention, embedding, and output layers totaling 5B parameters, these run on every token. One MoE FFN layer has N=4 experts, each expert FFN is 2B parameters. Top-k routing with k=1 selects one expert per token. Router weights are negligible. How many billion parameters are actively used per token (round to a whole number)?

Quick reference

TL;DR

Active params per token = always-on shared layers (5B) + k active experts (1 × 2B) = 7B; total stored would be 13B.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a building with a front desk and four specialist offices. Every visitor checks in at the front desk (shared layers, 5B). Then a receptionist sends each visitor to exactly one specialist office (k=1, one 2B expert). The visitor's journey uses the desk plus one office, 7B total, even though four offices exist in the building (13B if you counted every office).

Key concepts

Mixture of Experts Inference Optimization

Quick scan · 2–4 min

Everything important, quickly.

6 min: active vs total formula + k/N intuition + multi-layer extension + interview traps.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer uses top-1 routing, active FFN compute is one expert per token per MoE layer.
Capacity planning for MoE inference separates GPU memory (total) from throughput (active FLOPs).
Model cards for DeepSeek and Mixtral publish both total and active parameter counts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does active count change if the model has 32 MoE layers with the same config?

Sign in or go Pro to see the approach for this follow-up.

QWhy is total param count still 13B if only 7B activate?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.