Token-choice routing means each token picks its top-k experts from its hidden state, compatible with causal one token at a time decoding.
Token-choice is like each customer picking their favorite chef. Expert-choice is like each chef picking which customers to serve. During streaming text generation, you only know the current customer, so customers picking chefs works naturally.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token-choice versus expert-choice is one of the highest-value MoE distinctions for inference engineers. Get it wrong and you misunderstand why chat LLMs route the way they do, and why expert-choice papers rarely describe production decoding stacks.
The flashcard pairs definition with motivation: who selects whom, and why autoregressive decoding cares.
This deep dive makes the visibility argument concrete, what information each routing direction needs at each decoding step.
Token-choice versus expert-choice is the routing direction question that inference engineers must get right. The flashcard pairs a one-sentence definition with the autoregressive motivation. This deep dive makes the visibility argument concrete: what information each routing direction needs at each decoding step, and why production causal LLMs overwhelmingly choose token-choice.
Expert-choice has genuine advantages in training, better load balance, expert-centric optimization, but those advantages rarely transfer to autoregressive serving. Understanding both sides makes your token-choice answer stronger, not just "because production does it."
The visibility argument is the key insight interviewers want: token-choice routing decisions are local functions of h_t, making them composable with the standard autoregressive decode loop that produces one hidden state per step.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
Token-choice: the token picks experts
Mechanism. For hidden state h_t at position t, compute router logits z_t = W h_t ∈ R^E. Select top-k experts using z_t. Dispatch h_t to those expert FFNs. Combine weighted outputs.
No other tokens' hidden states are required for this token's routing decision. Different tokens in the same batch can route to different experts independently.
This is the default in Mixtral, Switch, DeepSeek, and essentially all production autoregressive LLM MoE.
Serving implementation. In vLLM, TGI, and TensorRT-LLM MoE paths, token-choice routing fits naturally into the per-token decode loop. After attention produces h_t, the router scores experts, dispatches to expert GPUs, collects results, and continues. No batch-wide coordination or future token visibility required. This is why essentially all production autoregressive LLM MoE uses token-choice.
The router logits z_t = W h_t depend only on information already in h_t, which encodes context from prior attention layers over tokens 1..t. No peeking at future positions, no waiting for batch-mates.
Batch inference with multiple concurrent sequences still uses token-choice: each sequence index routes independently within the batched tensor. No cross-sequence routing coordination is required.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral, DeepSeek-V3, and Llama 4 MoE inference stacks use token-choice routing for autoregressive serving.
- Switch Transformer training uses token-choice with auxiliary load-balancing loss.
What an interviewer would ask next. Try answering before peeking at the approach.
QCan expert-choice work at inference with batch size 1?
Discuss degenerate case, one token, E experts, and why token-choice is simpler.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing token-choice with expert-choice, where experts select tokens from a batch, that breaks autoregressive streaming.
60 second bullets to scan on the way to the call.
Token selects top-k experts
Router input is current hidden state only
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.