Match each MoE load-balancing mechanism to its defining property.

Zenaique

Attempt it

Switch Transformer auxiliary loss

DeepSeek-V3 dynamic bias

High α on auxiliary loss

Auxiliary-loss-free routing

f_i in the aux-loss formula

Quick reference

TL;DR

MoE load balancing splits into loss-based (Switch aux loss with f_i·p_i) and loss-free (DeepSeek dynamic bias on logits); high α on aux loss can distort task gradients.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine assigning customers to checkout lanes. Switch-style aux loss is a manager who docks the store's sales bonus when one lane gets too popular, the penalty is baked into the main scorecard everyone optimizes. DeepSeek's bias approach is a sign that nudges customers toward shorter lines without changing the sales bonus formula. The f_i term counts who actually went to each lane (a hard count you can't differentiate through), while p_i is how likely the router thought each lane would be picked.

Key concepts

Mixture of Experts

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

7 min: aux loss mechanics, DeepSeek bias balancing, differentiability, and α tradeoffs.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer and early GShard stacks train with auxiliary load-balancing loss and capacity factors for overflow handling.
DeepSeek-V3 reports aux loss free routing via dynamic per-expert bias adjustment on router logits.
Mixtral training inherits Switch-style aux loss conventions from the sparse MoE lineage.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy multiply f_i and p_i instead of penalizing f_i alone?

Sign in or go Pro to see the approach for this follow-up.

QHow would you debug collapse if bias balancing is already enabled?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating f_i and p_i as interchangeable, or assuming aux loss free balancing means experts are perfectly uniform.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.