How does DeepSeek-V3 balance expert load without an auxiliary loss term?

Most MoE models from the Switch/Mixtral lineage add an auxiliary load-balancing loss to the training objective. The router learns to spread tokens across experts because a separate penalty term pushes against collapse. **DeepSeek-V3** takes a different route: **auxiliary loss free balancing**. Before top-k expert selection, the model adds a per-expert bias scalar to each router logit. A side loop tracks how many tokens each expert has processed recently (moving-average load). If an expert is overloaded, its bias is decremented so future tokens are less likely to pick it. Underloaded experts get nudged upward. Crucially, this bias update happens outside the main backpropagation path for the language-modeling loss. No load-balancing term is summed into `L_total`. Task gradients still train the router through soft gating weights on selected experts, but they are not competing with a uniform-routing objective weighted by α.

DeepSeek-V3 balances experts without adding `L_aux` to the training loss. The mechanism: **per-expert bias terms** on router logits before softmax and top-k. Each expert maintains a moving-average of recent token load. An external adjustment loop (outside standard backprop) shifts biases, overloaded experts down, underloaded experts up, steering dispatch toward balance. Router weights still learn from the task loss via differentiable soft weights on selected expert outputs. The hard top-k mask remains non-differentiable; bias steering handles load, not gradient flow through the discrete choice. The win over Switch-style aux loss: no α hyperparameter where too-high values force near-uniform routing and interfere with task-loss gradients. The tradeoff: bias loops have their own tuning (EMA window, update rate) and do not guarantee perfect uniformity, monitoring per-expert token fraction and utilization CV remains mandatory.