What breaks when the auxiliary loss coefficient α is set too high versus too low?

α scales the auxiliary load-balancing loss relative to the language-modeling loss. It controls how hard the router is pushed toward even expert utilization. Too high: the router prioritizes balance over task fit, experts get equal traffic regardless of which specialist suits the token. Specialization weakens and validation quality can drop even though utilization dashboards look healthy. Too low: the balancing signal cannot counter routing collapse, a few experts dominate via positive feedback, idle experts waste capacity, and training destabilizes. Production runs still show 40–60% skew even with well-tuned α.

α too high forces near-uniform expert selection regardless of token-expert fit. The router optimizes for balance over quality. That causes gradient interference with the LM objective. Specialization weakens, and validation loss can rise, even when utilization metrics look even. When α is too low, routing collapse returns. A few experts attract most tokens through positive feedback, popular experts get more gradient updates and become more attractive. Idle experts waste capacity, and training destabilizes. Tuning α is a quality versus balance tradeoff. Switch Transformer uses α ≈ 0.01–0.1. Even well-tuned runs show substantial skew (40–60% gap between busiest and idlest expert). α is a regularizer, not a hard constraint.