Explain the positive feedback loop that causes routing collapse during MoE training. · Zenaique

Attempt it

Explain the positive feedback loop that causes routing collapse during MoE training.

Free · 2 AI evals / day

Quick reference

TL;DR

Routing collapse is a positive feedback loop: popular experts get more gradients, improve faster, win more routing share, while idle experts stagnate, without load-balancing pressure, a few experts dominate.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a tutoring center where students pick their favorite tutor. The most popular tutor gets more practice, gets better, and attracts even more students. Quiet tutors rarely teach, never improve, and lose even the few students they had. Soon one tutor handles everything and the rest sit idle.

Key concepts

Mixture of Experts Fine-Tuning

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

7 min: feedback loop stages + dynamical systems view + countermeasures + monitoring metrics.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer introduced auxiliary load-balancing loss specifically to combat routing collapse.
DeepSeek-V3 uses dynamic bias adjustment on router logits as an aux loss free alternative.
MoE training dashboards at major labs monitor per-expert token fraction and coefficient of variation (CV).

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does top-1 routing collapse faster than top-2?

Sign in or go Pro to see the approach for this follow-up.

QHow would you detect collapse early in a training run?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking routing collapse is a one-time bad initialization, it is a self-reinforcing loop that worsens over thousands of training steps.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.