In L_aux = α·N·Σ f_i·p_i, which term receives gradients from the balancing loss?
Only p_i (mean softmax routing probability) receives gradients, f_i is a hard dispatch count blocked by top-k.
Imagine the router keeps two records: a wish list (soft probabilities p_i) and a receipt (hard counts f_i of what actually happened). Training can only adjust the wish list, the receipt is a done deal after top-k picks experts. So the balancing loss teaches the router through p_i, not f_i.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Which term in L_aux = α·N·Σ f_i·p_i receives gradients is a precise differentiability question with a clear answer: p_i only. The MCQ distractors target common confusions, straight-through estimators, full top-k differentiability, and the myth that aux loss is logging-only.
Understanding gradient paths in MoE routing separates candidates who memorized the formula from those who understand why Switch Transformer designed it this way. The hard/soft split is not arbitrary, it solves the fundamental problem that discrete expert selection cannot carry standard backpropagation gradients.
This deep dive confirms p_i as the gradient channel, explains why f_i is detached, and dismantles each distractor.
If you implement MoE from scratch, add a unit test that checks router.weight.grad is non-zero after backward on L_aux alone, that test catches wiring bugs where aux loss is logged but not connected to the computation graph.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
Correct answer: p_i only
p_i is the batch-averaged softmax routing probability for expert i:
This is a smooth, continuous function of router weight matrix W_g. When L_aux backpropagates, gradients flow through p_i into W_g, updating the router to adjust soft routing preferences.
The gradient is:
Note f_i appears as a coefficient (detached constant), not as a gradient recipient.
Expert FFN weights receive gradients from L_task through the hard routing mask, a separate path from the aux loss router gradient through p_i. Two gradient channels coexist in MoE training.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer backprops aux loss through p_i, the standard reference implementation.
- Open-source MoE trainers (Megablocks, fairseq MoE) follow the same f_i detached, p_i trained pattern.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy not use straight-through on f_i for aux loss?
Noisy and biased; Switch design uses p_i as a clean differentiable channel for balance training.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming f_i backpropagates through top-k via straight-through estimator.
60 second bullets to scan on the way to the call.
L_aux = αNΣf_ip_i
p_i = differentiable softmax average
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.