In L_aux = α·N·Σ f_i·p_i, which term receives gradients from the balancing loss?

Attempt it

Quick reference

TL;DR

Only p_i (mean softmax routing probability) receives gradients, f_i is a hard dispatch count blocked by top-k.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the router keeps two records: a wish list (soft probabilities p_i) and a receipt (hard counts f_i of what actually happened). Training can only adjust the wish list, the receipt is a done deal after top-k picks experts. So the balancing loss teaches the router through p_i, not f_i.

Key concepts

Mixture of Experts Pretraining

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: formula + gradient paths + distractor analysis + straight-through contrast + z-loss mention.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer backprops aux loss through p_i, the standard reference implementation.
Open-source MoE trainers (Megablocks, fairseq MoE) follow the same f_i detached, p_i trained pattern.
Router z-loss in Gemma 2 MoE also trains router weights directly for logit stability.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy not use straight-through on f_i for aux loss?

Sign in or go Pro to see the approach for this follow-up.

QWhat is router z-loss and does it also train the router?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming f_i backpropagates through top-k via straight-through estimator.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.