Complete the Switch Transformer auxiliary load balancing loss formula.
L_aux = α × N × Σ f_i × p_i, hard dispatch fraction f_i times differentiable softmax probability p_i per expert.
Think of the aux loss like a coach comparing game plans to the final scoreboard. f_i counts how many tokens really went to expert i (hard fact). p_i is what the router said it wanted (soft intention). Multiplying them penalizes experts the router favors but underuses, or uses heavily but didn't prefer, pushing intentions and reality to align.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The Switch Transformer auxiliary load-balancing loss formula, L_aux = α × N × Σ f_i × p_i, is the most tested piece of MoE math in senior interviews. The fill-blank format checks whether you know both symbols and understand why they multiply rather than add, and which one carries gradients.
The formula's elegance is in pairing a hard statistic (what actually happened during dispatch) with a soft statistic (what the router intended) per expert. Their product creates a penalty that pushes router preferences toward balanced dispatch without requiring differentiability through the discrete top-k selection.
This deep dive completes the formula, explains each component, traces gradient paths, and connects to the broader load-balancing story.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
Completing the formula: f_i and p_i
The Switch Transformer auxiliary loss is:
f_i,fraction of tokens dispatched to expert i in the current batch step. Computed from top-k routing decisions: count tokens assigned to expert i, divide by total tokens. This is a hard, discrete statistic, the argmax/top-k operation blocks gradients.
p_i,mean softmax routing probability for expert i, averaged across all tokens in the batch: p_i = (1/B) Σ_b soft_max(router_logits_b)_i. This is a smooth, differentiable function of router weights.
The blanks in order: f_i, then p_i. Their product per expert is summed over all N experts, scaled by α and N.
When implementing from scratch, verify gradients reach router weights by checking grad norm on W_router after backward, if zero, your f_i detach or aux loss wiring is broken.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch
import torch.nn.functional as F
def switch_aux_loss(router_logits, expert_indices, num_experts, alpha=0.01):
# router_logits: [batch, num_experts]
# expert_indices: [batch, top_k] hard dispatch from top-k
batch = router_logits.size(0)
p = F.softmax(router_logits, dim=-1).mean(dim=0) # p_i, differentiable
counts = torch.bincount(expert_indices.reshape(-1), minlength=num_experts).float()
f = counts / counts.sum() # f_i, detached hard dispatch fraction
aux = alpha * num_experts * (f.detach() * p).sum()
return aux # gradients flow through p only
Real products, models, and research that use this idea.
- Switch Transformer introduced this f_i · p_i formulation for load balancing at trillion-token scale.
- GShard and early Google MoE stacks use the same auxiliary loss structure.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy multiply f_i and p_i instead of penalizing (f_i - 1/N)² directly?
Dot product decorrelates hard and soft signals; direct variance penalty is an alternative used in some implementations.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Swapping f_i and p_i, or assuming both terms receive gradients through top-k.
60 second bullets to scan on the way to the call.
L_aux = α × N × Σ f_i × p_i
f_i = hard dispatch fraction (no grad)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.