Complete the Switch Transformer auxiliary load-balancing loss formula.

Attempt it

Switch Transformer auxiliary load balancing loss: L_aux = α × N × Σ_i × . Here f_i is the fraction of tokens dispatched to expert i (non-differentiable through top-k), and p_i is the mean softmax routing probability for expert i (differentiable).

Quick reference

TL;DR

L_aux = α × N × Σ f_i × p_i, hard dispatch fraction f_i times differentiable softmax probability p_i per expert.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of the aux loss like a coach comparing game plans to the final scoreboard. f_i counts how many tokens really went to expert i (hard fact). p_i is what the router said it wanted (soft intention). Multiplying them penalizes experts the router favors but underuses, or uses heavily but didn't prefer, pushing intentions and reality to align.

Key concepts

Mixture of Experts Pretraining

Quick scan · 2–4 min

Everything important, quickly.

7 min: formula components + grad paths + design rationale + α role + alternatives.

Code example

python

import torch
import torch.nn.functional as F

def switch_aux_loss(router_logits, expert_indices, num_experts, alpha=0.01):
    # router_logits: [batch, num_experts]
    # expert_indices: [batch, top_k] hard dispatch from top-k
    batch = router_logits.size(0)
    p = F.softmax(router_logits, dim=-1).mean(dim=0)  # p_i, differentiable
    counts = torch.bincount(expert_indices.reshape(-1), minlength=num_experts).float()
    f = counts / counts.sum()  # f_i, detached hard dispatch fraction
    aux = alpha * num_experts * (f.detach() * p).sum()
    return aux  # gradients flow through p only

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer introduced this f_i · p_i formulation for load balancing at trillion-token scale.
GShard and early Google MoE stacks use the same auxiliary loss structure.
DeepSeek-V3 replaced this with bias adjustment, but Switch formula remains the interview standard.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy multiply f_i and p_i instead of penalizing (f_i - 1/N)² directly?

Sign in or go Pro to see the approach for this follow-up.

QHow does DeepSeek-V3 balance without this formula?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.