How does dropless MoE dispatch differ from capacity-factor token dropping?

Attempt it

How does dropless MoE dispatch differ from capacity-factor token dropping during training?

Free · 2 AI evals / day

Quick reference

TL;DR

Capacity-factor MoE drops overflow tokens when expert buffers fill; dropless MoE redistributes or reschedules so every routed token gets FFN compute, trading scheduling complexity for zero skips.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Capacity-factor routing is like a restaurant that seats only 10 people per chef, when an 11th order arrives, it goes out without the special sauce (token skips FFN, keeps residual). Dropless MoE is the manager who rearranges orders, opens overflow prep stations, or batches smarter so nobody misses their dish, harder to run the kitchen, but no customer leaves without the full meal.

Key concepts

Mixture of Experts

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: capacity factor drops vs dropless dispatch, quality impact, and training systems tradeoffs.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer and GShard introduced capacity factors with token dropping as a standard training efficiency tool.
Dropless MoE research and production stacks (including Tutel-style dispatch optimizations) target zero-drop training at trillion-token scale.
Inference serving stacks typically avoid token drops, latency SLOs require every token to complete full expert compute.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would you accept capacity-factor drops in production training?

Sign in or go Pro to see the approach for this follow-up.

QHow do dropped tokens affect expert gradients?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming dropless MoE eliminates all routing imbalance, it eliminates token drops, not necessarily perfectly even expert load.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.