MoE tail latency exceeds dense baselines because expert-parallel dispatch creates variable per-request paths and hot-expert queuing under uneven load.
A dense model is like a single-lane highway, every car takes the same route. MoE is like a city with many specialist shops, some trips need detours to different neighborhoods, and popular shops have longer lines. Most trips are fine, but the slowest detours set the worst-case wait.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MoE serving tail latency is one of the most underappreciated systems costs of sparse architectures. Median latency may match a dense model with equal active FLOPs, but p99 typically does not. The MCQ tests whether you understand the two structural sources of variance: expert-parallel dispatch paths and hot-expert queuing under routing skew.
Candidates who pick option C, that top-k eliminates cross-GPU communication, reveal they understand sparsity saves FLOPs but not that dispatch is still required when experts are sharded. This deep dive explains the correct answer, debunks distractors, and covers production SLO implications.
Understanding tail latency is what separates candidates who know MoE saves FLOPs from those who have operated MoE serving infrastructure. The median can look fine because most requests hit reasonably balanced expert paths, but p99 captures the slowest path through the expert-parallel graph, and that path is structurally more variable than dense uniform compute. This deep dive builds the two-source model (dispatch variance and hot-expert queuing) and connects it to production SLO monitoring.
Expert-parallel dispatch variance
In multi-GPU MoE serving, expert weights are sharded across devices. Each forward pass requires dispatching tokens to the GPUs holding their selected experts, running local expert FFNs, and collecting results.
Different requests activate different expert subsets on different shards. A request heavy on Expert 3 (GPU 2) has a different communication pattern than one heavy on Expert 7 (GPU 5). All to all or point to point collectives add latency that varies with the routing pattern of each batch.
Dense models on the same GPU pool follow an identical compute graph for every request. No dispatch, no shard-dependent paths. This structural uniformity is why dense p99/p50 ratios are typically tighter than MoE at equal active FLOPs.
Concrete dispatch picture. Imagine 8 experts sharded across 4 GPUs, 2 experts per GPU. Request A routes heavily to Experts 2 and 5 (GPUs 1 and 2). Request B routes to Experts 0 and 7 (GPUs 0 and 3). Same token count, same active FLOPs, but different all to all collective patterns and different shard queue depths. Dense models on the same GPU pool execute an identical compute graph for every request, no shard-dependent paths, no routing-dependent communication volume.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | Dense serving | MoE serving |
|---|---|---|
| Per-request compute path | Uniform every token | Varies by expert shard routing |
| Cross-GPU communication | Minimal at inference | Dispatch collectives each MoE layer |
| Tail latency driver | Shared queue on one graph | Hot expert shard queues |
| p99 vs median ratio | Typically tighter | Often wider under routing skew |
Real products, models, and research that use this idea.
- vLLM and TensorRT-LLM MoE serving docs highlight dispatch overhead and per-expert load balancing.
- Production MoE APIs at major labs report higher p99/p50 ratios than dense equivalents at similar active compute.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does routing collapse during training affect serving tail latency?
Training-time skew creates hot experts at serving time, same experts that dominated training receive disproportionate inference traffic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming MoE serving has uniform latency because active FLOPs match a dense model, dispatch variance and hot-expert queuing inflate p99.
60 second bullets to scan on the way to the call.
Expert-parallel dispatch variance
Hot-expert queuing under routing skew
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.