Explain why your MoE server's cost per token falls sharply as batch size grows.
Your latency sensitive MoE deployment runs near batch size 1 and looks shockingly expensive per token compared to a dense model with the same active parameters. Load testing shows cost per token dropping much faster with batch size than the dense baseline's does. Explain the mechanism.
Decode is memory bound: at batch 1 each step reads full expert weights to serve one token, so MoE pays for huge weights with tiny output. Batching spreads those reads over many tokens.
Imagine you rent a giant food court with dozens of specialist stalls because together they cook better meals than any single kitchen. At 3 a.m. one customer walks in. To serve their order, two stalls must fire up their whole setup, and you are paying rent on every stall in the building. That single meal is absurdly expensive. At lunch rush, the same rent and the same stall setups serve a thousand customers, so the cost folded into each meal collapses. An MoE server is the food court: the experts are stalls that must all stay rented (loaded in memory), and each request only uses a couple of them. The economics only work when a steady crowd keeps every stall busy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A team ships an MoE because the benchmark math looks great: dense-model quality at a fraction of the per-token compute. Then the first invoice arrives from a latency-sensitive pilot running at batch size 1, and the cost per token is worse than a dense model of similar quality. Load testing deepens the mystery: the MoE's cost curve plunges as batch grows, far faster than the dense baseline's.
Nothing is broken. The pilot simply measured the one regime where sparse models look worst. Understanding why requires holding two ideas at once: decode at small batch is a memory bandwidth problem rather than a compute problem, and an MoE deliberately carries an enormous weight footprint relative to the compute each token uses.
This deep dive builds the argument from the hardware up: what a decode step actually waits on, why batch 1 punishes sparse architectures specifically, how amortization rescues the economics, and what the serving playbook looks like once you accept that MoE is a throughput machine.
What a decode step actually waits on
Generating one token requires a full forward pass, and a forward pass requires every weight matrix involved to travel from HBM into the GPU's compute units. The useful mental model is the roofline: a kernel is either limited by arithmetic throughput or by memory bandwidth, and which one depends on arithmetic intensity, the ratio of FLOPs performed to bytes moved.
A matrix multiply that processes a batch of B token vectors against a weight matrix performs B times more useful work per weight byte than the same multiply at B = 1. At batch 1, intensity is roughly 2 FLOPs per parameter read, which is far below the hundreds of FLOPs per byte a modern accelerator needs to stay busy. The GPU spends the step waiting on memory.
This is why decode cost at small batch tracks bytes read per step, not FLOPs. The 'active parameters' number that makes MoE attractive describes FLOPs. It says almost nothing about the batch-1 bill.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 serving uses large-scale expert parallelism with big decode batches, exactly because per-token economics of a 671B-total model depend on amortizing weight reads.
- Mixtral 8x7B deployments on vLLM lean on continuous batching to keep decode batches full; at trickle traffic the 47B resident footprint dominates cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what point does growing the batch stop helping?
Find the crossover where step time becomes compute bound; past the GPU's balance point, bigger batches add latency instead of saving cost.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reasoning from FLOPs alone: decode at small batch is bound by reading weights from HBM, so active parameter count says little about cost per token at batch 1.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.