Walk through distilling an MoE teacher into a dense student for an edge deployment.

**Distillation from MoE to dense** transfers the teacher's behaviour into a single resident weight set. The student is typically sized around the MoE's active-parameter count because that matches the per-token compute the teacher uses. The training loop is standard knowledge distillation. You feed the student the same inputs as the teacher and minimise a combination of two losses: a soft-target loss that matches the teacher's output logit distribution (often with temperature), and a normal language-modelling loss on the ground-truth tokens. Some recipes also match intermediate hidden states to give the student more signal per example. The deployment win is real. One weight set sits on the device. There is no routing decision per token, no dispatch buffer, no all to all communication, and latency is predictable because the same parameters run for every input. The ceiling is also real. The teacher gets its quality by routing different tokens to different specialists across a much larger total parameter pool. A student with only active-size capacity cannot store all that specialisation at the same time. Expect strong everyday performance and regressions concentrated in tail domains.

**Distillation setup.** Pick a dense architecture roughly matching the MoE's active size. Train on a broad data mix using a combined loss: KL divergence between student and teacher logit distributions (typically with temperature in the 2 to 4 range), plus the standard cross-entropy LM loss on ground-truth tokens. Optional add-ons include hidden-state matching at chosen layers, attention pattern matching, and on-policy fine-tuning where the student generates and the teacher rates. A practical refinement is to weight the distillation corpus toward your product's actual traffic distribution. The teacher's specialist experts cover a wide skill range; the student has finite capacity to absorb that range, so spend its parameters on the slice your users actually hit. **The deployment win** is what makes the project worth doing. The student has one resident weight set, runs without expert-parallel infrastructure, has no routing scratch buffers, no all to all dispatch, and latency is predictable per token. These properties are non-negotiable for on-device deployment where memory and compute are tight and tail latency cannot spike. **The ceiling stakeholders need to hear.** The MoE's quality comes from conditional computation over a total parameter pool that may be five to twenty times the active size. Different tokens use different specialists. A dense student with only active-size capacity cannot store all that specialised knowledge simultaneously, so it averages it. Average performance can look close on aggregate benchmarks while tail-domain capability quietly degrades. **Evaluation.** Do not trust aggregate scores. Build per-domain eval that mirrors your product's actual distribution and tail use cases. If aggregate looks fine but the rare-skill bucket drops, that is the distillation ceiling showing.