You upcycle a dense checkpoint into an 8-expert MoE: every expert is an exact copy of the dense FFN, the router is freshly initialized with small random weights, and top-2 routing with softmax normalized gate weights is enabled. Before any training step, what does the MoE layer output compared to the original dense layer, and does the routing choice matter yet?
At step zero an upcycled MoE reproduces the dense layer exactly: identical experts plus gate weights that sum to one collapse to the plain FFN output, so the routing choice cannot matter yet.
Picture eight photocopies of the same recipe handed to eight cooks. An order comes in and a brand new dispatcher, who knows nothing yet, picks two cooks at random and splits the order between them. Does it matter which two were picked, or how the order was split? Not at all: every cook follows the identical recipe, so the dish comes out exactly as it did when the restaurant had one cook. The dispatcher's choices only start to matter weeks later, after each cook has scribbled their own notes in the margins and the copies are no longer identical. Until the recipes diverge, any combination of cooks produces the same meal, which is why day one at the new restaurant tastes exactly like the old one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like it is about routing, and the trap is to reason about the router at all. A freshly initialized router making random choices sounds like it should produce chaos, so candidates predict degraded output, noisy logits, or instability. The correct prediction is the opposite: the upcycled MoE outputs exactly what the dense parent outputs, and nothing the router does can change that.
The reason is a symmetry argument, and symmetry arguments are worth internalizing because they recur across ML systems: when all branches of a computation are identical, the selection mechanism among them is invisible. This deep dive works through the algebra, the condition that makes it hold, the engineering value of a function preserving initialization, and the training dynamics that gradually destroy the symmetry and make routing meaningful.
The one line proof
Fix a token with hidden state x. Upcycling made every expert an exact copy of the dense FFN, so expert_i(x) = FFN(x) for all eight values of i. The router scores the experts, picks two, and softmax normalizes their scores into gate weights g1 and g2 with g1 + g2 = 1. The MoE layer then outputs the gate weighted combination:
Every quantity the router controls has vanished from the result. Which two experts were chosen: gone, they all compute the same function. How the weights were split between them, 0.5 and 0.5 or 0.9 and 0.1: gone, only the sum matters, and the sum is one.
This is why the question says the routing choice does not matter yet. It is not that routing is approximately harmless or that errors average out. The routing decision is algebraically absent from the output. A reviewer can verify the claim without running anything: substitute identical experts into the MoE equation and watch the gates factor out.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The Google sparse upcycling work relies on exactly this function preserving property so the upcycled T5 and ViT models start at their dense parents' quality.
- Qwen1.5-MoE-A2.7B was upcycled from dense Qwen-1.8B and continued training from the dense quality floor instead of from random initialization.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the prediction change under Switch style top-1 routing that scales the output by the raw gate probability?
The single gate p is less than one after softmax over 8 experts, so the layer outputs p times FFN(x); think about what that rescaling does to the residual stream.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Predicting a degraded or noisy output because the router is random. Router randomness is harmless here: identical experts make every routing decision equivalent, provided the gate weights are normalized to sum to one.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.