Your lab has a strong 7B model and three weeks of spare compute. A proposal lands on your desk: duplicate a contiguous span of middle layers to build a deeper ~10B model, then continue pretraining (the SOLAR 10.7B depth up scaling playbook). Assess why this can work at all, what you would watch for, and when you would reject it in favor of training the larger model from scratch.
Cloning middle layers is a benign init under pre-norm; continued pretraining differentiates the copies. Reject when you need width changes or have from-scratch budget.
Picture a relay team where each runner adds a little bit of distance to the team's total. If you clone one runner and put them right behind their original, the team's total distance changes only slightly at first because both copies run the same small leg. The team can keep racing while the copies practice running slightly different routes. Cloning a runner from the middle of the race is safer than cloning the first runner (who has a special start position) or the last runner (who has the finish line camera on them). Train the new team for a few weeks and they often beat building an entirely new team from scratch.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Depth up-scaling is the technique of taking a strong pretrained transformer, duplicating a contiguous span of its middle layers to grow the depth, then continuing pretraining to consolidate the result. SOLAR 10.7B famously built this way from Mistral 7B and beat same-size peers trained from scratch at a fraction of the cost. The technique is attractive when compute is scarce and a strong base model is available, and it rests on a specific structural property of modern transformers: pre-norm residual blocks add small, near-incremental writes to a shared stream, so cloning a block initially doubles a small contribution rather than catastrophically rewriting anything.
The decision to depth up rather than train from scratch hinges on three factors. The compute budget available; whether the upgrade you need is depth or width; and how sensitive your deployment is to per-token inference latency. This deep dive walks through why depth up works, why middle layers specifically are the safe choice, how to run the continuation correctly, and when to reject the move entirely.
The takeaway will be that depth up is not a free upgrade. It is a specific tradeoff: cheaper training, possibly slower inference, and a fixed-shape model that inherits both the strengths and the weaknesses of its base.
Why pre-norm makes layer duplication a benign initialization
The residual stream is a shared communication channel. Every block reads from the stream (via its input norm), computes a small update, and adds that update back. The update is small because the model has been trained to do useful work with limited per-block writes; if any block wrote a giant delta, the deeper stack would be unstable.
Cloning a block doubles a small contribution. Duplicate block B of a trained model and the two copies initially apply the same small write to the stream in sequence. The stream after the first copy looks slightly different from what B saw originally, so the second copy's read is mildly off-distribution, but only mildly. The model is in a function-space neighborhood of its trained behavior, and continued pretraining nudges the two copies onto different specializations within a small number of training steps.
Compare to random init. A randomly initialized deeper model has to learn every layer's function from scratch, including the basic tokenization and embedding-interaction features. The training-loss curve starts at the base of the random-init valley and climbs the long ascent that scaling laws describe. A depth-up model starts most of the way up that ascent. The continued pretraining run only has to refine the cloned layers and re-anchor the rest of the stack to the slightly different stream dynamics.
Empirical signature. Loss at depth-up initialization is bumped by a small amount (typically 0.1-0.3 nats) versus the base model. The bump closes within the first 5-10% of the continued pretraining budget. If the bump never closes, something else is wrong: wrong layer choice, wrong schedule, or wrong data mixture.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SOLAR 10.7B by Upstage AI was built by duplicating a middle span of Mistral 7B layers and continuing pretraining, outperforming same-parameter-count peers at lower cost
- Gemma 2 9B uses related model-surgery and distillation techniques to scale from smaller base models with continued training
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide which contiguous span of middle layers to duplicate?
Run per-layer attribution or activation-norm analysis on the base model. Choose a span where layer functions look reasonably homogeneous; avoid spans where adjacent layers show very different roles. Empirically, layers 12-20 of a 32-layer 7B model are common targets.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Cloning early or late layers without justification, treating the move as a free upgrade, and skipping the continued pretraining that actually makes the deeper model better than the original.
60 second bullets to scan on the way to the call.
Why pre-norm residual writes make duplication a benign initialization
Why middle layers are the safe choice and end layers are not
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.