A researcher on your team claims their new FFN variant beats SwiGLU, based on one 125M-parameter run where their variant got lower loss. You are asked to design the ablation study that would actually justify adopting it in the next flagship pretrain. Lay out the methodology: what must be controlled, what must be swept, and which results would convince you.
Match params and FLOPs, tune both recipes equally, run a multi-scale sweep with multiple seeds, and only ship if the gap survives extrapolation, downstream evals, and systems cost.
Imagine two chefs claim their bread recipe is better, but only one of them baked a single loaf. Before betting the bakery on the new recipe, you make them both bake the same loaf, with the same flour, same oven, same time, and the same chance to fine tune their salt. Then you have them bake five sizes of loaf, several times each, and you taste every one cold and warm. Only if the new recipe wins at every size, every taste test, and bakes just as fast, do you switch the whole bakery over. One lucky loaf is not enough to retool a kitchen that runs three meals a day.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An ablation table is one of the most misleading artifacts in machine learning research. A new component beats the baseline by 0.3 nats at 125M parameters and the paper claims a win. Six months later, a flagship team tries to reproduce at 70B and finds the gap is gone, or worse, has flipped sign. The cost of that mistake is millions of dollars in wasted compute and weeks of slipped schedule.
This deep dive walks through the methodology that prevents the trap. It covers iso-FLOPs controls, equal hyperparameter tuning effort, seed noise estimation, multi-scale scaling sweeps, evaluation breadth beyond training loss, and the systems cost line item that a loss-obsessed researcher will forget. The aim is a single, defensible decision: ship this change to the flagship, or do not.
Iso-FLOPs is the real control, not iso-params
The naive control is to match parameter count. The disciplined control is to match parameter count AND training FLOPs AND token budget. The difference matters.
Consider a new FFN variant with three matrices instead of two. If you match parameter count by shrinking the hidden width, you also change the FLOPs per forward pass per token. At constant tokens, the variant trains for fewer FLOPs, and a loss win at iso-params may simply be a loss loss when re-plotted at iso-FLOPs. SwiGLU's 2/3 hidden rule was designed precisely to preserve both at once, by setting the hidden width so that 3 x d x (2/3) x 4d = 8d^2, matching the original GELU FFN's compute budget.
The right framing is: at a fixed compute budget C and a fixed data budget D, which architecture lands at lower validation loss? Everything else, including the parameter count at any specific size, follows from that.
Fit this curve, not point estimates.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The SwiGLU paper (Shazeer 2020) used iso-parameter comparisons with the 2/3 hidden width trick against ReLU FFNs, and the recipe still ships in Llama 3, Mistral, and DeepSeek-V3
- Chinchilla (Hoffmann et al. 2022) showed that single-size compute claims about Gopher were wrong once a scaling sweep was fit, reshaping how the industry sizes pretraining runs
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between a slightly better loss curve and a 10% throughput regression?
Translate both into constant-compute terms: at fixed FLOPs, the slower variant gets fewer tokens. Re-plot loss vs compute, not loss vs tokens, and pick the curve that wins at the flagship FLOPs budget.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating a single 125M run as evidence, comparing the tuned variant against an undertuned baseline, and skipping the scaling sweep that reveals whether the gap survives at flagship size.
60 second bullets to scan on the way to the call.
Why iso-FLOPs matters and how it differs from iso-params alone
How to allocate equal hyperparameter tuning budget to both recipes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.