Tokens per second can rise while MFU falls when easier workload shape or pipeline effects increase throughput but leave arithmetic units less saturated.
Think of a factory line: boxes per hour can rise while machine efficiency falls if workers are moving empty boxes. MFU is machine efficiency; tokens per second is boxes per hour. Both numbers matter, but they answer different questions.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When tokens/sec rises, why can MFU still fall? is an interview favorite because it reveals whether a candidate can connect theory, systems constraints, and product outcomes in one coherent explanation. Many answers fail by repeating a definition and skipping operational implications, but senior interview loops expect the opposite: show the mechanism, name the tradeoffs, and describe how you would monitor or validate the decision in a real training pipeline.
A useful structure is to move from first principles to field practice. Start with what the metric, pattern, or claim formally means. Then test where that framing breaks under realistic constraints such as fixed compute, skewed data mixtures, distributed training overhead, or deployment economics. This transition from textbook statement to operating playbook is exactly what separates a passable answer from a high-signal one.
A useful validation habit is to separate directional confidence from quantitative confidence. Directional confidence asks whether the mechanism is probably right. Quantitative confidence asks whether the expected gain is large enough to justify operational risk. Teams that skip this split often overreact to small metric movement. Teams that keep the split can move faster because they demand the right level of evidence for each decision.
Another senior-level move is to state what evidence would change your mind. If a counter-ablation disproves your assumption, say exactly which decision you would reverse and why. This turns the explanation from static theory into an adaptive engineering strategy, which is how real pretraining programs avoid expensive path dependency.
Mechanism-level framing
The mechanism behind this question is captured by one core idea: Tokens per second can rise while MFU falls when easier workload shape or pipeline effects increase throughput but leave arithmetic units less saturated. If you cannot restate that idea crisply, every downstream design choice becomes fuzzy. Interviewers are checking whether you understand which variable is causal versus which variable is merely correlated with better outcomes.
The strongest way to explain the mechanism is to name invariants and failure boundaries. Invariants are the assumptions that must stay true when scaling a run or changing infrastructure. Failure boundaries are the regimes where the same heuristic no longer applies cleanly. This gives your answer structure and prevents overconfident universal claims.
A useful validation habit is to separate directional confidence from quantitative confidence. Directional confidence asks whether the mechanism is probably right. Quantitative confidence asks whether the expected gain is large enough to justify operational risk. Teams that skip this split often overreact to small metric movement. Teams that keep the split can move faster because they demand the right level of evidence for each decision.
Another senior-level move is to state what evidence would change your mind. If a counter-ablation disproves your assumption, say exactly which decision you would reverse and why. This turns the explanation from static theory into an adaptive engineering strategy, which is how real pretraining programs avoid expensive path dependency.
Another senior-level move is to state what evidence would change your mind. If a counter-ablation disproves your assumption, say exactly which decision you would reverse and why. This turns the explanation from static theory into an adaptive engineering strategy, which is how real pretraining programs avoid expensive path dependency.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Metric | What it captures | Common pitfall |
|---|---|---|
| Tokens/sec | Pipeline output rate | Assuming hardware saturation improved |
| MFU | Useful FLOPs utilization | Treating it as model-quality score |
| Both together | Speed plus efficiency | Ignoring workload-shape changes |
Real products, models, and research that use this idea.
- Large GPU clusters often see higher tokens per second after shorter sequence settings, while MFU drops due to lower arithmetic intensity.
- Performance teams inspect both MFU and communication traces before claiming kernel-level efficiency improvements.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhich invariant would you monitor first after an infrastructure change?
Pick one measurable invariant and explain why it is the highest-leverage early warning signal.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Teams often celebrate throughput gains without checking whether they came from true compute utilization improvements or from easier token workload.
60 second bullets to scan on the way to the call.
Core invariant behind mfu versus throughput interpretation
Failure mode that looks healthy in logs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.