Drag each answer to line up with its matching prompt
Broad factual recall
Mostly adjusted by post-pretraining alignment steps.
Instruction following format
Sharpened during supervised instruction tuning.
Latent pattern completion skill
Largely emerges during large scale base pretraining.
Refusal and policy style
Usually improved by instruction response supervision.
Helpfulness tone consistency
Built mostly through next token training over diverse corpora.
Base pretraining mostly builds broad knowledge and latent pattern skills, while instruction tuning mostly shapes response format, tone, and policy behavior.
Think of building a chef. Pretraining is years of tasting ingredients and learning cooking patterns. Instruction tuning is later coaching on how to plate dishes for specific customers and house rules. The chef's core knowledge comes first; serving style and policy behavior get sharpened later. Mixing those stages causes confusion in interviews.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Interviewers ask about mapping capabilities to pretraining versus instruction-tuning stages because this decision controls run quality, cost, and failure risk in real pretraining programs. A surface-level answer often repeats one slogan, but the actual decision lives in how assumptions, metrics, and constraints interact over time. In modern large-model development, teams cannot afford that gap. One planning mistake can burn weeks of cluster time and still leave weaker checkpoints.
This deep dive is structured as a practical walkthrough. First we build the mechanism and objective framing. Next we show where the popular shortcut breaks. Then we connect that to run-time telemetry, decision gates, and failure diagnostics. We close with deployment-facing consequences and a concrete numerical scenario. The goal is not trivia recall. The goal is to explain the concept in a way that sounds like someone who has operated a real training program and can justify tradeoffs under pressure.
Build the mechanism before the slogan
Mechanism first. Start with the core statement: Different data and objectives produce different capability shifts across the lifecycle. In practice, this means the question is never isolated from budget and objective context. A ratio, optimizer, masking rule, or parallelism choice only makes sense once you specify what is fixed and what can move. Teams that skip this framing often end up comparing unlike runs and then drawing false conclusions from noisy curves.
The right way to reason is to separate invariants from knobs. Invariants include hardware budget, objective type, and safety constraints. Knobs include model size, token budget, batch, sequence length, optimizer settings, and parallelism strategy. Once those are explicit, you can reason in cause and effect form rather than slogan form.
A good interview answer names this structure out loud: what is fixed, what is being changed, and what metric you optimize. That alone signals maturity because it prevents category errors.
A compact expression often used in this context is:
You do not need to derive every constant during an interview. You do need to explain what the expression means operationally and what assumptions make it useful.
\text{Base quality} + \text{Post-training behavior shaping}Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Open model releases often show base checkpoints with strong completion ability but weaker instruction adherence before post-training.
- Chat assistant stacks commonly report instruction-following lift after supervised and preference-based post-training phases.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you prove a behavior change came from SFT and not base scale?
Use checkpoint lineage with controlled comparisons and stage-specific eval suites.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Candidates often assign instruction-following improvements to pretraining when they mostly come from supervised post-training.
60 second bullets to scan on the way to the call.
Base capability versus behavior shaping
Objective boundary between stages
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.