Pre-training
Also known as: Pretraining
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.
The initial self-supervised phase where a model learns from a massive unlabeled corpus, typically via next-token prediction. Produces a 'base model' that's a strong generic predictor but not yet aligned with human preferences.
In practice
The foundation everything else builds on. Senior interviews probe data mixture, curriculum, and the post-train (SFT + RLHF) stack.
How it compares
Pre-training is from-scratch on massive generic data; fine-tuning adapts a pre-trained model on a smaller curated set.
Related topics
Questions that mention this term
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
Fine-tuning
Continue training a pre-trained model on task-specific data to specialize its behavior.
FSDP (Fully Sharded Data Parallel)
Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.
GRPO (Group Relative Policy Optimization)
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.
Instruction Tuning
Train a base model on many (instruction, response) pairs to make it follow natural-language commands.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.