Synthetic Data
Also known as: Synthetic training data
Use a stronger LLM to generate training data: bootstrap fine-tuning, fill data gaps, distill knowledge.
Training data generated by an existing LLM rather than collected from humans. Used to bootstrap fine-tuning sets, augment scarce domains, distill capabilities into smaller models, or seed RLHF preference pairs.
In practice
Underpins most modern open-model post-training pipelines (Phi, Tulu, distilled SFT). Interviews probe quality filtering and contamination.
Related topics
Questions that mention this term
- Where does quality leak in a self-instruct synthetic data pipeline?
- How do teams contain synthetic data collapse risk in pretraining?
- What is the root…
- Why does using the same model family as synthetic data teacher AND eval judge inflate FT scores?
- Which quality controls are essential when using an LLM to generate eval test cases?
- Define the pretraining data wall and its consequence.
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
Fine-tuning
Continue training a pre-trained model on task-specific data to specialize its behavior.
FSDP (Fully Sharded Data Parallel)
Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.
GRPO (Group Relative Policy Optimization)
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.
Instruction Tuning
Train a base model on many (instruction, response) pairs to make it follow natural-language commands.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.