Pretraining and fine-tuning both run the same underlying next-token loss on a transformer. So what actually distinguishes them? Name at least two key differences, one about the data, one about the training setup, and briefly explain why those differences exist.
Free · 2 AI evals / day
**Pretraining** and **fine-tuning** are two stages of training the same kind of model, and they share more than you might think. The actual loss function is identical: next-token cross-entropy on a causal-LM forward pass.
What changes is everything around the loss. Pretraining feeds the model trillions of raw web tokens (CommonCrawl, books, code, papers) and runs for months on thousands of GPUs. The model starts with random weights and learns the statistics of language from scratch.
Fine-tuning starts from a pretrained model that already knows the language. It feeds the model a tiny, curated set, usually thousands or low millions of examples, often formatted as chat conversations or instruction-response pairs. Training runs for hours or a few days on far fewer GPUs.
The learning rate gap is large. Pretraining uses high learning rates around 1e-4 to 6e-4. Fine-tuning uses much lower rates, typically 1e-5 to 2e-5 for full fine-tuning, because the goal is to nudge an already-trained model gently. A high learning rate during fine-tuning is one of the fastest ways to cause catastrophic forgetting, where the model loses the knowledge it gained during pretraining.
**Pretraining and fine-tuning** share the next-token cross-entropy objective but differ on every other dimension that matters in practice.
**Data is the first major axis.** Pretraining ingests trillions of largely unlabelled, raw tokens scraped from the web, books, papers, and code. There is no notion of prompt versus response; the model just learns the statistics of language as continuous text. Fine-tuning consumes a tiny, curated set, typically anywhere from 1k to a few hundred k examples, of structured demonstrations: SFT chat pairs, preference triples, tool-calling traces. The model learns format and behaviour on top of knowledge it already has.
**Training setup is the second major axis.** Pretraining uses high learning rates around 1e-4 to 6e-4 and runs for months across thousands of GPUs because the model is being built from scratch. Fine-tuning uses much smaller rates, around 1e-5 to 2e-5 for full fine-tuning or 1e-4 for LoRA, and runs for hours to a few days on far less hardware. A high fine-tuning learning rate is the fastest way to trigger catastrophic forgetting, where the model overwrites the pretraining distribution and loses its general capability.
**A third worth mentioning is loss masking.** Pretraining computes loss on every token in the sequence. SFT typically masks the prompt and computes loss only on the response, because the model is being graded on what it should produce, not on what the user typed.
Quick reference
TL;DR
Same next-token loss, totally different data and training setup. Pretraining: trillions of raw tokens, high LR, months. Fine-tuning: thousands of curated demos, low LR, hours.
Picture a person learning a language by reading every book and webpage they can get their hands on for a decade. They end up fluent but a bit unfocused. That is pretraining. Now picture the same person spending a single weekend with a tutor practising how to politely answer customer-support emails. The tutor does not teach them new vocabulary or grammar; the tutor teaches them style, tone, and which formats to use. That weekend is fine-tuning. The same brain, the same language, just a tiny bit of shaping on top. Same skill underneath, fresh manners on top. The decade builds the knowledge; the weekend installs the behaviour.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You need the model to acquire genuinely new factual knowledge at scale, not just a new behaviour Fine-tuning is poor at knowledge injection; continued pretraining or retrieval-augmented generation is the right tool for new facts.
Your base model already handles the task well with prompting and few-shot examples Fine-tuning has setup and maintenance costs; if a prompt change solves the problem, ship that and skip the training run.
You only have a handful of examples (under a few hundred) and they cover a narrow distribution Tiny datasets risk overfitting and catastrophic forgetting without giving enough signal to install robust new behaviour.
Quick scan · 2–4 min
Everything important, quickly.
6 min: same loss but different worlds + data shape and volume gap + LR and duration gap + loss masking in SFT + catastrophic forgetting + when each is the right tool.
Side-by-side
Dimension
Pretraining
Fine-tuning
Data volume
Trillions of tokens
Thousands to low millions of examples
Data shape
Raw text, no labels
Structured demos, often chat-formatted
Learning rate
1e-4 to 6e-4
1e-5 to 2e-5 full, 1e-4 LoRA
Duration and hardware
Months on thousands of GPUs
Hours to days on tens of GPUs
Loss masking
Every token contributes
Prompt masked, response only
What changes
World knowledge built from scratch
Behaviour, format, style
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick was pretrained on roughly 30T tokens at peak LR 3e-4, then SFT-fine-tuned on a curated dataset around 1e-5.
2DeepSeek V4 publishes its pretraining LR schedule (warmup + cosine to 1e-4) and its post-training LR (around 2e-5), illustrating the standard order of magnitude gap.
3OpenAI fine-tuning endpoints default to a learning rate multiplier around 0.1 relative to the base model's pretraining LR, a heuristic that mirrors the same gap.
4The Tulu and Zephyr open-weight families showed that 1-10k high-quality SFT examples at low LR can install instruction-following on a strong base model.
5Hugging Face TRL's SFTTrainer applies prompt masking by default, computing loss only on assistant turns when given a chat-format dataset.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does a high learning rate cause catastrophic forgetting during fine-tuning?
Sign in or go Pro to see the approach for this follow-up.
QWhy does SFT mask the prompt tokens from the loss?
Sign in or go Pro to see the approach for this follow-up.
QHow would you decide between continued pretraining and SFT for a domain adaptation task?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming pretraining and fine-tuning use different loss functions. They run the same next-token cross-entropy; what changes is the data, the learning rate, the duration, and which tokens contribute to the loss.