Pretraining vs fine-tuning: two key differences in data and objective.

Short answer·Easy·4.0 · 0·~3 min·Asked atAutodesk Contextual Ai Moveworks·Relevant atDatabricks Fireworks Ai Together Ai

Attempt it

Pretraining and fine-tuning both run the same underlying next token loss on a transformer. So what actually distinguishes them? Name at least two key differences, one about the data, one about the training setup, and briefly explain why those differences exist.

Free · 2 AI evals / day

**Pretraining** and **fine-tuning** are two stages of training the same kind of model, and they share more than you might think. The actual loss function is identical: next-token cross-entropy on a causal-LM forward pass. What changes is everything around the loss. Pretraining feeds the model trillions of raw web tokens (CommonCrawl, books, code, papers) and runs for months on thousands of GPUs. The model starts with random weights and learns the statistics of language from scratch. Fine-tuning starts from a pretrained model that already knows the language. It feeds the model a tiny, curated set, usually thousands or low millions of examples, often formatted as chat conversations or instruction-response pairs. Training runs for hours or a few days on far fewer GPUs. The learning rate gap is large. Pretraining uses high learning rates around 1e-4 to 6e-4. Fine-tuning uses much lower rates, typically 1e-5 to 2e-5 for full fine-tuning, because the goal is to nudge an already-trained model gently. A high learning rate during fine-tuning is one of the fastest ways to cause catastrophic forgetting, where the model loses the knowledge it gained during pretraining.

**Pretraining and fine-tuning** share the next-token cross-entropy objective but differ on every other dimension that matters in practice. **Data is the first major axis.** Pretraining ingests trillions of largely unlabelled, raw tokens scraped from the web, books, papers, and code. There is no notion of prompt versus response; the model just learns the statistics of language as continuous text. Fine-tuning consumes a tiny, curated set, typically anywhere from 1k to a few hundred k examples, of structured demonstrations: SFT chat pairs, preference triples, tool-calling traces. The model learns format and behaviour on top of knowledge it already has. **Training setup is the second major axis.** Pretraining uses high learning rates around 1e-4 to 6e-4 and runs for months across thousands of GPUs because the model is being built from scratch. Fine-tuning uses much smaller rates, around 1e-5 to 2e-5 for full fine-tuning or 1e-4 for LoRA, and runs for hours to a few days on far less hardware. A high fine-tuning learning rate is the fastest way to trigger catastrophic forgetting, where the model overwrites the pretraining distribution and loses its general capability. **A third worth mentioning is loss masking.** Pretraining computes loss on every token in the sequence. SFT typically masks the prompt and computes loss only on the response, because the model is being graded on what it should produce, not on what the user typed.

TL;DR

Same next-token loss, totally different data and training setup. Pretraining: trillions of raw tokens, high LR, months. Fine-tuning: thousands of curated demos, low LR, hours.

Memory aid

Easy to grasp

Picture a person learning a language by reading every book and webpage they can get their hands on for a decade. They end up fluent but a bit unfocused. That is pretraining. Now picture the same person spending a single weekend with a tutor practising how to politely answer customer-support emails. The tutor does not teach them new vocabulary or grammar; the tutor teaches them style, tone, and which formats to use. That weekend is fine-tuning. The same brain, the same language, just a tiny bit of shaping on top. Same skill underneath, fresh manners on top. The decade builds the knowledge; the weekend installs the behaviour.

Key concepts

Pretraining Supervised fine-tuning Catastrophic forgetting

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Pretraining and fine-tuning are easy to confuse because they look identical on the inside. Same model, same forward pass, same cross-entropy loss. Beginners often assume they must therefore be roughly the same operation at different scales. They are not. The two stages serve different goals, run on different data, use different learning rates, and depend on different infrastructure. Treating them as interchangeable is the most expensive misconception in modern post-training.

This deep dive separates the two stages cleanly. We start with the one thing they do share, the next-token cross-entropy loss, and then walk through every dimension where they diverge: data volume and shape, learning rate, training duration, hardware footprint, and loss masking. Each divergence is explained in terms of what the stage is trying to accomplish.

The headline is straightforward. Pretraining builds the model's world from trillions of unlabelled tokens with high learning rates over months of compute on thousands of GPUs. Fine-tuning shapes the already-built model with a few thousand to a few million curated demonstrations at much lower learning rates, often finishing in hours on a single node. Pretraining installs knowledge; fine-tuning installs behaviour. The cost ratio is roughly 100x to 1000x in dollars, and that cost gap reflects a fundamental difference in what each stage achieves.

Get this distinction right and the rest of the post-training stack (SFT, DPO, RLHF, continued pretraining, retrieval augmentation) falls into a clear hierarchy. Get it wrong and you will overfit a fine-tune, torch a base model with too-high a learning rate, or try to use SFT to teach knowledge that needs continued pretraining or retrieval instead.

The one thing they share: the loss function

Same next-token cross-entropy

Both stages run autoregressive language modelling. For a sequence of tokens x_1, x_2, ..., x_T, the model predicts each token from the ones before it, and the loss for each position is the cross-entropy between the predicted distribution and the true next token:

\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})

This is identical in pretraining and SFT. The forward pass is the same. The backward pass is the same. The model architecture is the same. If you watched the GPU utilisation graph alone you could not tell which stage was running.

Why this leads to confusion

Because the loss is identical, candidates sometimes claim the two stages "use different objectives." They do not. The objective is the same; what changes is everything around it. The data the loss sees, the rate at which weights update, the number of steps, and which tokens contribute to the loss all differ dramatically.

The framing that helps

Think of pretraining and fine-tuning as the same algorithm applied at different scales and on different distributions, with different goals. Pretraining is the algorithm building knowledge from scratch. Fine-tuning is the same algorithm gently reshaping that knowledge into the behaviour you want. Same recipe, very different ingredients.

Data: volume, shape, and labelling

Optimisation: learning rate and duration

Loss masking and what it implies

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Dimension	Pretraining	Fine-tuning
Data volume	Trillions of tokens	Thousands to low millions of examples
Data shape	Raw text, no labels	Structured demos, often chat-formatted
Learning rate	1e-4 to 6e-4	1e-5 to 2e-5 full, 1e-4 LoRA
Duration and hardware	Months on thousands of GPUs	Hours to days on tens of GPUs
Loss masking	Every token contributes	Prompt masked, response only
What changes	World knowledge built from scratch	Behaviour, format, style

Real products, models, and research that use this idea.

Llama 4 Maverick was pretrained on roughly 30T tokens at peak LR 3e-4, then SFT-fine-tuned on a curated dataset around 1e-5.
DeepSeek V4 publishes its pretraining LR schedule (warmup + cosine to 1e-4) and its post-training LR (around 2e-5), illustrating the standard order of magnitude gap.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does a high learning rate cause catastrophic forgetting during fine-tuning?

Pretrained weights sit in a sharp loss basin learned over trillions of tokens. A large update step pushes the parameters out of that basin, and the small fine-tuning dataset is not enough signal to rebuild the general distribution. Small LR keeps you near the basin while still allowing local shaping.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming pretraining and fine-tuning use different loss functions. They run the same next-token cross-entropy; what changes is the data, the learning rate, the duration, and which tokens contribute to the loss.

60 second bullets to scan on the way to the call.

Why the loss function is the same in both stages
The order of magnitude gap in data volume

Primary sources. Browse if you want the original framing.