Pretraining vs fine-tuning: two key differences in data and objective.

**Pretraining** and **fine-tuning** are two stages of training the same kind of model, and they share more than you might think. The actual loss function is identical: next-token cross-entropy on a causal-LM forward pass. What changes is everything around the loss. Pretraining feeds the model trillions of raw web tokens (CommonCrawl, books, code, papers) and runs for months on thousands of GPUs. The model starts with random weights and learns the statistics of language from scratch. Fine-tuning starts from a pretrained model that already knows the language. It feeds the model a tiny, curated set, usually thousands or low millions of examples, often formatted as chat conversations or instruction-response pairs. Training runs for hours or a few days on far fewer GPUs. The learning rate gap is large. Pretraining uses high learning rates around 1e-4 to 6e-4. Fine-tuning uses much lower rates, typically 1e-5 to 2e-5 for full fine-tuning, because the goal is to nudge an already-trained model gently. A high learning rate during fine-tuning is one of the fastest ways to cause catastrophic forgetting, where the model loses the knowledge it gained during pretraining.

**Pretraining and fine-tuning** share the next-token cross-entropy objective but differ on every other dimension that matters in practice. **Data is the first major axis.** Pretraining ingests trillions of largely unlabelled, raw tokens scraped from the web, books, papers, and code. There is no notion of prompt versus response; the model just learns the statistics of language as continuous text. Fine-tuning consumes a tiny, curated set, typically anywhere from 1k to a few hundred k examples, of structured demonstrations: SFT chat pairs, preference triples, tool-calling traces. The model learns format and behaviour on top of knowledge it already has. **Training setup is the second major axis.** Pretraining uses high learning rates around 1e-4 to 6e-4 and runs for months across thousands of GPUs because the model is being built from scratch. Fine-tuning uses much smaller rates, around 1e-5 to 2e-5 for full fine-tuning or 1e-4 for LoRA, and runs for hours to a few days on far less hardware. A high fine-tuning learning rate is the fastest way to trigger catastrophic forgetting, where the model overwrites the pretraining distribution and loses its general capability. **A third worth mentioning is loss masking.** Pretraining computes loss on every token in the sequence. SFT typically masks the prompt and computes loss only on the response, because the model is being graded on what it should produce, not on what the user typed.