Contrast SFT and continued pretraining when adapting a base model to medical text

Short answer·Medium·4.0 · 0·~3 min·Asked atBanana Dev Contextual Ai Sap·Relevant atAnthropic Coreweave Databricks Fireworks Ai

Attempt it

A team wants to specialize a general base model on medical content. Contrast supervised fine-tuning and continued pretraining as the adaptation method, what each objective looks like, what each one actually changes inside the model, and how a serious team would combine them.

Free · 2 AI evals / day

**Continued pretraining** and **SFT** are two different ways to specialize a model, and they solve different problems. Continued pretraining keeps the original pretraining objective: predict the next token on a large corpus of raw, unlabeled text. The data is what changes. For a medical model, that means PubMed abstracts, clinical notes, drug labels, and textbooks. The training loss is identical to original pretraining; only the data distribution shifts. The effect is to move the model's knowledge prior: term frequencies, factual associations, and the latent shape of medical concepts all drift toward the domain. The model gets smarter about medicine. SFT runs on a small set of (instruction, response) pairs in the exact format your product will use. Examples for a medical setting: 'Summarize this clinical note in two paragraphs' paired with a model output, or 'List differential diagnoses for these symptoms' paired with an expert response. The loss is still next-token, but only on the response portion. The effect is to shape behaviour: how the model answers, what style it produces, how it follows the chat or task format. A serious team stacks both, in this order: continued pretraining first to inject medical knowledge, then SFT to shape behaviour. Doing SFT alone leaves knowledge gaps the instruction tuning cannot patch. Doing pretraining alone yields a model that knows medicine but cannot hold a conversation.

**The two methods change different things inside the model.** **Continued pretraining** runs raw next-token prediction on a large, unlabeled domain corpus. Same loss as the original pretraining, different data. For medicine that is PubMed abstracts, clinical notes, drug labels, medical textbooks, maybe tens of billions of tokens total. The effect is to move the knowledge prior: term frequencies shift, factual associations form, the latent geometry of medical concepts reorganises. The model gets smarter about the domain in a way that is independent of any specific task. **SFT** runs on paired (instruction, response) examples in the chat or task format the product will serve. Loss is still next-token, but the prompt tokens are masked (typically labelled -100) so only the response contributes to the loss. The data is small (thousands to tens of thousands of high-quality examples) and curated. The effect is to shape behaviour: response format, instruction following, chat conventions, style. The two methods are complementary, not substitutes. Pretraining moves knowledge. SFT moves behaviour. A model that has only seen SFT on medical Q&A pairs will produce well-formatted responses that hallucinate confidently on facts it does not really know, because the underlying knowledge prior never moved. A model that has only seen continued pretraining will know the domain cold but fail to follow chat conventions or instruction structure. The canonical stack is continued pretraining first, SFT second. Pretraining injects domain knowledge into the weights; SFT then teaches the model how to deploy that knowledge in the product's format. Skip pretraining when the base already covers the domain (Llama 4 Maverick and Claude Opus 4.7 base have strong medical priors); skip SFT when serving is non-chat like embedding lookup.

TL;DR

Continued pretraining shifts what the model knows via raw next-token on unlabeled corpora; SFT shifts how the model responds via (instruction, response) pairs. Stack them: pretraining first, SFT second.

Memory aid

Easy to grasp

Imagine training a doctor for a new specialty. One way is to lock them in a library full of medical journals and have them read for months until the vocabulary, the diagnoses, the drug names, and the rare diseases all become second nature. They walk out knowing the subject cold but unsure how to actually talk to a patient. The other way is to hand them a thick book of example patient conversations and teach them exactly how a good doctor responds in each one. They learn bedside manner perfectly, but if they did not read the journals first, they will sometimes give confident answers about diseases they do not really understand. A serious training program stacks both: first the reading, then the practice conversations. Skip either step and the doctor either knows everything or sounds professional, but rarely both.

Key concepts

Supervised fine-tuning Continued pretraining Domain adaptation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Domain adaptation is the question senior interviewers use to test whether a candidate understands what fine-tuning actually does inside a model versus what it just looks like from the outside. The medical adaptation framing is a classic because it has clear failure modes that surface only when you understand the difference between knowledge and behaviour.

The wrong answer that loses the interview is treating SFT and continued pretraining as interchangeable techniques on a difficulty ladder. They are not. They serve different purposes, change different parts of the model, and fail in different ways when used alone. A team that runs SFT on medical Q&A pairs and ships the result will see fluent, well-formatted responses that hallucinate confidently on facts the base model never internalised, because no amount of instruction tuning rewrites the knowledge prior. A team that runs continued pretraining without SFT will produce a model that knows medicine cold but cannot hold a conversation in the product's chat format.

The rest of this section walks the two objectives in detail, explains what each one actually changes in the model, lays out the canonical pipeline stack, addresses catastrophic forgetting as the main risk in continued pretraining, and ends with the decision rule for when to skip either step.

Two objectives, two purposes

Both methods use a next-token cross-entropy loss. The difference is the data shape and what gets masked.

Continued pretraining runs the loss on raw, unlabeled text from a domain corpus. No instruction structure, no prompt-response separation. The model sees a stream of tokens (a PubMed abstract, a clinical note, a drug monograph) and predicts each next token. The loss is computed on every token. This is identical to original pretraining at the loss level; only the data distribution differs.

The effect of continued pretraining is on the weights as a knowledge prior. Token frequencies in the domain shift the model's expectations: medical abbreviations stop looking unusual, drug names stop being out-of-distribution, factual associations between symptoms and conditions strengthen, the latent geometry of medical concepts reorganises. The model gets smarter about the domain in a generalised way that is independent of any specific task or response format.

SFT runs the loss on paired (instruction, response) examples in the chat or task format the product will serve. Crucially, the prompt tokens are masked out of the loss (typically labelled -100 in the dataset). Only the response tokens contribute to the gradient. The model sees the prompt as context and learns to produce the response.

The effect of SFT is on the conditional distribution P(response given prompt) for the formats the product uses. It teaches the model how to respond, not what to know. The knowledge prior built by pretraining is left largely intact; what changes is how that knowledge gets deployed in the product's interaction shape.

What each method actually changes in the model

The canonical stack and pipeline order

Catastrophic forgetting and when to skip steps

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Continued pretraining	SFT
Objective	Next-token on raw corpus	Next-token on instruction-response, prompt masked
Data shape	Unlabeled domain text	Paired (instruction, response) examples
Data volume	Billions of tokens	5k to 50k high-quality pairs
What changes	Knowledge prior, latent geometry	Behaviour, format, instruction following
Failure if alone	Knows domain, cannot hold a conversation	Follows format, hallucinates on unknowns
Risk	Catastrophic forgetting of general capability	Overfitting to labellers, format brittleness

Real products, models, and research that use this idea.

Med-PaLM 2 used continued pretraining on biomedical corpora followed by SFT on clinical instruction data, the canonical medical-specialization stack.
BioGPT extended general LLM training with continued pretraining on PubMed to build a biomedical generation model before any task-specific fine-tuning.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is catastrophic forgetting in continued pretraining and how do you mitigate it?

Domain-only continued pretraining drifts general capability, especially reasoning and instruction following. Mitigate by mixing 10 to 30 percent general data into the corpus, using a learning rate an order of magnitude below original pretraining, and stopping after a few billion tokens rather than running long.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating SFT on medical Q&A pairs as sufficient for domain adaptation. SFT teaches format and style; it cannot patch underlying knowledge gaps in the base model.

60 second bullets to scan on the way to the call.

The objective and data shape for each method
Why both are next-token but SFT masks prompt tokens

Primary sources. Browse if you want the original framing.