A team wants to specialize a general base model on medical content. Contrast supervised fine-tuning and continued pretraining as the adaptation method, what each objective looks like, what each one actually changes inside the model, and how a serious team would combine them.
Free · 2 AI evals / day
**Continued pretraining** and **SFT** are two different ways to specialize a model, and they solve different problems.
Continued pretraining keeps the original pretraining objective: predict the next token on a large corpus of raw, unlabeled text. The data is what changes. For a medical model, that means PubMed abstracts, clinical notes, drug labels, and textbooks. The training loss is identical to original pretraining; only the data distribution shifts. The effect is to move the model's knowledge prior: term frequencies, factual associations, and the latent shape of medical concepts all drift toward the domain. The model gets smarter about medicine.
SFT runs on a small set of (instruction, response) pairs in the exact format your product will use. Examples for a medical setting: 'Summarize this clinical note in two paragraphs' paired with a model output, or 'List differential diagnoses for these symptoms' paired with an expert response. The loss is still next-token, but only on the response portion. The effect is to shape behaviour: how the model answers, what style it produces, how it follows the chat or task format.
A serious team stacks both, in this order: continued pretraining first to inject medical knowledge, then SFT to shape behaviour. Doing SFT alone leaves knowledge gaps the instruction tuning cannot patch. Doing pretraining alone yields a model that knows medicine but cannot hold a conversation.
**The two methods change different things inside the model.**
**Continued pretraining** runs raw next-token prediction on a large, unlabeled domain corpus. Same loss as the original pretraining, different data. For medicine that is PubMed abstracts, clinical notes, drug labels, medical textbooks, maybe tens of billions of tokens total. The effect is to move the knowledge prior: term frequencies shift, factual associations form, the latent geometry of medical concepts reorganises. The model gets smarter about the domain in a way that is independent of any specific task.
**SFT** runs on paired (instruction, response) examples in the chat or task format the product will serve. Loss is still next-token, but the prompt tokens are masked (typically labelled -100) so only the response contributes to the loss. The data is small (thousands to tens of thousands of high-quality examples) and curated. The effect is to shape behaviour: response format, instruction following, chat conventions, style.
The two methods are complementary, not substitutes. Pretraining moves knowledge. SFT moves behaviour. A model that has only seen SFT on medical Q&A pairs will produce well-formatted responses that hallucinate confidently on facts it does not really know, because the underlying knowledge prior never moved. A model that has only seen continued pretraining will know the domain cold but fail to follow chat conventions or instruction structure.
The canonical stack is continued pretraining first, SFT second. Pretraining injects domain knowledge into the weights; SFT then teaches the model how to deploy that knowledge in the product's format. Skip pretraining when the base already covers the domain (Llama 4 Maverick and Claude Opus 4.7 base have strong medical priors); skip SFT when serving is non-chat like embedding lookup.
Quick reference
TL;DR
Continued pretraining shifts what the model knows via raw next-token on unlabeled corpora; SFT shifts how the model responds via (instruction, response) pairs. Stack them: pretraining first, SFT second.
Imagine training a doctor for a new specialty. One way is to lock them in a library full of medical journals and have them read for months until the vocabulary, the diagnoses, the drug names, and the rare diseases all become second nature. They walk out knowing the subject cold but unsure how to actually talk to a patient. The other way is to hand them a thick book of example patient conversations and teach them exactly how a good doctor responds in each one. They learn bedside manner perfectly, but if they did not read the journals first, they will sometimes give confident answers about diseases they do not really understand. A serious training program stacks both: first the reading, then the practice conversations. Skip either step and the doctor either knows everything or sounds professional, but rarely both.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Base model already covers the domain well (general medical knowledge in modern flagships) Continued pretraining adds little when the base is already strong; SFT or retrieval is enough.
Use case is fact retrieval or lookup rather than generation Retrieval over a domain knowledge base often beats fine-tuning, with lower cost and better updatability.
Non-conversational deployment (embedding lookup, classifier head) SFT teaches chat behaviour the product does not use, so the adaptation budget is better spent elsewhere.
Small domain corpus under 100 million tokens Continued pretraining at small data scale risks overfitting and forgetting without delivering meaningful knowledge shift.
Quick scan · 2–4 min
Everything important, quickly.
5 min: the two objectives compared, what each changes in the model, the canonical stack order, catastrophic forgetting and its mitigations, the data shape and volume for each step, and when to skip either.
Side-by-side
Aspect
Continued pretraining
SFT
Objective
Next-token on raw corpus
Next-token on instruction-response, prompt masked
Data shape
Unlabeled domain text
Paired (instruction, response) examples
Data volume
Billions of tokens
5k to 50k high-quality pairs
What changes
Knowledge prior, latent geometry
Behaviour, format, instruction following
Failure if alone
Knows domain, cannot hold a conversation
Follows format, hallucinates on unknowns
Risk
Catastrophic forgetting of general capability
Overfitting to labellers, format brittleness
Real-world examples
Real products, models, and research that use this idea.
1Med-PaLM 2 used continued pretraining on biomedical corpora followed by SFT on clinical instruction data, the canonical medical-specialization stack.
2BioGPT extended general LLM training with continued pretraining on PubMed to build a biomedical generation model before any task-specific fine-tuning.
3BloombergGPT 50B used continued pretraining on financial corpora as the dominant adaptation step before instruction tuning.
4Llama 4 Maverick base shows strong medical priors out of pretraining, so many 2026 medical fine-tunes skip continued pretraining and start with SFT plus retrieval.
5Clinical NER fine-tunes for ICD-10 coding use SFT only because the deployment is classification, not chat, so the chat-conventions step is unnecessary.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is catastrophic forgetting in continued pretraining and how do you mitigate it?
Sign in or go Pro to see the approach for this follow-up.
QWhy does SFT use masked prompts (labels = -100) on the prompt tokens?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you choose retrieval over fine-tuning for a medical use case?
Sign in or go Pro to see the approach for this follow-up.
QHow would you structure the data for SFT on medical chat?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating SFT on medical Q&A pairs as sufficient for domain adaptation. SFT teaches format and style; it cannot patch underlying knowledge gaps in the base model.