Contrast SFT and continued pretraining when adapting a base model to medical text

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

A team wants to specialize a general base model on medical content. Contrast supervised fine-tuning and continued pretraining as the adaptation method, what each objective looks like, what each one actually changes inside the model, and how a serious team would combine them.

Free · 2 AI evals / day

**Continued pretraining** and **SFT** are two different ways to specialize a model, and they solve different problems. Continued pretraining keeps the original pretraining objective: predict the next token on a large corpus of raw, unlabeled text. The data is what changes. For a medical model, that means PubMed abstracts, clinical notes, drug labels, and textbooks. The training loss is identical to original pretraining; only the data distribution shifts. The effect is to move the model's knowledge prior: term frequencies, factual associations, and the latent shape of medical concepts all drift toward the domain. The model gets smarter about medicine. SFT runs on a small set of (instruction, response) pairs in the exact format your product will use. Examples for a medical setting: 'Summarize this clinical note in two paragraphs' paired with a model output, or 'List differential diagnoses for these symptoms' paired with an expert response. The loss is still next-token, but only on the response portion. The effect is to shape behaviour: how the model answers, what style it produces, how it follows the chat or task format. A serious team stacks both, in this order: continued pretraining first to inject medical knowledge, then SFT to shape behaviour. Doing SFT alone leaves knowledge gaps the instruction tuning cannot patch. Doing pretraining alone yields a model that knows medicine but cannot hold a conversation.

**The two methods change different things inside the model.** **Continued pretraining** runs raw next-token prediction on a large, unlabeled domain corpus. Same loss as the original pretraining, different data. For medicine that is PubMed abstracts, clinical notes, drug labels, medical textbooks, maybe tens of billions of tokens total. The effect is to move the knowledge prior: term frequencies shift, factual associations form, the latent geometry of medical concepts reorganises. The model gets smarter about the domain in a way that is independent of any specific task. **SFT** runs on paired (instruction, response) examples in the chat or task format the product will serve. Loss is still next-token, but the prompt tokens are masked (typically labelled -100) so only the response contributes to the loss. The data is small (thousands to tens of thousands of high-quality examples) and curated. The effect is to shape behaviour: response format, instruction following, chat conventions, style. The two methods are complementary, not substitutes. Pretraining moves knowledge. SFT moves behaviour. A model that has only seen SFT on medical Q&A pairs will produce well-formatted responses that hallucinate confidently on facts it does not really know, because the underlying knowledge prior never moved. A model that has only seen continued pretraining will know the domain cold but fail to follow chat conventions or instruction structure. The canonical stack is continued pretraining first, SFT second. Pretraining injects domain knowledge into the weights; SFT then teaches the model how to deploy that knowledge in the product's format. Skip pretraining when the base already covers the domain (Llama 4 Maverick and Claude Opus 4.7 base have strong medical priors); skip SFT when serving is non-chat like embedding lookup.

Quick reference

TL;DR

Continued pretraining shifts what the model knows via raw next-token on unlabeled corpora; SFT shifts how the model responds via (instruction, response) pairs. Stack them: pretraining first, SFT second.

Memory aid

Easy to grasp

Imagine training a doctor for a new specialty. One way is to lock them in a library full of medical journals and have them read for months until the vocabulary, the diagnoses, the drug names, and the rare diseases all become second nature. They walk out knowing the subject cold but unsure how to actually talk to a patient. The other way is to hand them a thick book of example patient conversations and teach them exactly how a good doctor responds in each one. They learn bedside manner perfectly, but if they did not read the journals first, they will sometimes give confident answers about diseases they do not really understand. A serious training program stacks both: first the reading, then the practice conversations. Skip either step and the doctor either knows everything or sounds professional, but rarely both.

Key concepts

Supervised fine-tuning Continued pretraining Domain adaptation

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: the two objectives compared, what each changes in the model, the canonical stack order, catastrophic forgetting and its mitigations, the data shape and volume for each step, and when to skip either.

Side-by-side

Aspect	Continued pretraining	SFT
Objective	Next-token on raw corpus	Next-token on instruction-response, prompt masked
Data shape	Unlabeled domain text	Paired (instruction, response) examples
Data volume	Billions of tokens	5k to 50k high-quality pairs
What changes	Knowledge prior, latent geometry	Behaviour, format, instruction following
Failure if alone	Knows domain, cannot hold a conversation	Follows format, hallucinates on unknowns
Risk	Catastrophic forgetting of general capability	Overfitting to labellers, format brittleness

Real-world examples

Real products, models, and research that use this idea.

Med-PaLM 2 used continued pretraining on biomedical corpora followed by SFT on clinical instruction data, the canonical medical-specialization stack.
BioGPT extended general LLM training with continued pretraining on PubMed to build a biomedical generation model before any task-specific fine-tuning.
BloombergGPT 50B used continued pretraining on financial corpora as the dominant adaptation step before instruction tuning.
Llama 4 Maverick base shows strong medical priors out of pretraining, so many 2026 medical fine-tunes skip continued pretraining and start with SFT plus retrieval.
Clinical NER fine-tunes for ICD-10 coding use SFT only because the deployment is classification, not chat, so the chat-conventions step is unnecessary.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is catastrophic forgetting in continued pretraining and how do you mitigate it?

QWhy does SFT use masked prompts (labels = -100) on the prompt tokens?

QWhen would you choose retrieval over fine-tuning for a medical use case?

QHow would you structure the data for SFT on medical chat?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating SFT on medical Q&A pairs as sufficient for domain adaptation. SFT teaches format and style; it cannot patch underlying knowledge gaps in the base model.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.