How should domain and general data balance in a code assistant SFT mix?

Short answer·Medium·4.0 · 0·~3 min·Asked atCerebras Rephrase Ai Shield Ai·Relevant atCohere Databricks Meta Mistral AI

Attempt it

A team is SFT'ing a strong base instruct model into a code assistant product. They have access to high quality code instruction data (CodeAlpaca, Magicoder, internal code review pairs) and general chat data (Tulu, UltraChat). Specify the starting mixture ratio, the role each component plays, how to tune the ratio, and what fails at both extremes.

Free · 2 AI evals / day

**Start at roughly 80% code instruction data and 20% general chat data.** The two slices play different roles. **The code slice** (CodeAlpaca, Magicoder, internal code review pairs) is the primary signal. It carries the behaviour the product needs: writing code, explaining bugs, suggesting fixes. **The general slice** (Tulu, UltraChat, OpenAssistant) is anti-forgetting replay. Its job is to refresh the broad chat distribution the base already had. This way, the model does not lose its ability to explain code in plain English or hold a follow-up conversation. **Interleave the two at the row level**, not as big blocks. If you concatenate code rows then chat rows, the model overfits the early block and partially forgets it. Per batch mixing keeps both signals present every optimizer step. **Tune empirically.** Run pilots at 90/10, 80/20, 70/30. Read a code metric (HumanEval, MBPP, or internal pass-at-1) and a general metric (MMLU, MT-Bench). Pick the lowest general share that keeps the general metric within 1 to 2 points of base. **Both extremes fail in opposite ways.** Below 5% general, you get catastrophic forgetting: code climbs but general drops. Above 30% general, code plateaus because batches are mostly off-topic.

**Start at 80% code / 20% general**, with the two slices doing distinct jobs. **The code slice is the primary signal.** It contains CodeAlpaca, Magicoder, and internal code review pairs, the data that teaches product behaviour. Most of the gradient comes from here. **The general slice is anti-forgetting replay.** It contains Tulu, UltraChat, and OpenAssistant: broad chat data that refreshes the base's general capabilities. It does not teach new behaviour. It prevents the base's existing capabilities from being washed out. **Mix at the row level, not in blocks.** Per batch interleaving keeps both signals present every optimizer step. Block concatenation creates curriculum effects: the model overfits the early block and partially forgets it later. **Tune empirically.** Run three pilots at 90/10, 80/20, 70/30. Read two evals in parallel: a code metric (HumanEval, MBPP, internal pass-at-1) and a general metric (MMLU, MT-Bench). The right ratio is the lowest general share that keeps the regression within tolerance (under 1 to 2 points) while the code metric climbs. **Both extremes fail in opposite ways.** Under 5% general gives catastrophic forgetting: HumanEval climbs, MMLU and MT-Bench drop, users complain the model forgot how to explain. Over 30% general plateaus the code metric because batches are mostly off-topic. You then ship a weaker assistant than the tighter mix would produce. The 70 to 85 percent domain range is the sweet spot for most code SFT runs.

TL;DR

Start at 80% code / 20% general, with the general slice acting as anti-forgetting replay. Interleave at the row level. Tune empirically with pilots at 90/10, 80/20, 70/30 reading both a code metric and a general metric.

Memory aid

Easy to grasp

Picture training a specialist doctor who also has to talk to patients in plain language. Most of the training is medical cases. That is what makes them good at the speciality. But a small share is conversational practice. If they spend a year on only technical cases, they forget how to explain a diagnosis to a frightened parent. The medical slice teaches the new skill. The conversational slice keeps the old skill from rotting. The right balance is mostly medical, with enough conversation to keep the bedside manner. Finding it means watching both their technical scores and their patient-feedback scores. Both extremes fail in opposite ways.

Key concepts

Data mixture Catastrophic forgetting Supervised fine-tuning

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mixing domain and general data for a specialist SFT run is one of the most consequential decisions in fine-tuning. It is also one of the most often handled by guesswork. Teams that go too narrow ship a one-trick model that lost the base's prior capabilities. Teams that go too broad ship a model that does not specialise. The right answer is neither extreme. It is a principled mix tuned empirically.

The framework that works is the two-job framing. The domain slice teaches the new capability, and the general slice prevents the existing capability from washing out. Calling the general slice 'anti-forgetting replay' makes the role explicit and prevents the common mistake of treating it as additional content to learn.

The two-job framing

When you fine-tune a strong base instruct model into a specialist, two things happen at once. The model acquires the specialist behaviour you trained on. The model also drifts away from its prior capabilities, because the new training distribution dominates the gradient.

Both processes are inevitable. The question is what balance you want between them.

The two-slice framing treats them as separate jobs. The domain slice carries the new capability. The general slice prevents the drift. The two slices do not compete for capacity. They cooperate to produce a model that has the new capability without losing the old ones.

This framing is borrowed from continual learning research, where it is called experience replay. When you train a model on a sequence of tasks, you keep some examples from earlier tasks in every batch as 'replay'. This prevents the model from forgetting them. The same mechanism works in SFT: the general slice is your replay buffer for the base model's prior distribution.

The framing matters because it changes how you reason about the general slice composition. You are not trying to teach the model new general knowledge. The base already knows that. You are trying to keep its general distribution alive while the domain training pushes its representations in a new direction. The slice should look like what the base model already does well: chat, instruction following, explanation, multi-turn dialogue. Sourcing from Tulu, UltraChat, or OpenAssistant captures that distribution faithfully.

The 80/20 ratio and the range around it

Row-level interleaving matters

The pilot and eval protocol

Failure modes at both extremes

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Code Llama's code training mixed 85% code with 8% natural language about code and 7% general natural language, kept to retain natural language understanding. It is the same domain plus replay structure, applied to continued pretraining.
Ai2's Tulu 3 report ablates its SFT mixture by removing individual data subsets and reports which skills drop. That is the same kind of evidence the pilot protocol produces.

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how you would adapt the 80/20 mix for a fine-tune on a low-resource programming language the base model barely knows.

Harder domain shifts need more domain data, so push to 90/10 or 85/15. The base knows little of the language, so there is more new knowledge to acquire. The balance shifts toward the new capability.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Going 100% domain and discovering at deployment that the model lost its ability to explain code in plain English or hold a multi-turn conversation. The anti-forgetting replay slice is small but key.

60 second bullets to scan on the way to the call.

The two-job framing: domain as primary signal, general as anti-forgetting replay
The 80/20 starting ratio and the 70 to 85 percent range for most cases

Primary sources. Browse if you want the original framing.