Designing a code assistant SFT mix: how should domain and general data balance?
A team is SFT'ing a strong base instruct model into a code assistant product. They have access to high quality code instruction data (CodeAlpaca, Magicoder, internal code review pairs) and general chat data (Tulu, UltraChat). Specify the starting mixture ratio, the role each component plays, how to tune the ratio, and what fails at both extremes.
Start at 80% code / 20% general, with the general slice acting as anti-forgetting replay. Interleave at the row level. Tune empirically with pilots at 90/10, 80/20, 70/30 reading both a code metric and a general metric.
Picture training a specialist doctor who also has to talk to patients in plain language. Most of the training is medical cases, that is what makes them good at the speciality. But a small share is conversational practice, because if they spend a year on only technical cases they forget how to explain a diagnosis to a frightened parent. The medical slice teaches the new skill. The conversational slice keeps the old skill from rotting. The right balance is mostly medical with enough conversation to keep the bedside manner, and finding it means watching both their technical scores and their patient-feedback scores. Both extremes fail in opposite ways.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mixing domain and general data for a specialist SFT run is one of the most consequential decisions in fine-tuning, and one of the most often handled by guesswork. Teams that go too narrow ship a one-trick model that lost the base's prior capabilities. Teams that go too broad ship a model that does not actually specialise. The right answer is neither extreme; it is a principled mix tuned empirically.
The framework that works is the two-job framing: the domain slice teaches the new capability, the general slice prevents the existing capability from washing out. Calling the general slice 'anti-forgetting replay' makes the role explicit and prevents the common mistake of treating it as additional content to learn.
This deep dive walks through the two-job framing, the 80/20 starting ratio and the range around it, why row-level interleaving matters more than people expect, the pilot and eval protocol for tuning the ratio, and the symmetric failure modes at both extremes.
The two-job framing
When you fine-tune a strong base instruct model into a specialist, two things happen at once. The model acquires the specialist behaviour you trained on. The model also drifts away from its prior capabilities, because the gradient is dominated by the new training distribution.
Both processes are inevitable. The question is what balance you want between them.
The two-slice framing treats them as separate jobs. The domain slice carries the new capability. The general slice prevents the drift. The two slices do not compete for capacity; they cooperate to produce a model that has the new capability without losing the old ones.
This framing is borrowed from continual-learning research, where it is called experience replay. When training a model on a sequence of tasks, you keep some examples from earlier tasks in every batch as 'replay' to prevent the model from forgetting them. The same mechanism works in SFT: the general slice is your replay buffer for the base model's prior distribution.
The framing matters because it changes how you reason about the general slice composition. You are not trying to teach the model new general knowledge; the base already knows that. You are trying to keep its general distribution alive while the domain training pushes its representations in a new direction. The slice should look like what the base model already does well: chat, instruction following, explanation, multi-turn dialogue. Sourcing from Tulu, UltraChat, or OpenAssistant captures that distribution faithfully.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek V4 Coder, a 2026 code-specialist model, uses a high-domain SFT mix with documented anti-forgetting replay drawn from general chat sources.
- Allen AI's Tulu 3 mixture demonstrates the two-slice structure at open-data scale, with per-slice ablation results showing the trade-off the question describes.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through how you would adapt the 80/20 mix for a fine-tune on a low-resource programming language the base model barely knows.
Harder domain shifts need more domain data, so push to 90/10 or 85/15. The base has less prior knowledge to forget but more new knowledge to acquire; the balance shifts toward the new capability.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Going 100% domain and discovering at deployment that the model lost its ability to explain code in plain English or hold a multi-turn conversation. The anti-forgetting replay slice is small but load-bearing.
60 second bullets to scan on the way to the call.
The two-job framing: domain as primary signal, general as anti-forgetting replay
The 80/20 starting ratio and the 70 to 85 percent range for most cases
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.