Designing a code-assistant SFT mix: how should domain and general data balance?

Short answer·Medium·4.0 · 0·~3 min·Relevant atCohere Databricks Meta Mistral AI

Attempt it

A team is SFT'ing a strong base instruct model into a code-assistant product. They have access to high-quality code-instruction data (CodeAlpaca, Magicoder, internal code-review pairs) and general chat data (Tulu, UltraChat). Specify the starting mixture ratio, the role each component plays, how to tune the ratio, and what fails at both extremes.

Free · 2 AI evals / day

**Start at roughly 80% code instruction data and 20% general chat data.** The two slices play different roles. **The code slice** (CodeAlpaca, Magicoder, internal code-review pairs) is the primary signal. It carries the behaviour the product needs: writing code, explaining bugs, suggesting fixes. **The general slice** (Tulu, UltraChat, OpenAssistant) is anti-forgetting replay. Its job is to refresh the broad-chat distribution the base already had, so the model does not lose its ability to explain code in plain English or hold a follow-up conversation. **Interleave the two at the row level**, not as big blocks. If you concatenate code rows then chat rows, the model overfits the early block and partially forgets it. Per-batch mixing keeps both signals present every optimizer step. **Tune empirically.** Run pilots at 90/10, 80/20, 70/30. Read a code metric (HumanEval, MBPP, or internal pass-at-1) and a general metric (MMLU, MT-Bench). Pick the lowest general share that keeps the general metric within 1 to 2 points of base. **Both extremes fail in opposite ways.** Below 5% general, catastrophic forgetting: code climbs but general drops. Above 30% general, code plateaus because batches are mostly off-topic.

**Start at 80% code / 20% general**, with the two slices doing distinct jobs. **The code slice is the primary signal.** It contains CodeAlpaca, Magicoder, and internal code-review pairs, the data that teaches product behaviour. Most of the gradient comes from here. **The general slice is anti-forgetting replay.** It contains Tulu, UltraChat, OpenAssistant, broad-chat data that refreshes the base's general capabilities. It does not teach new behaviour; it prevents the base's existing capabilities from being washed out. **Mix at the row level, not in blocks.** Per-batch interleaving keeps both signals present every optimizer step. Block-concatenation creates curriculum effects: the model overfits the early block and partially forgets it later. **Tune empirically.** Run three pilots at 90/10, 80/20, 70/30. Read two evals in parallel: a code metric (HumanEval, MBPP, internal pass-at-1) and a general metric (MMLU, MT-Bench). The right ratio is the lowest general share that keeps the regression within tolerance (under 1 to 2 points) while the code metric climbs. **Both extremes fail in opposite ways.** Under 5% general gives catastrophic forgetting: HumanEval climbs, MMLU and MT-Bench drop, users complain the model forgot how to explain. Over 30% general plateaus the code metric because batches are mostly off-topic, and you ship a weaker assistant than the tighter mix would produce. The 70 to 85 percent domain range is the sweet spot for most code SFT runs.

Quick reference

TL;DR

Start at 80% code / 20% general, with the general slice acting as anti-forgetting replay. Interleave at the row level. Tune empirically with pilots at 90/10, 80/20, 70/30 reading both a code metric and a general metric.

Memory aid

Easy to grasp

Picture training a specialist doctor who also has to talk to patients in plain language. Most of the training is medical cases, that is what makes them good at the speciality. But a small share is conversational practice, because if they spend a year on only technical cases they forget how to explain a diagnosis to a frightened parent. The medical slice teaches the new skill. The conversational slice keeps the old skill from rotting. The right balance is mostly medical with enough conversation to keep the bedside manner, and finding it means watching both their technical scores and their patient-feedback scores. Both extremes fail in opposite ways.

Key concepts

Data mixture Catastrophic forgetting Supervised fine-tuning

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: name the 80/20 starting ratio, distinguish the two slice roles, describe the row-level interleaving and the pilot protocol, and end with both failure modes.

Real-world examples

Real products, models, and research that use this idea.

DeepSeek V4 Coder, a 2026 code-specialist model, uses a high-domain SFT mix with documented anti-forgetting replay drawn from general chat sources.
Allen AI's Tulu 3 mixture demonstrates the two-slice structure at open-data scale, with per-slice ablation results showing the trade-off the question describes.
Anthropic's published guidance on building Claude Opus 4.7 character training discusses general-capability replay as a deliberate component of specialist tuning.
Llama 4 Maverick fine-tunes for medical and legal verticals in 2026 use the same domain plus replay structure with domain shares typically in the 75 to 85 percent range.
Hugging Face's TRL examples for code-specialist fine-tunes recommend per-batch row-level interleaving and provide the dataset mixing utilities needed to implement it.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how you would adapt the 80/20 mix for a fine-tune on a low-resource programming language the base model barely knows.

QHow would you measure whether the anti-forgetting replay is actually working, beyond just reading MMLU?

QWhy does row-level interleaving outperform block-concatenation even though both sees the same total tokens?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Going 100% domain and discovering at deployment that the model lost its ability to explain code in plain English or hold a multi-turn conversation. The anti-forgetting replay slice is small but load-bearing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.