A red-teamer probes a 7B medical assistant that was fine-tuned on roughly 8,000 internal records. With targeted prompts ("continue this clinical note...") the model emits verbatim credit-card numbers, names, and dates of birth drawn from the training set. Name the failure mode, explain why it shows up so cleanly on a small dataset, and lay out the standard mitigation stack a serious team would deploy before the next training run.
Free · 2 AI evals / day
The failure mode is **training-data memorization**. The model has not learned a general pattern; it has memorised specific strings from training and emits them verbatim when a prompt happens to match the beginning of one of those strings.
Why it shows up so cleanly on a small dataset. With only 8,000 examples and a few epochs, every individual example contributes a much larger share of the total gradient signal than it would in a larger run. Rare strings like specific credit-card numbers, names, and dates of birth become high-influence training points. The model effectively learns each one. At inference, a targeted prompt acts as a key that retrieves the memorised continuation.
The standard mitigation stack has four layers.
First, scrub PII and deduplicate before any training runs. A PII detector masks or drops credit-card numbers, social security numbers, phone numbers, names where possible, and exact dates. Deduplication makes sure no single record gains extra influence by appearing many times.
Second, train fewer epochs and at a lower learning rate. Memorisation rises sharply with both. A single pass at a conservative learning rate is the safest regime for small datasets.
Third, use DP-SGD if the regulatory bar is high. Differential privacy bounds the influence of any single example at a measurable cost to quality and training speed.
Fourth, stand up an extraction-attack red-team as a pre-deploy gate. Run targeted prompts against held-out PII before the model ships.
**The failure mode is training-data memorization**, the mechanism that powers membership-inference and extraction attacks. Small datasets amplify it because per-example gradient influence scales inversely with dataset size. An 8,000-example corpus across three epochs encodes rare strings into the weights almost verbatim. The leaked PII is then retrievable by any prompt overlapping the start of the memorised continuation.
The mitigation stack has four layers, ordered cheapest to most expensive.
First, the data layer. Run a PII detector over the corpus before any training touches it. Mask or drop credit-card numbers, social security numbers, phone numbers, exact dates, and named entities where the schema allows. Then deduplicate near-identical examples to catch records that gained influence through accidental repetition.
Second, the training layer with conservative hyperparameters. Memorisation scales sharply with epoch count and learning rate. A single epoch at a learning rate 2 to 4 times lower than the default is the safest regime for small datasets.
Third, the optimisation layer with DP-SGD when the regulatory bar demands a formal privacy guarantee. DP-SGD clips per-example gradients and adds calibrated noise so no single example's contribution exceeds a measurable bound. It costs measurable quality and slows training; reserve it for cases where legal cost of any leak exceeds quality cost.
Fourth, the verification layer with a pre-deploy extraction-attack red-team. Hold out a small set of PII strings, run targeted prompts that would surface them if the model memorised them, and treat any extraction as a release-gate failure. This layer catches what the others missed and provides the audit trail regulators ask for.
Quick reference
TL;DR
Training-data memorization: small dataset gives each example outsized gradient influence, so rare PII gets memorized; fix with PII scrub, fewer epochs, lower LR, optional DP-SGD, and a red-team gate.
Imagine teaching someone a song by playing it only ten times instead of a thousand times. The few repetitions land hard, and the student remembers the exact wording of the choruses verbatim, including the strange lyrics they would normally forget. Now imagine one of those choruses contains a real credit-card number that someone slipped in. The student can recite it back when prompted with the opening line. The cure is not to make the student smarter; it is to clean the songbook before practice begins, sing each song fewer times, sing more softly, and have a friend probe the student afterward to check they have not memorised anything sensitive. The model behaves the same way. Small data plus many passes equals verbatim recall of whatever you forgot to scrub.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Large pretraining corpora with billions of tokens and no specific high-entropy PII Per-example influence is naturally low at pretraining scale, and the memorisation risk shifts to long-document extraction rather than single-record PII leakage.
Reference-data tasks where retrieval-augmented generation can replace fine-tuning Keeping the sensitive content in a retrieval index rather than the weights sidesteps memorisation entirely and lets access controls apply per-document.
Synthetic or fully de-identified datasets with no real PII present If the corpus is verifiably clean before training, the memorisation risk for PII disappears, though general memorisation of distinctive examples remains.
Quick scan · 2–4 min
Everything important, quickly.
7 min: name memorisation as the mode, explain small-dataset amplification of per-example influence, prescribe the four-layer stack (PII scrub + dedup, conservative hyperparameters, DP-SGD when needed, extraction red-team gate), and call out RAG as an alternative architecture.
Real-world examples
Real products, models, and research that use this idea.
1Carlini et al. demonstrated extraction of memorised training strings from GPT-2 with targeted prompts, establishing the threat model this question instantiates.
2Google's DP-SGD via TF-Privacy and Opacus for PyTorch are production-grade differential privacy implementations used in regulated industries like healthcare and finance.
3Microsoft's Presidio and AWS Comprehend PII are widely used PII detection layers that production teams deploy at ingest time before any fine-tuning corpus is assembled.
4Modern flagship deployments like Claude Opus 4.7 and Llama 4 in healthcare contexts pair conservative SFT recipes with extraction-attack red-team gates as standard release-gate practice.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you calibrate the DP-SGD epsilon for a healthcare fine-tune, and what quality cost should you budget?
Sign in or go Pro to see the approach for this follow-up.
QWhat does a canary-based memorisation audit look like in practice?
Sign in or go Pro to see the approach for this follow-up.
QWhen should RAG replace fine-tuning entirely for sensitive reference data?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming the model architecture and proposing 'add a safety filter at inference'. The leak originates in training and any inference-time filter is a band-aid; the real fix is upstream of training.