Match training regime to sensible peak LR range
Drag each answer to line up with its matching prompt
Full fine-tuning of a 7B model
Peak LR around 1e-5 to 5e-5 with cosine decay and ~3% warmup.
LoRA r=16 SFT
Peak LR around 1e-5, near the lowest point of the original pretraining cosine schedule: to avoid catastrophic forgetting.
QLoRA on a 70B base
Peak LR around 1e-4 to 5e-4, since LoRA tolerates ~10× higher LR than full FT.
DPO from an SFT checkpoint
Peak LR around 1e-4 to 2e-4: same regime as LoRA, with the 4-bit base unchanged by LR choice.
Continued pretraining on a new corpus
Peak LR around 5e-7 to 5e-6, much lower than SFT because preference gradients are fragile.
Peak LR tracks how fragile the update is: LoRA SFT runs hottest near 1e-4 to 5e-4, full fine-tuning sits mid, and DPO plus continued pretraining run coldest.
Picture how bold each change is when you fix up something you already care about. If you scribble on a fresh sticky note that started blank, you can press hard and write fast, since there is nothing to ruin. If you touch up a finished painting, you dab gently so you do not smear years of work. If you nudge someone toward a very subtle taste, the hint is so faint that any big stroke wipes it out, so you barely tap. And if you keep teaching a wise old friend brand-new books, you whisper the lessons so they do not forget everything they already knew. The rule is the same everywhere: the more precious and finished the thing, the smaller your step should be.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This match question hides a single organising principle behind five seemingly unrelated numbers. The peak learning rate for each regime is not chosen by model size or by tradition. It is chosen by how fragile the update is and how much already-useful knowledge is exposed to corruption.
The bands span four orders of magnitude, from 5e-7 for DPO up to 5e-4 for LoRA SFT. That spread looks intimidating until you reorder the regimes by one axis: how much is at stake per step. Frozen-base adapter methods have almost nothing to lose, so they run hot. Full-parameter and preference methods expose precious signal to every gradient, so they run cold.
The rest of this walkthrough builds that fragility axis explicitly, places each of the five regimes on it, and explains why two of them share a band while another sits a hundred times below the rest. By the end the matching is mechanical rather than memorised.
The organising axis: fragility, not size
The instinct most candidates bring is that bigger models need smaller learning rates. That correlation is weak and misleading here. A 70B QLoRA run uses a higher peak rate than a 7B full fine-tune, which immediately breaks the size heuristic. Size barely moves the safe band; what moves it is what the gradient can break.
The real axis is fragility: how much damage one optimiser step can do. Two factors set it. First, how many live parameters carry pretrained knowledge that a hot step could overwrite. Second, how noisy or small the gradient signal is, since a faint signal amplified by a large step produces unstable updates. A regime that scores low on both can afford a hot rate; a regime that scores high on either must run cold.
There is a third factor worth naming: how far the update travels from a known-good starting point. A LoRA adapter starts at zero and adds a small correction, so even a hot rate keeps the effective model close to the frozen base. Full fine-tuning and continued pretraining move the actual weights, so every step compounds and drift accumulates across thousands of updates. DPO moves weights too, but from an already-aligned checkpoint, where any large move is a regression rather than progress.
When you score each regime on these factors, the ordering falls out cleanly. LoRA and QLoRA freeze the base, so live-knowledge risk is near zero. Full fine-tuning exposes every weight. DPO carries an unusually fragile gradient on top of a checkpoint you do not want to disturb. Continued pretraining risks forgetting an entire pretraining corpus. That single axis, not parameter count, predicts the band, and it predicts the schedule shape too.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Regime | Peak LR band | Why this band |
|---|---|---|
| LoRA r=16 SFT | 1e-4 to 5e-4 | Frozen base, zero-init adapters tolerate ~10x higher LR |
| QLoRA on 70B | 1e-4 to 2e-4 | Same as LoRA; 4-bit base is frozen and LR-inert |
| Full FT of 7B | 1e-5 to 5e-5 | All weights live; hot LR forgets or diverges |
| Continued pretraining | around 1e-5 | Near schedule floor to avoid catastrophic forgetting |
| DPO from SFT checkpoint | 5e-7 to 5e-6 | Fragile log-ratio gradient destabilises easily |
Real products, models, and research that use this idea.
- Hugging Face TRL ships default DPO configs near 5e-7 and LoRA SFT configs near 2e-4, mirroring this fragility ordering.
- Axolotl and Unsloth recipes for Llama 4 and Qwen 3 LoRA fine-tunes default to 1e-4 to 2e-4 peak LR with cosine decay.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does a zero-initialised LoRA adapter tolerate a much higher peak learning rate than full fine-tuning?
Reason about what is at risk. The base is frozen, so no pretrained weight can be corrupted; the adapter starts at zero and adds nothing until trained, so a larger step fills capacity without destroying prior knowledge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reusing one favourite learning rate across regimes. A LoRA-tuned 2e-4 will blow up full fine-tuning and is roughly 100 times too hot for DPO.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.