Match training regime to sensible peak LR range
Drag each answer to line up with its matching prompt
Full fine-tuning of a 7B model
Peak LR around 1e-5 to 5e-5 with cosine decay and ~3% warmup.
LoRA r=16 SFT
Peak LR re-warmed toward the original pretraining peak (Code Llama reused Llama 2's rate), usually with replay of old data to limit catastrophic forgetting.
QLoRA on a 70B base
Peak LR around 1e-4 to 5e-4, since LoRA tolerates ~10× higher LR than full FT.
DPO from an SFT checkpoint
Peak LR around 1e-4 to 2e-4: same regime as LoRA, with the 4-bit base unchanged by LR choice.
Continued pretraining on a new corpus
Peak LR around 5e-7 to 5e-6, much lower than SFT because preference gradients are fragile.
Peak LR tracks how fragile the update is: LoRA SFT runs hot near 1e-4 to 5e-4, full fine-tuning sits mid, DPO runs coldest. Continued pretraining re-warms toward the pretraining rate and uses replay.
Picture how bold each change is when you fix up something you already care about. If you scribble on a fresh sticky note that started blank, you can press hard and write fast, since there is nothing to ruin. If you touch up a finished painting, you dab gently so you do not smear years of work. If you nudge someone toward a very subtle taste, the hint is so faint that any big stroke wipes it out, so you barely tap. And if you teach a wise old friend a whole new library, you can teach at a normal pace again. You just need to keep reviewing the old books so they do not forget them. For small edits the rule is the same: the more precious and finished the thing, the smaller your step should be.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A single organising principle sits behind five seemingly unrelated numbers. The peak learning rate for each fine-tuning regime is not chosen by model size or by tradition. It is chosen by how fragile the update is and how much already-useful knowledge is exposed to corruption.
The bands span three orders of magnitude, from 5e-7 for DPO up to 5e-4 for LoRA SFT. That spread looks intimidating until you reorder the regimes by one axis: how much is at stake per step. Frozen-base adapter methods have almost nothing to lose, so they run hot. Full-parameter and preference methods expose precious signal to every gradient, so they run cold.
The organising axis: fragility, not size
A common instinct is that bigger models need smaller learning rates. That correlation is weak and misleading here. A 70B QLoRA run uses a higher peak rate than a 7B full fine-tune, which immediately breaks the size heuristic. Size barely moves the safe band. What moves it is what the gradient can break.
The real axis is fragility: how much damage one optimiser step can do. Two factors set it. First, how many live parameters carry pretrained knowledge that a hot step could overwrite. Second, how noisy or small the gradient signal is, since a faint signal amplified by a large step produces unstable updates. A regime that scores low on both can afford a hot rate. A regime that scores high on either must run cold.
There is a third factor: how far the update travels from a known-good starting point. A LoRA adapter starts at zero and adds a small correction, so even a hot rate keeps the effective model close to the frozen base. Full fine-tuning and continued pretraining move the actual weights, so every step compounds and drift accumulates across thousands of updates. DPO moves weights too, but from an already-aligned checkpoint, where any large move is a regression rather than progress.
When you score each regime on these factors, the ordering falls out cleanly. LoRA and QLoRA freeze the base, so live-knowledge risk is near zero. Full fine-tuning exposes every weight. DPO carries an unusually fragile gradient on top of a checkpoint you do not want to disturb. Continued pretraining is the exception: it wants large moves on a new corpus, so recipes re-warm the rate and protect old knowledge with replay instead. For the fine-tuning regimes, that single axis, not parameter count, predicts the band, and it predicts the schedule shape too.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Regime | Peak LR band | Why this band |
|---|---|---|
| LoRA r=16 SFT | 1e-4 to 5e-4 | Frozen base, zero-init adapters tolerate ~10x higher LR |
| QLoRA on 70B | 1e-4 to 2e-4 | Same as LoRA; 4-bit base is frozen and LR-inert |
| Full FT of 7B | 1e-5 to 5e-5 | All weights live; hot LR forgets or diverges |
| Continued pretraining | re-warmed toward the original pretraining peak | Many new tokens; replay old data to limit forgetting |
| DPO from SFT checkpoint | 5e-7 to 5e-6 | Fragile log-ratio gradient destabilises easily |
Real products, models, and research that use this idea.
- Hugging Face TRL's DPOConfig defaults to a learning rate of 1e-6, while LoRA SFT recipes commonly use about 2e-4, mirroring this fragility ordering.
- Axolotl example configs and Unsloth notebooks for LoRA fine-tunes of open models such as Llama 3.1 and Qwen3 commonly use a peak LR of 1e-4 to 2e-4.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does a zero-initialised LoRA adapter tolerate a much higher peak learning rate than full fine-tuning?
Reason about what is at risk. The base is frozen, so no pretrained weight can be corrupted. The adapter starts at zero and adds nothing until trained, so a larger step fills capacity without destroying prior knowledge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reusing one favourite learning rate across regimes. A LoRA-tuned 2e-4 will blow up full fine-tuning and is roughly 100 times too hot for DPO.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.