Full FT a 7B model around 1e-5 to 5e-5; LoRA roughly 10x higher near 1e-4 to 5e-4; DPO much lower near 5e-7 to 5e-6. Always cosine schedule with linear warmup.
Imagine fixing up something valuable, and the size of your step is how bold each change is. If you repaint a whole finished masterpiece, you dab gently, because one big swipe ruins years of work. If you only add a few stick-on tabs that started blank, you can press hard and move fast, since nothing precious is underneath. The trickiest job is nudging someone's taste toward what they should like best: the hint is so faint that any big shove smears it, so you barely tap. And no matter the job, you start slow to find your footing, speed up, then slow down again as you near the finish, like easing into a parking spot.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Learning rate is the single most consequential hyperparameter in fine-tuning, and the right value swings across orders of magnitude depending on which tuning method you run. A value that is perfect for LoRA will blow up a DPO run, and a value safe for DPO will barely train a LoRA adapter. The numbers in this question are not arbitrary folklore; they fall out of two underlying questions.
The first question is how many parameters you are moving, and from where. The second is how fragile the loss landscape is around your starting point. Full fine-tuning moves every weight from an already-converged minimum. LoRA moves only a handful of zero-initialized adapters. DPO moves the whole policy along a sharp preference gradient tethered to a reference. Each regime answers those two questions differently, and the peak learning rate follows.
Interviewers like this question because it separates people who memorized one tutorial number from people who understand why the numbers differ. If you can explain the why, you can reason about a base model size or training method the tutorial never covered. The bands below are the defaults shipped by mainstream frameworks in 2026, so they double as a sanity check: if your config is far outside them, you usually have a bug rather than a clever idea.
On top of the peak, every modern recipe wraps the run in a schedule: a short linear warmup followed by cosine decay toward zero. The schedule is not decoration. It is what makes the aggressive peaks survivable and what lets the model settle cleanly at the end. This deep dive walks each method, explains why its band sits where it does, and covers the schedule that ties them together.
Full fine-tuning: small steps from a delicate minimum
When you full fine-tune a 7B model, every weight is trainable and every weight already sits in a carefully converged pretrained state. That state encodes broad language ability, and the minimum is delicate. Large updates do not gently improve it; they shove the weights into a different region and erase pretrained knowledge, the failure mode known as catastrophic forgetting.
The practical consequence is a small peak learning rate, roughly 1e-5 to 5e-5 for a 7B base. Within that band you nudge the weights toward your task without flattening what the model already knew. Push above it and you see loss spikes, divergence, or a model that suddenly fails simple capability checks. The contrast with pretraining is instructive: pretraining a base model from scratch often runs at 1e-4 or higher, because the weights start random and there is no delicate prior to protect. Fine-tuning is gentler precisely because there is something worth preserving.
The band scales with model size too. Larger bases generally want the lower end or below, because the minimum gets more delicate as parameter count grows. The 7B numbers here are the common reference point precisely because 7B and 8B open-weight models are the most fine-tuned size class. They are also memory-feasible to full fine-tune on a single high-end node, which is part of why the community converged on well-tested defaults for that size.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Method | Peak LR (7B) | Why | Schedule |
|---|---|---|---|
| Full fine-tuning | 1e-5 to 5e-5 | All weights move from a delicate converged minimum | Cosine, 3 to 10 percent warmup |
| LoRA (rank 16) | 1e-4 to 5e-4 | Only zero-init adapters train, so more headroom | Cosine, short warmup |
| DPO from SFT | 5e-7 to 5e-6 | Fragile preference gradient near the reference policy | Cosine or constant, short warmup |
Real products, models, and research that use this idea.
- Axolotl and Hugging Face TRL ship cosine schedule with linear warmup as the default for SFT and DPO runs on Llama 4 and similar open-weight bases.
- Unsloth's LoRA notebooks for Llama 4 and Qwen3 default to a peak around 2e-4 with a short warmup ratio, matching the 1e-4 to 5e-4 LoRA band.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can LoRA tolerate a higher learning rate than full fine-tuning of the same base?
Frame around what moves and where it starts. Only low-rank adapters train, they initialize to zero, and the frozen base anchors the output, so early large steps cannot destabilize pretrained knowledge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reusing one learning rate across full FT, LoRA, and DPO. The three differ by orders of magnitude, and a DPO run at SFT learning rate collapses into degenerate output.
60 second bullets to scan on the way to the call.
Full fine-tuning peak range for a 7B model
Why LoRA tolerates roughly ten times higher
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.