Pick the three first things to check when LoRA training loss sits flat from step 0

Multi-select·Medium·4.0 · 0·~1 min·Relevant atDatabricks Meta OpenAI

Attempt it

**Flat loss from step 0** is a structural problem. The model is not learning anything, which almost always means no useful gradient is flowing into the trainable parameters. Three first checks, each targeting a separate way the optimization path can be broken. First, **the learning rate after warmup**. Log the actual LR value at step 100 or 200, not just the configured one. A common bug is misconfigured total_steps or warmup_steps that leaves the post-warmup LR at zero. The cosine decay schedule needs total_steps to be set correctly; if it is wrong, the LR can decay to zero almost immediately. A zero LR multiplies every update by zero, so the model never moves regardless of gradient. Second, **the target_modules names**. PEFT wraps modules by exact name match. If target_modules contains 'q_proj' and 'v_proj' but the base model uses 'wq' and 'wv' (a common Llama versus Mistral naming gotcha), PEFT silently wraps nothing. No wrapped modules means no trainable parameters on the forward path, which means no gradient. Run `model.print_trainable_parameters()` immediately after wrapping; if the count is zero or implausibly small, the target_modules are wrong. Third, **the label masking**. SFT trainers mask prompt tokens to -100 so only response tokens contribute to loss. If the chat template fails to produce assistant tokens (a template bug, or a tokenizer not configured with the assistant role), every label in the batch can end up as -100. The loss is then structurally zero. Inspect one tokenized batch and confirm at least some labels are not -100. The wrong instincts: bumping rank or switching base models. Neither helps when the gradient pathway is broken.

**Flat loss from step 0** is structural, not a capacity or precision problem. No gradient is reaching the trainable parameters. The three first checks target independent failure modes. **Learning rate after warmup.** Log the actual LR at step 100, not the configured value. The common bug is a schedule misconfig: total_steps set wrong so cosine decay zeros immediately, or warmup_steps larger than total_steps. A zero LR multiplies every gradient by zero, and parameters never move. Verify the value the Trainer logs each step; do not trust the config. **target_modules names.** PEFT's LoraConfig wraps modules by exact string match. Llama-3 and Mistral use q_proj, k_proj, v_proj, o_proj. Older or custom models use different names. If target_modules contains a name not in the base, PEFT silently wraps nothing. Call `model.print_trainable_parameters()` immediately after wrapping; a count of zero or implausibly small means target_modules is wrong. Fix: introspect via `named_modules()` and match exactly. **Assistant tokens after label masking.** SFT trainers mask prompt tokens to -100. If the chat template fails to produce assistant tokens, every label becomes -100 and the loss is structurally zero. Common causes: tokenizer has no assistant role, template emits empty turns, dataset format mismatches the template. Verify `(batch['labels'] != -100).sum() > 0` and decode a sample. The distractors fail for one reason: they target capacity, model size, or precision, none of which help when the gradient path is broken. Higher rank is irrelevant if no gradient flows; a bigger base with the same broken wiring is still broken; fp32 does not produce loss out of nothing.

Quick reference

TL;DR

Flat loss from step 0 is a structural break, not a tuning problem. Check the LR schedule, the target_modules wiring, and whether assistant tokens survive label masking.

Memory aid

Easy to grasp

Imagine trying to drive a car that will not move. Three things to check first: is there fuel reaching the engine (the labels you are training on), is the engine connected to the wheels (the adapter actually wrapping a layer), and is the gas pedal pressed (the step-size knob the trainer uses not stuck at zero)? Only after confirming all three are working do you start asking whether the engine is too small or the wheels are the wrong size. People who jump straight to swapping engines waste hours when the real fix is plugging in a cable or pressing the pedal. Flat loss is the same: check the structural breaks before changing capacity or precision.

Key concepts

LoRA adapters Chat template Learning rate schedule

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: why flat loss from step 0 is structural, the three independent failure modes (LR, wrapping, labels), the diagnostic one-liners for each, why capacity and precision fixes do not apply, and the general diagnostic path order.

Side-by-side

Symptom	Likely cause	First check
Flat loss from step 0 (this case)	Structural break in optimization path	LR, wrapping, label masking
Loss drops then spikes to NaN	Numerical overflow or schedule discontinuity	Mixed precision, gradient clipping, beta
Loss drops then plateaus	Model is converged or undercapacity	Eval loss trend; capacity if eval flat
Loss drops then eval rises	Overfitting	Early stopping; reduce capacity next run
Loss oscillates wildly	LR too high or batch size too small	Lower LR, larger batch, more warmup

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT issues GitHub history is full of target_modules name mismatches between Llama-3 and Mistral, and the standard fix is print_trainable_parameters before training starts.
TRL SFTTrainer in 2026 logs a warning when more than a configurable fraction of labels are -100, exactly because the all-masked failure mode is common.
Axolotl YAML configs default to verifying the actual post-warmup LR in the first logging output to catch schedule misconfig immediately.
Unsloth single-GPU Llama 4 fine-tunes ship example configs that print a tokenized batch sample on startup so users can spot label masking bugs before wasting compute.
Production Qwen 3.5 fine-tunes at large open-source orgs include a 'sanity training' step of 10 iterations to verify loss drops before launching the full run.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you verify the actual post-warmup learning rate is nonzero in Hugging Face Trainer?

QWhy does target_modules silently fail instead of raising an error on a name mismatch?

QWhat is the right way to check label masking in an SFT batch?

QIf all three checks pass and loss is still flat, what would you look at next?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Bumping rank or switching models when loss is flat from step 0. Capacity changes do not help when no gradient is flowing; the fix is upstream in the data, the wiring, or the optimizer.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.