**Flat loss from step 0** is a structural problem. The model is not learning anything, which almost always means no useful gradient is flowing into the trainable parameters.
Three first checks, each targeting a separate way the optimization path can be broken.
First, **the learning rate after warmup**. Log the actual LR value at step 100 or 200, not just the configured one. A common bug is misconfigured total_steps or warmup_steps that leaves the post-warmup LR at zero. The cosine decay schedule needs total_steps to be set correctly; if it is wrong, the LR can decay to zero almost immediately. A zero LR multiplies every update by zero, so the model never moves regardless of gradient.
Second, **the target_modules names**. PEFT wraps modules by exact name match. If target_modules contains 'q_proj' and 'v_proj' but the base model uses 'wq' and 'wv' (a common Llama versus Mistral naming gotcha), PEFT silently wraps nothing. No wrapped modules means no trainable parameters on the forward path, which means no gradient. Run `model.print_trainable_parameters()` immediately after wrapping; if the count is zero or implausibly small, the target_modules are wrong.
Third, **the label masking**. SFT trainers mask prompt tokens to -100 so only response tokens contribute to loss. If the chat template fails to produce assistant tokens (a template bug, or a tokenizer not configured with the assistant role), every label in the batch can end up as -100. The loss is then structurally zero. Inspect one tokenized batch and confirm at least some labels are not -100.
The wrong instincts: bumping rank or switching base models. Neither helps when the gradient pathway is broken.
**Flat loss from step 0** is structural, not a capacity or precision problem. No gradient is reaching the trainable parameters. The three first checks target independent failure modes.
**Learning rate after warmup.** Log the actual LR at step 100, not the configured value. The common bug is a schedule misconfig: total_steps set wrong so cosine decay zeros immediately, or warmup_steps larger than total_steps. A zero LR multiplies every gradient by zero, and parameters never move. Verify the value the Trainer logs each step; do not trust the config.
**target_modules names.** PEFT's LoraConfig wraps modules by exact string match. Llama-3 and Mistral use q_proj, k_proj, v_proj, o_proj. Older or custom models use different names. If target_modules contains a name not in the base, PEFT silently wraps nothing. Call `model.print_trainable_parameters()` immediately after wrapping; a count of zero or implausibly small means target_modules is wrong. Fix: introspect via `named_modules()` and match exactly.
**Assistant tokens after label masking.** SFT trainers mask prompt tokens to -100. If the chat template fails to produce assistant tokens, every label becomes -100 and the loss is structurally zero. Common causes: tokenizer has no assistant role, template emits empty turns, dataset format mismatches the template. Verify `(batch['labels'] != -100).sum() > 0` and decode a sample.
The distractors fail for one reason: they target capacity, model size, or precision, none of which help when the gradient path is broken. Higher rank is irrelevant if no gradient flows; a bigger base with the same broken wiring is still broken; fp32 does not produce loss out of nothing.
Quick reference
TL;DR
Flat loss from step 0 is a structural break, not a tuning problem. Check the LR schedule, the target_modules wiring, and whether assistant tokens survive label masking.
Imagine trying to drive a car that will not move. Three things to check first: is there fuel reaching the engine (the labels you are training on), is the engine connected to the wheels (the adapter actually wrapping a layer), and is the gas pedal pressed (the step-size knob the trainer uses not stuck at zero)? Only after confirming all three are working do you start asking whether the engine is too small or the wheels are the wrong size. People who jump straight to swapping engines waste hours when the real fix is plugging in a cable or pressing the pedal. Flat loss is the same: check the structural breaks before changing capacity or precision.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Loss drops normally for a while then stalls Different symptom from flat from step 0; the optimization path is working, the issue is capacity or data quality.
Loss spikes to NaN after a clean start Mid-run failure points at numerical instability, not at the structural checks for flat from zero.
Eval loss diverges from train loss Overfitting signature, not a flat-loss diagnostic; reach for early stopping and capacity reduction instead.
Quick scan · 2–4 min
Everything important, quickly.
4 min: why flat loss from step 0 is structural, the three independent failure modes (LR, wrapping, labels), the diagnostic one-liners for each, why capacity and precision fixes do not apply, and the general diagnostic path order.
Side-by-side
Symptom
Likely cause
First check
Flat loss from step 0 (this case)
Structural break in optimization path
LR, wrapping, label masking
Loss drops then spikes to NaN
Numerical overflow or schedule discontinuity
Mixed precision, gradient clipping, beta
Loss drops then plateaus
Model is converged or undercapacity
Eval loss trend; capacity if eval flat
Loss drops then eval rises
Overfitting
Early stopping; reduce capacity next run
Loss oscillates wildly
LR too high or batch size too small
Lower LR, larger batch, more warmup
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT issues GitHub history is full of target_modules name mismatches between Llama-3 and Mistral, and the standard fix is print_trainable_parameters before training starts.
2TRL SFTTrainer in 2026 logs a warning when more than a configurable fraction of labels are -100, exactly because the all-masked failure mode is common.
3Axolotl YAML configs default to verifying the actual post-warmup LR in the first logging output to catch schedule misconfig immediately.
4Unsloth single-GPU Llama 4 fine-tunes ship example configs that print a tokenized batch sample on startup so users can spot label masking bugs before wasting compute.
5Production Qwen 3.5 fine-tunes at large open-source orgs include a 'sanity training' step of 10 iterations to verify loss drops before launching the full run.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you verify the actual post-warmup learning rate is nonzero in Hugging Face Trainer?
Sign in or go Pro to see the approach for this follow-up.
QWhy does target_modules silently fail instead of raising an error on a name mismatch?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right way to check label masking in an SFT batch?
Sign in or go Pro to see the approach for this follow-up.
QIf all three checks pass and loss is still flat, what would you look at next?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Bumping rank or switching models when loss is flat from step 0. Capacity changes do not help when no gradient is flowing; the fix is upstream in the data, the wiring, or the optimizer.