Pick the three first things to check when LoRA training loss sits flat from step 0

Multi-select·Medium·4.0 · 0·~1 min·Asked atCrewai OpenAI Unity·Relevant atCoreweave Databricks Fireworks Ai Lambda Labs

Attempt it

**Flat loss from step 0** is a structural problem. The model is not learning anything, which almost always means no useful gradient is flowing into the trainable parameters. Three first checks, each targeting a separate way the optimization path can be broken. First, **the learning rate after warmup**. Log the actual LR value at step 100 or 200, not just the configured one. A common bug is misconfigured total_steps or warmup_steps that leaves the post-warmup LR at zero. The cosine decay schedule needs total_steps to be set correctly; if it is wrong, the LR can decay to zero almost immediately. A zero LR multiplies every update by zero, so the model never moves regardless of gradient. Second, **the target_modules names**. PEFT wraps modules by exact name match. If target_modules contains 'q_proj' and 'v_proj' but the base model uses 'wq' and 'wv' (a common Llama versus Mistral naming gotcha), PEFT silently wraps nothing. No wrapped modules means no trainable parameters on the forward path, which means no gradient. Run `model.print_trainable_parameters()` immediately after wrapping; if the count is zero or implausibly small, the target_modules are wrong. Third, **the label masking**. SFT trainers mask prompt tokens to -100 so only response tokens contribute to loss. If the chat template fails to produce assistant tokens (a template bug, or a tokenizer not configured with the assistant role), every label in the batch can end up as -100. The loss is then structurally zero. Inspect one tokenized batch and confirm at least some labels are not -100. The wrong instincts: bumping rank or switching base models. Neither helps when the gradient pathway is broken.

**Flat loss from step 0** is structural, not a capacity or precision problem. No gradient is reaching the trainable parameters. The three first checks target independent failure modes. **Learning rate after warmup.** Log the actual LR at step 100, not the configured value. The common bug is a schedule misconfig: total_steps set wrong so cosine decay zeros immediately, or warmup_steps larger than total_steps. A zero LR multiplies every gradient by zero, and parameters never move. Verify the value the Trainer logs each step; do not trust the config. **target_modules names.** PEFT's LoraConfig wraps modules by exact string match. Llama-3 and Mistral use q_proj, k_proj, v_proj, o_proj. Older or custom models use different names. If target_modules contains a name not in the base, PEFT silently wraps nothing. Call `model.print_trainable_parameters()` immediately after wrapping; a count of zero or implausibly small means target_modules is wrong. Fix: introspect via `named_modules()` and match exactly. **Assistant tokens after label masking.** SFT trainers mask prompt tokens to -100. If the chat template fails to produce assistant tokens, every label becomes -100 and the loss is structurally zero. Common causes: tokenizer has no assistant role, template emits empty turns, dataset format mismatches the template. Verify `(batch['labels'] != -100).sum() > 0` and decode a sample. The distractors fail for one reason: they target capacity, model size, or precision, none of which help when the gradient path is broken. Higher rank is irrelevant if no gradient flows; a bigger base with the same broken wiring is still broken; fp32 does not produce loss out of nothing.

TL;DR

Flat loss from step 0 is a structural break, not a tuning problem. Check the LR schedule, the target_modules wiring, and whether assistant tokens survive label masking.

Memory aid

Easy to grasp

Imagine trying to drive a car that will not move. Three things to check first: is there fuel reaching the engine (the labels you are training on), is the engine connected to the wheels (the adapter actually wrapping a layer), and is the gas pedal pressed (the step-size knob the trainer uses not stuck at zero)? Only after confirming all three are working do you start asking whether the engine is too small or the wheels are the wrong size. People who jump straight to swapping engines waste hours when the real fix is plugging in a cable or pressing the pedal. Flat loss is the same: check the structural breaks before changing capacity or precision.

Key concepts

LoRA adapters Chat template Learning rate schedule

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Debugging flat loss is one of the cleanest signals fine-tuning produces. The symptom is unambiguous (loss does not move from step 0), the failure modes are structural, and the diagnostic order is well-defined. Senior engineers should be able to walk the checks in under five minutes; junior engineers often waste hours bumping rank or switching base models because they reach for the levers they know rather than the diagnostics the symptom calls for.

The specific failure mode in this question is the easiest to debug and the easiest to misdiagnose. Flat from step 0 specifically excludes the categories of mid-run failures (NaN spikes, plateaus, oscillation, overfitting) and points squarely at structural breaks in the optimization path. The three first checks (learning rate after warmup, target_modules wiring, assistant tokens after label masking) cover the failure modes that actually fire in production fine-tuning stacks, and each one has a one-line diagnostic that takes seconds to run.

The rest of this section walks each failure mode in detail with its diagnostic and fix, explains why the distractors (bump rank, switch model, disable mixed precision) cannot help, and ends with the general diagnostic path order that should govern any fine-tuning debugging session.

Learning rate after warmup

The first check is the optimizer's learning rate. A zero LR multiplies every gradient by zero in the update rule, so the parameters never move regardless of how large the gradient actually is. The cleanest way this happens in production is a schedule misconfig.

The canonical bug: total_steps is set incorrectly. With a cosine decay schedule, LR drops from the peak value at the end of warmup down to zero at total_steps. If total_steps is much smaller than the actual training duration (because of a mis-counted batch size, gradient accumulation, or epoch count), the cosine decay zeros out within the first few hundred steps and stays there. Loss never moves.

Another variant: warmup_steps is set larger than total_steps. The LR is still ramping up when the schedule nominally ends, so the actual training LR stays at zero or near-zero throughout.

The diagnostic is to log the actual LR at step 100 or 200, after warmup should be complete. Hugging Face Trainer includes learning_rate in the metrics dict at each logging step. If you see 0.0 or 1e-10, the schedule is broken. The fix is to recalculate total_steps as len(train_dataloader) * num_epochs // gradient_accumulation_steps and reset warmup_steps to a small fraction (typically 0.03 to 0.10 of total_steps).

A related failure mode: the optimizer itself is wrong. AdamW with weight_decay set to something massive (like 1.0 instead of 0.01) can produce updates that effectively cancel each other on the small adapter parameters. Less common but worth checking if the LR diagnostic comes back clean.

target_modules wiring and the silent zero-wrap failure

Assistant tokens after label masking

Why the distractors cannot help and the general diagnostic order

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Symptom	Likely cause	First check
Flat loss from step 0 (this case)	Structural break in optimization path	LR, wrapping, label masking
Loss drops then spikes to NaN	Numerical overflow or schedule discontinuity	Mixed precision, gradient clipping, beta
Loss drops then plateaus	Model is converged or undercapacity	Eval loss trend; capacity if eval flat
Loss drops then eval rises	Overfitting	Early stopping; reduce capacity next run
Loss oscillates wildly	LR too high or batch size too small	Lower LR, larger batch, more warmup

Real products, models, and research that use this idea.

Hugging Face PEFT issues GitHub history is full of target_modules name mismatches between Llama-3 and Mistral, and the standard fix is print_trainable_parameters before training starts.
TRL SFTTrainer in 2026 logs a warning when more than a configurable fraction of labels are -100, exactly because the all-masked failure mode is common.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you verify the actual post-warmup learning rate is nonzero in Hugging Face Trainer?

Trainer logs include the learning_rate field at each logging step. Check the metric dictionary at step 100 or 200, after warmup is complete. If the value is zero or near-zero, the schedule is broken; common cause is total_steps mis-set or warmup_steps larger than total_steps.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Bumping rank or switching models when loss is flat from step 0. Capacity changes do not help when no gradient is flowing; the fix is upstream in the data, the wiring, or the optimizer.

60 second bullets to scan on the way to the call.

Why flat loss from step 0 is always a structural break, not a tuning issue
The three first checks: LR, target_modules wiring, label masking

Primary sources. Browse if you want the original framing.