QLoRA OOMs at step 7000 after one clean epoch on a 24 GB GPU: which non-obvious causes deserve a look?

Multi-select·Medium·4.0 · 0·~1 min·Relevant atDatabricks Meta Microsoft NVIDIA

Attempt it

The real culprits are A, B, and E. Each one matches the failure pattern of OOM at step 7000 after a clean first epoch. A is about sequence packing. The trainer concatenates short examples up to max_seq_len to use the GPU efficiently. The lengths of packed sequences vary; most packs are smaller than the cap, but the distribution has a long tail. After 7000 steps you have drawn enough packs to eventually hit one near the cap. Peak activation memory for that pack spikes well above the typical step, and a card already running near its limit tips over. B is about eval. If the periodic eval loop runs without torch.no_grad or inference_mode, every forward pass during eval builds an autograd graph as if you were about to backprop. That transient graph eats memory the same way a training forward does, doubling working memory during eval. On a tight card the eval spike is what OOMs. E is about logging. If you accumulate per-step losses in a Python list without calling .detach() or .item() first, every loss tensor keeps its autograd graph alive. Memory grows linearly with steps. After 7000 steps you have 7000 graphs in memory, and eventually that bumps the card over. C and D are wrong. Frameworks cache base weights locally and do not silently re-download mid-run. AdamW allocates its m and v buffers once at the first optimizer step and reuses them; state does not grow per step.

**Correct: A, B, E.** Each matches the failure signature (clean first epoch, OOM mid-second epoch). **A (packed-sequence spike).** Sequence packing concatenates short examples up to max_seq_len. The pack-length distribution has a long tail; enough samples eventually produce a near-cap pack. Peak attention activations scale O(L^2), so a long pack spikes far above the average. After 7000 steps you have drawn enough samples to hit one. **B (eval without inference_mode).** Forward passes inside torch.no_grad or torch.inference_mode skip autograd graph construction. Without that wrapper, every forward during eval transiently allocates activation memory as if a backward will follow, doubling working memory at the eval moment. **E (undetached loss accumulation).** Storing loss tensors in a Python list without `.detach()` or `.item()` keeps the autograd graph attached. Memory grows linearly with steps. After thousands of steps the cumulative graph mass becomes significant. **C is wrong** because Hugging Face caches weights on first load and serves from cache; there is no silent re-download mid-run. **D is wrong** because AdamW allocates m and v buffers once at the first optimizer step and reuses them. State is constant after step 1. Diagnostic: plot peak memory per step. Monotonic growth points at E; rare spikes at A; cadence-aligned spikes at B.

Quick reference

TL;DR

A, B, and E are the real causes: rare packed-sequence spikes (A), eval without inference_mode (B), and undetached loss accumulation (E). C does not happen; D misdescribes AdamW.

Memory aid

Easy to grasp

Picture a backpack you can just barely close. Most days it holds the same kit of items and zips fine. Then one rare day a friend hands you a long umbrella that does not normally fit, and the backpack will not close. Or you start carrying a small souvenir from every day on top of the regular kit, and after enough days the backpack overflows. Or one afternoon you decide to also pack a duplicate of your bag inside it temporarily, and that is too much weight at once. Three of the listed problems are stories like these. The other two are not really problems at all; they describe things that never actually happen in the system.

Key concepts

QLoRA Memory debugging Sequence packing Training stability

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: classify each option as slow-growth, rare-spike, or impossible, then explain the three real causes (A, B, E) with their diagnostic signals and mitigations.

Side-by-side

Failure mode	Time signature	Diagnostic signal	Fix
A: long-pack spike	Random late step	Peak memory correlates with max seq length	Pre-allocate worst-case or cap pack length
B: eval graph	On eval step multiples	OOM aligns with eval cadence	Wrap eval in torch.inference_mode()
E: undetached leak	Monotonic growth	memory_allocated climbs steadily	Call .detach() or .item() on logged tensors
C: re-download (wrong)	n/a	Does not happen	n/a
D: optimizer growth (wrong)	n/a	Constant after step 1	n/a

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL SFTTrainer documentation explicitly warns about sequence packing producing long-tail packs, which is the canonical setup for the A failure mode.
PyTorch documentation for torch.inference_mode covers the eval-time memory implications and recommends it over torch.no_grad for inference and eval loops.
The detach-or-item pattern for loss logging is in every modern training tutorial precisely because the E failure mode is so common in custom training loops.
Practitioners debugging QLoRA on consumer 24 GB cards (RTX 4090, RTX 5090) regularly trace mid-run OOMs to one of A, B, or E rather than to model size itself.
Production fine-tuning tooling like Unsloth pre-allocates worst-case buffers for sequence packing to make the A failure mode impossible by construction.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument a training loop to distinguish A, B, and E in practice?

QWhy does sequence-packing peak memory scale O(L^2) and what mitigates it?

QWhat is the difference between torch.no_grad and torch.inference_mode, and when should you prefer each?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking D because AdamW state sounds like a growth source, or picking C because re-downloads sound plausible. Neither describes actual framework behavior; the real culprits are spikes (A), eval-time graph builds (B), and undetached accumulation (E).

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.