QLoRA OOMs at step 7000 after one clean epoch on a 24 GB GPU: which non-obvious causes deserve a look?

Attempt it

Quick reference

TL;DR

A, B, and E are the real causes: rare packed-sequence spikes (A), eval without inference_mode (B), and undetached loss accumulation (E). C does not happen; D misdescribes AdamW.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a backpack you can just barely close. Most days it holds the same kit of items and zips fine. Then one rare day a friend hands you a long umbrella that does not normally fit, and the backpack will not close. Or you start carrying a small souvenir from every day on top of the regular kit, and after enough days the backpack overflows. Or one afternoon you decide to also pack a duplicate of your bag inside it temporarily, and that is too much weight at once. Three of the listed problems are stories like these. The other two are not really problems at all; they describe things that never actually happen in the system.

Key concepts

QLoRA Memory debugging Sequence packing Training stability

Quick scan · 2–4 min

Everything important, quickly.

5 min: classify each option as slow-growth, rare-spike, or impossible, then explain the three real causes (A, B, E) with their diagnostic signals and mitigations.

Side-by-side

Failure mode	Time signature	Diagnostic signal	Fix
A: long-pack spike	Random late step	Peak memory correlates with max seq length	Pre-allocate worst-case or cap pack length
B: eval graph	On eval step multiples	OOM aligns with eval cadence	Wrap eval in torch.inference_mode()
E: undetached leak	Monotonic growth	memory_allocated climbs steadily	Call .detach() or .item() on logged tensors
C: re-download (wrong)	n/a	Does not happen	n/a
D: optimizer growth (wrong)	n/a	Constant after step 1	n/a

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL SFTTrainer documentation explicitly warns about sequence packing producing long-tail packs, which is the canonical setup for the A failure mode.
PyTorch documentation for torch.inference_mode covers the eval-time memory implications and recommends it over torch.no_grad for inference and eval loops.
The detach-or-item pattern for loss logging is in every modern training tutorial precisely because the E failure mode is so common in custom training loops.
Practitioners debugging QLoRA on consumer 24 GB cards (RTX 4090, RTX 5090) regularly trace mid-run OOMs to one of A, B, or E rather than to model size itself.
Production fine-tuning tooling like Unsloth pre-allocates worst-case buffers for sequence packing to make the A failure mode impossible by construction.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument a training loop to distinguish A, B, and E in practice?

Sign in or go Pro to see the approach for this follow-up.

QWhy does sequence-packing peak memory scale O(L^2) and what mitigates it?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the difference between torch.no_grad and torch.inference_mode, and when should you prefer each?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.